Second pass at: Journal club: STEP 4 and what a withdrawal design can prove

Critical appraisal template: (1) What did the trial set out to estimate? (2) Could the design answer that question? (3) Was the population sufficiently similar to your population to apply the results? (4) What was the absolute effect, not just the relative one? (5) What are the two strongest criticisms available?

That is a description of practice, not a recommendation of it.

On STEP 4, the part that usually goes wrong is that the question is asked as though it has one answer. It has a range, and the width of the range is the interesting bit.

If you can post the two or three numbers you are working from, several people here will check the arithmetic rather than argue about the conclusion.

The honest answer on STEP 4 is that it depends, and the useful part is the list of what it depends on. Four items, in rough order of how much they matter.

Most people get the first two right and then argue about the fourth.

The most common failure mode is spending fifty minutes on the effect size and ten on the population. Reversing that ratio would improve most sessions.

Two sources, same conclusion, and I could not rule out that one copied the other.

Two claims get bundled together under STEP 4 and they need separating. The descriptive one — this is what was observed — is usually well supported. The causal one — this is why — usually is not.

Almost every disagreement in threads like this one dissolves once you say which of the two you are making.

STEP 1 (N Engl J Med 2021): The pivotal obesity trial for semaglutide and the reference point for most subsequent comparison. Mean weight reduction was substantially larger than anything previously achieved pharmacologically.

That is the version I use. It may not be the version that is correct.

The most common failure mode is spending fifty minutes on the effect size and ten on the population. Reversing that ratio would improve most sessions.

Worth one more sentence than it usually gets.

Critical appraisal template: (1) What did the trial set out to estimate? (2) Could the design answer that question? (3) Was the population sufficiently similar to your population to apply the results? (4) What was the absolute effect, not just the relative one? (5) What are the two strongest criticisms available?

The short answer was in the first line; everything after is the working.

The baseline characteristics table is the single most useful page for the questions that get asked here, because it tells you who the result applies to.

That is the version I would defend. It is not the version I started with.