Evidence · Journal club

Journal club: STEP 4 and what a withdrawal design can prove

Evidence › Journal club randomised trial risk of bias effect size

Solved

Solved by JFitzgibbon in post #2

The opening post describes the usual case. This is about the unusual one. Having read the whole STEP 4 thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead.

Jump to the accepted answer →

DKwiatkowskiTL3Regular14 Feb 2025#1

On the subject in the title: Journal club: STEP 4 and what a withdrawal design can prove Working notes rather than a conclusion.

Session topic: STEP 4 (JAMA, 2021). Please read it before posting; the discussion is much better when everyone has.

The question I would like us to start with is what the trial set out to estimate, rather than what it found. Once that is on the table we can talk about whether the design could have answered it, and only then about the numbers.

Specific things I would like covered: the population and how far it generalises, how discontinuation was handled, whether the comparator was a fair one, and what the absolute rather than relative effect looks like.

I will summarise at the end and the summary will feed the relevant digest page.

11 likes 17mo

JFitzgibbonTL2Member Solution19 Feb 2025#2

The opening post describes the usual case. This is about the unusual one.

Having read the whole STEP 4 thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead.

15 likes 17mo

to.vargaTL223 Feb 2025#3

DKwiatkowski, post #1: On the subject in the title: Journal club: STEP 4 and what a withdrawal design can prove Working notes rather than a conclusion. Session topic: STEP 4 ( JAMA , 2021). Please read it before posting; the discussion is much better when everyone has. The question I would like us to start with is what the trial set out to estimate, rather… Go to post

Thank you for taking the time. That was more work than a reply usually is.

30 likes in reply to #1 17mo

g.haalandTL3Regular26 Feb 2025#4

STEP 4 is worth one more sentence than it usually gets, and the sentence is the one about how the number was arrived at.

0 likes in reply to #1 17mo

i.oseiTL21 Mar 2025#5

Picking up post #2: that is the part I would want checked first.

The figures usually contain the finding and the text usually contains the interpretation. Separating them for the first twenty minutes is a discipline worth keeping.

Not disagreeing with anyone above, just adding the bit I keep having to look up.

3 likes 17mo

cohort_driftTL3Regular4 Mar 2025 · edited#6

On post #4 — agreed on the reasoning, with one qualification.

STEP 4 sits at the boundary between what this community can usefully discuss and what it cannot, and I think it falls on the discussable side, narrowly.

10 likes 17mo

s.oyelaranTL27 Mar 2025#7

An update on my earlier STEP 4 post: the pattern held for another six weeks and then stopped, which I did not predict and cannot explain.

23 likes 17mo

OTeixeiraTL3Regular10 Mar 2025#8

to.varga, post #3: Thank you for taking the time. That was more work than a reply usually is. Go to post

Where the group cannot agree, record the disagreement rather than resolving it by seniority. The recorded disagreement is more honest and more useful later.

I have kept the units in throughout, for the obvious reason.

0 likes in reply to #3 17mo

r.coelhoTL212 Mar 2025#9

This follows post #6 rather than contradicting it.

STEP 4: I have looked for the primary source twice and failed twice. Either it does not exist or it is somewhere I do not know to look, and I would like to know which.

15 likes 17mo

MJayawardenaTL3Regular15 Mar 2025#10

Careful with the language on STEP 4. "Not detected" and "not present" are different findings and the first is a statement about the method.

29 likes 16mo

s.lindqvistTL217 Mar 2025#11

Post #7 put the caveat in the right place and I want to underline it.

Sessions on negative or null results are consistently the most instructive and the hardest to get anyone to attend.

The short version is the first sentence; the rest is why.

0 likes 16mo

s.grigorescuTL2Member20 Mar 2025#12

What I would want before treating STEP 4 as settled: the method, the sample, and whether anyone tried to find the opposite result. Two of the three are usually missing.

24 likes 16mo

g.tammTL222 Mar 2025 · edited#13

JFitzgibbon, post #2: The opening post describes the usual case. This is about the unusual one. Having read the whole STEP 4 thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead. Go to post

Understood. Thank you for being specific about the limits of it.

7 likes in reply to #2 16mo

figure_reviewTL2Member24 Mar 2025#14

Nobody has said the unglamorous part of STEP 4 yet, so: most of the variation is explained by things that are boring to write about and easy to check.

1 like 16mo

i.ilungaTL227 Mar 2025#15

Collapsed as off-topic by two members at trust level 3 or above

Where I part company with post #11, and it is a narrow parting.

Reading order that works for these sessions: registry entry, methods, baseline table, primary result, then abstract last. Reading the abstract first anchors everything that follows.

sleep_logTL2Regular29 Mar 2025#16

i.osei, post #5: Picking up post #2: that is the part I would want checked first. The figures usually contain the finding and the text usually contains the interpretation. Separating them for the first twenty minutes is a discipline worth keeping. Not disagreeing with anyone above, just adding the bit I keep having to look up. Go to post

Post #14 is the version of this I will quote in future. One addition.

Taking STEP 4 seriously for a moment rather than deflecting: the honest position is that the community has observations and no controlled comparison, and those two things support very different sentences.

33 likes in reply to #5 16mo

sa.rasmussenTL231 Mar 2025#17

STEP 1 (N Engl J Med 2021): The pivotal obesity trial for semaglutide and the reference point for most subsequent comparison. Mean weight reduction was substantially larger than anything previously achieved pharmacologically.

Somebody will have a better source than mine, and I hope they post it.

11 likes in reply to #1 16mo

first_vialTL1Member2 Apr 2025#18

STEP 4 is well covered in the tag pages, and the older discussions are better than the recent ones because they were argued out properly. Worth twenty minutes before adding to this one.

3 likes 16mo

an.ibarraTL25 Apr 2025#19

Small correction to my own earlier position on STEP 4. I had the units the wrong way round, which changes the conclusion by an order of magnitude and therefore changes it entirely.

1 like 16mo

formulary_notesTL3Regular7 Apr 2025#20

Confirming post #19 from a second method, which matters more than confirming it from a second person.

The most common failure mode is spending fifty minutes on the effect size and ten on the population. Reversing that ratio would improve most sessions.

0 likes in reply to #5 16mo

ak.kravchenkoTL29 Apr 2025#21

Reading order that works for these sessions: registry entry, methods, baseline table, primary result, then abstract last. Reading the abstract first anchors everything that follows.

0 likes 16mo

ambient_draftTL3Regular11 Apr 2025#22

Same experience here, different supplier, so it is at least not unique to one of them.

2 likes 16mo

s.lundgrenTL213 Apr 2025#23

The strongest argument against my own position on STEP 4, stated as well as I can state it, since nobody else has yet.

8 likes 15mo

LJankowiakTL3Regular15 Apr 2025#24

formulary_notes, post #20: Confirming post #19 from a second method, which matters more than confirming it from a second person. The most common failure mode is spending fifty minutes on the effect size and ten on the population. Reversing that ratio would improve most sessions. Go to post

A conflict-of-interest statement is worth reading aloud and is worth very little as a predictor. Design quality is what to spend the session on.

19 likes in reply to #20 15mo

h.kimaniTL217 Apr 2025#25

Adding the measurement that post #24 says would settle it.

Distinguishing three things in the STEP 4 discussion that keep getting used interchangeably: the observation, the proposed mechanism, and the recommendation that gets attached to both.

0 likes 15mo

a.schaefferTL2Member19 Apr 2025#26

Post #23 describes the usual case. This is about the unusual one.

What I would tell a new member reading about STEP 4 for the first time: the confident posts are not the reliable ones, and the reliable ones are longer.

0 likes 15mo

n.kirchnerTL221 Apr 2025#27

The baseline characteristics table is the single most useful page for the questions that get asked here, because it tells you who the result applies to.

5 likes in reply to #20 15mo

integrator_traceTL223 Apr 2025#28

Collapsed as off-topic by two members at trust level 3 or above

STEP 4 would be much easier to settle if anyone reported the denominator. Almost nobody reports the denominator.

e.mwangiTL225 Apr 2025#29

integrator_trace, post #28: STEP 4 would be much easier to settle if anyone reported the denominator. Almost nobody reports the denominator. Go to post

Critical appraisal template: (1) What did the trial set out to estimate? (2) Could the design answer that question? (3) Was the population sufficiently similar to your population to apply the results? (4) What was the absolute effect, not just the relative one? (5) What are the two strongest criticisms available?

It reads as pedantry until the day it does not.

28 likes in reply to #28 15mo

KTurkingtonTL3Regular27 Apr 2025#30

Trying to state the STEP 4 position in a way that someone who disagrees would recognise as fair, because I do not think the version in this thread passes that test.

0 likes 15mo

Sponsored PeptideMeter — peptide analytics and supplier verificationPurity analysis and structured supplier verification, published as dated records rather than scores. Referenced throughout our sourcing documentation.View analytics →