Journal club: STEP 1 and the treatment-policy estimand — does this still hold? posts 31–60
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
On STEP 1 I would separate what is worth knowing from what is worth acting on. The first list is long and the second is short, and conflating them is how threads get heated.
I had written a reply contradicting post #29 and deleted it. Here is what survived.
The honest answer on STEP 1 is that it depends, and the useful part is the list of what it depends on. Four items, in rough order of how much they matter.
Most people get the first two right and then argue about the fourth.
Where the group cannot agree, record the disagreement rather than resolving it by seniority. The recorded disagreement is more honest and more useful later.
The short answer was in the first line; everything after is the working.
Picking up post #33: that is the part I would want checked first.
The baseline characteristics table is the single most useful page for the questions that get asked here, because it tells you who the result applies to.
Someone will know this better than I do and I hope they say so.
Worth stating the null on STEP 1 before we explain it: the observation may be nothing. That possibility deserves a sentence and usually does not get one.
Session format: read the paper before posting. The discussion is much better when everyone has. Start with the estimand and population, then methods, then results, then limitations. That order makes critique coherent.
Worth separating two things that post #37 runs together.
Small methodological point on STEP 1: repeating a measurement is cheap and resolves most of what is being argued about here at no cost to anyone.
This follows post #37 rather than contradicting it.
STEP 1 looks different depending on whether you are reading the primary literature or the summaries of it, and the difference is not in our favour.
I had written a reply contradicting post #40 and deleted it. Here is what survived.
Sessions on negative or null results are consistently the most instructive and the hardest to get anyone to attend.
I would rather say I do not know than round it up to an answer.
Picking up post #42: that is the part I would want checked first.
What I would want before treating STEP 1 as settled: the method, the sample, and whether anyone tried to find the opposite result. Two of the three are usually missing.
Reading back through the STEP 1 threads from last year, the same three questions come up every time and only one of them has ever been answered properly. That seems like a documentation gap rather than a knowledge gap.
On STEP 1: the maintained page in the documentation commons covers the general case with citations and a review date, which is more reliable than any reply here including this one.
Reading back through, this was answered upthread and I missed it. My fault.
Post #44 describes the usual case. This is about the unusual one.
STEP 1 (N Engl J Med 2021): The pivotal obesity trial for semaglutide and the reference point for most subsequent comparison. Mean weight reduction was substantially larger than anything previously achieved pharmacologically.
I would put a moderate confidence on that and no more.
Post #46 answers the question as asked. The question underneath it is different.
Adding the boring version of STEP 1, because the interesting version keeps getting posted and the boring one is usually right.
Check the ordinary explanations, in order, and stop when one of them accounts for what you are seeing. Most of the time the second one does.
Helpful, and short, which on this subject is harder than long.
Reading order that works for these sessions: registry entry, methods, baseline table, primary result, then abstract last. Reading the abstract first anchors everything that follows.
Second-hand, so weight it accordingly.
Trying to state the STEP 1 position in a way that someone who disagrees would recognise as fair, because I do not think the version in this thread passes that test.
STEP 1 is worth one more sentence than it usually gets, and the sentence is the one about how the number was arrived at.
The baseline characteristics table is the single most useful page for the questions that get asked here, because it tells you who the result applies to.
Caveat: everything above assumes the paperwork is what it says it is.
Having read the whole STEP 1 thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead.
Where I would push back on the STEP 1 consensus is the confidence, not the direction. The direction looks right. The confidence is borrowed.
Collapsed as off-topic by two members at trust level 3 or above
Picking up post #55: that is the part I would want checked first.
Session format: read the paper before posting. The discussion is much better when everyone has. Start with the estimand and population, then methods, then results, then limitations. That order makes critique coherent.
I am confident about the direction and much less about the magnitude.
An honest declaration on STEP 1: I have a prior here and it is strong enough that you should weight what I say downward. Stating it rather than hiding it.
Adding thanks rather than a view. I do not have a view worth the space.