Posting this under the heading it deserves: A well-designed study with a badly written abstract Everything below is what sits behind that.
Reading back through what has been written here about well-designed study, three questions come up every time and only one has ever been answered properly.
Listing all three, with what I think the state of the answer is for each, so the thread can start further along than the last one did.
Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.
On well-designed study, I would rather understate and be corrected upward than overstate and be quoted. That is a house style here and it is a good one.
Narrowing post #4, because the general version has more than one answer.
The number people quote for well-designed study is a central estimate presented without its interval, and the interval is wide enough that the estimate is nearly uninformative on its own.
This is the answer, and the reason it is the answer is the more useful part.
Post #7 is right about the mechanism and I think understates the practical bit.
Practical answer on well-designed study, since the theoretical one is upthread: do the simplest check first, write down the result, and only then decide whether the complicated explanation is needed. It usually is not.
Coming back to post #6, because the follow-up matters more than the original answer.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
Worth reading the earlier posts in this thread before acting on mine.
Where the well-designed study reasoning breaks down for me is the step from the group result to the individual case. That step is almost never argued for.
Worth separating two things that post #9 runs together.
Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.
Filing this under things that are true until someone shows me otherwise.
This follows post #13 rather than contradicting it.
A note on scope: what I am saying about well-designed study applies to the case in the first post and I would not extend it further without checking.
The strongest argument against my own position on well-designed study, stated as well as I can state it, since nobody else has yet.
Well-designed study would be much easier to settle if anyone reported the denominator. Almost nobody reports the denominator.
Narrowing post #18, because the general version has more than one answer.
I changed my mind about well-designed study after someone here asked me for the source and I could not produce one. That is worth saying out loud because it is the ordinary way it happens.
Where I part company with post #21, and it is a narrow parting.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
I have three months of notes on well-designed study and the honest summary is that the trend is real and the week-to-week numbers are noise. I nearly drew the opposite conclusion from the first fortnight.
Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.
Happy to be corrected if someone holds better data than mine.
Building on post #22 rather than restating it.
Reporting bias within a paper — a result in a figure that is absent from the abstract — is checkable and is worth checking when the claim matters.
Good question, well framed, and I would like to see it answered properly.
Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.
Two sources, same conclusion, and I could not rule out that one copied the other.
Offering a way to settle well-designed study rather than another opinion about it. Two measurements, taken the same way, a fortnight apart. If the difference is within the noise, the question was not answerable at this precision.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
If the premise is wrong, everything after it is decoration.