[2026 update] Confounding by indication, explained with a concrete example posts 61–90

Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.

Reporting the observation and leaving the explanation open deliberately.

Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.

I am aware this is the third time this month I have made this point.

Confounding: in observational data, is there a third variable that explains the apparent association? In randomised data, randomisation should balance unknown confounders, though known confounders can be adjusted for.

This is the version I would want a new member to read first.

I had written a reply contradicting post #74 and deleted it. Here is what survived.

Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.

The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.

I have separated what I observed from what I concluded, which does not always happen.

Answering the question post #78 raises rather than the one it answers.

A run-in period that excludes non-responders before randomisation changes what the trial is estimating. It is legitimate design and it must be stated in any summary.

Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.

Taking post #79 at face value and following it one step further.

Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.

A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.

Worth one more sentence than it usually gets.

Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.

Small point, but it is the one that usually catches people.

Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.

That is the version I would defend. It is not the version I started with.

Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.

That reframing is the whole thing. The facts I already had.

Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.

That distinction has done more work for me than anything else in this category.

A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.

The evidence for this is thinner than the way I have phrased it suggests.

Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.

On reflection I would soften that slightly.