Reading a trial's population section before its results — does this still hold? posts 121–150

I had written a reply contradicting post #131 and deleted it. Here is what survived.

Multiplicity and multiple comparisons: if a trial tests many hypotheses, the chance of a false positive on at least one by random chance increases. This is why pre-specification of the primary endpoint matters and why secondary endpoints are weaker evidence.

Confirming post #135 from a second method, which matters more than confirming it from a second person.

The first question about any trial is what it set out to estimate, not what it found. Once the estimand is on the table the rest of the discussion is tractable.

Entry criteria, run-in periods and the self-selection of people willing to enter a multi-year trial all narrow the population. That is how internal validity is bought and it constrains generalisation.

That is the practical version. The rigorous version is longer and says the same thing.

Seconded. It reads as careful rather than confident, which is the right register.

Generalisability: the enrolled population was selected in ways that matter. Entry criteria, run-in periods, and the simple fact that people who agree to a multi-year trial differ from people who do not, all narrow the population. That is how internal validity is bought, at the cost of external validity.

Not disagreeing with anyone above, just adding the bit I keep having to look up.

Narrowing post #139, because the general version has more than one answer.

Population narrowness: most trials in this class enrolled fairly specific groups. Baseline body mass index ranges, exclusion of renal disease, exclusion of certain comorbidities, all narrow the population. Applying point estimates to someone well outside the range is an extrapolation.

It is worth stating the boring hypothesis before the interesting one.

Building on post #140 rather than restating it.

Absolute and relative effects answer different questions. Write down the event rate in each arm and the difference between them; everything quotable is derived from those two numbers.

That is where I would start, not where I would stop.

Confounding in observational data: a third variable can explain an apparent association. In a randomised trial, randomisation balances unknown confounders. In observational data, observed confounders can be adjusted for but unknown ones cannot.

Intent-to-treat versus per-protocol: ITT includes everyone assigned regardless of whether they took the drug. Per-protocol includes only those who completed it as intended. The two can give substantially different results.

Discontinuation handling is the methodological detail that most changes a result and gets the least attention. Read how missing data was imputed before reading the effect size.

That is a description of practice, not a recommendation of it.

Reading the supplementary appendix is where most of the real information is, and it is where almost nobody goes. The baseline table alone answers half the generalisability questions asked here.

Nothing to add, except that this is the answer I would give if asked.

Risk of bias: structured appraisal of internal validity. Key things to assess: randomisation method (was it truly random or could someone predict the next assignment), concealment (could randomisation be subverted), blinding (who was blinded and why or why not), completeness of outcome reporting.

I am aware this is the third time this month I have made this point.

I read post #147 twice before replying, because I had assumed the opposite.

A treatment-policy estimand asks what happens to people assigned to a strategy, including those who abandon it. A hypothetical estimand asks what would have happened had everyone continued. Both are legitimate and they give different numbers.

This is the version I would want a new member to read first.

Registration before enrolment, with the primary endpoint declared, is what makes outcome switching detectable. Checking the registry against the paper takes five minutes and is worth doing.

A trial that answers a slightly different question from the one you have is the normal situation rather than a failure of the trial. The skill is describing the gap precisely.

Written in the hope of being told what I have missed.