Posting this under the heading it deserves: Second pass at: Trial registration and comparing the protocol with the paper Everything below is what sits behind that.
Two things I would like separated before anyone answers on Trial registration, because they get bundled and then argued about as one thing.
The first is descriptive: what has actually been observed, by whom, and how. The second is causal: why. I am asking about the first only.
This settles it for me, at least until somebody posts a reason it should not.
Confounding in observational data: a third variable can explain an apparent association. In a randomised trial, randomisation balances unknown confounders. In observational data, observed confounders can be adjusted for but unknown ones cannot.
Where I part company with post #3, and it is a narrow parting.
Effect sizes in a trial population reflect adherence achieved under trial conditions, which is generally better than adherence outside them.
The figure that circulates in coverage is almost always whichever estimand gives the larger effect. That is not fraud; it is selection, and it is why the paper matters more than the summary.
Building on post #4 rather than restating it.
Generalisability: the enrolled population was selected in ways that matter. Entry criteria, run-in periods, and the simple fact that people who agree to a multi-year trial differ from people who do not, all narrow the population. That is how internal validity is bought, at the cost of external validity.
Post #6 put the caveat in the right place and I want to underline it.
Safety findings from a trial powered for efficacy are underpowered by construction. Absence of a signal in that setting is weak evidence of absence.
Scoping that to what I have actually seen rather than what I have read.
Worth separating two things that post #6 runs together.
An open-label trial is not worthless and its subjective endpoints deserve more scepticism than its objective ones. That is a graded judgement rather than a verdict.
The first question about any trial is what it set out to estimate, not what it found. Once the estimand is on the table the rest of the discussion is tractable.
Post #10 is the version of this I will quote in future. One addition.
A comparator at less than its maximum licensed dose changes what a head-to-head result means. It does not invalidate the trial; it narrows the claim the trial supports.
The interesting part of this is the exception, and I do not understand the exception.
Nothing to add, except that this is the answer I would give if asked.
Post #14 put the caveat in the right place and I want to underline it.
Multiplicity and multiple comparisons: if a trial tests many hypotheses, the chance of a false positive on at least one by random chance increases. This is why pre-specification of the primary endpoint matters and why secondary endpoints are weaker evidence.
The evidence for this is thinner than the way I have phrased it suggests.
Building on post #14 rather than restating it.
Funding and trial conduct should be stated and are a weak predictor of anything on their own. Design quality is the stronger signal and it is checkable.
On reflection I would soften that slightly.
The estimand: what the trial set out to estimate. Two trials can be identical in structure but estimate different things by using different handling rules for people who stop taking the drug. Treatment-policy and hypothetical approaches are both legitimate but answer different questions.
Coming back to post #17, because the follow-up matters more than the original answer.
Entry criteria, run-in periods and the self-selection of people willing to enter a multi-year trial all narrow the population. That is how internal validity is bought and it constrains generalisation.
If it helps: the failure mode here is usually boring rather than dramatic.
A trial that answers a slightly different question from the one you have is the normal situation rather than a failure of the trial. The skill is describing the gap precisely.
I have changed my mind on this once already, so take it as current rather than settled.
The estimand: what the trial set out to estimate. Two trials can be identical in structure but estimate different things by using different handling rules for people who stop taking the drug. Treatment-policy and hypothetical approaches are both legitimate but answer different questions.
Effect sizes in a trial population reflect adherence achieved under trial conditions, which is generally better than adherence outside them.
Narrowing post #22, because the general version has more than one answer.
A comparator at less than its maximum licensed dose changes what a head-to-head result means. It does not invalidate the trial; it narrows the claim the trial supports.
That is one dataset and I would not build a rule on it.
Everything in post #20 holds. The case it does not cover is the one I have.
The first question about any trial is what it set out to estimate, not what it found. Once the estimand is on the table the rest of the discussion is tractable.
Worth reading the earlier posts in this thread before acting on mine.
Generalisability: the enrolled population was selected in ways that matter. Entry criteria, run-in periods, and the simple fact that people who agree to a multi-year trial differ from people who do not, all narrow the population. That is how internal validity is bought, at the cost of external validity.
Marking that as an opinion rather than a finding.
Worth separating two things that post #24 runs together.
Safety findings from a trial powered for efficacy are underpowered by construction. Absence of a signal in that setting is weak evidence of absence.
That is what the documentation says. What happens in practice is usually close.
Post #26 answers the question as asked. The question underneath it is different.
Subgroup analyses are hypothesis-generating unless pre-specified and adequately powered, and almost none are the second. The interaction test matters more than the subgroup point estimate.
A single observation, in a thread that deserves better than single observations.
Noted, and I have changed what I was going to do on the strength of it.
Picking up post #26: that is the part I would want checked first.
Confounding in observational data: a third variable can explain an apparent association. In a randomised trial, randomisation balances unknown confounders. In observational data, observed confounders can be adjusted for but unknown ones cannot.
I would be interested in a counterexample if anyone has one.
Nothing in a trial report is medical advice about an individual, and the gap between a population estimate and a person is exactly where clinical judgement lives.
That holds for the case as described. Change the assumptions and it may not.