This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
None of the above is medical advice and I am not qualified to give any.
Understood. Thank you for being specific about the limits of it.
Taking post #90 at face value and following it one step further.
Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.
Same conclusion as the reply above, reached differently, which is mildly reassuring.
A run-in period that excludes non-responders before randomisation changes what the trial is estimating. It is legitimate design and it must be stated in any summary.
If anyone can point at the primary source I would be grateful.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
The claim is narrower than it sounds, and deliberately so.
Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.
That holds for the case as described. Change the assumptions and it may not.
This follows post #94 rather than contradicting it.
Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.
Worth separating two things that post #96 runs together.
The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.
Anyone who has looked at this more carefully, please correct the record.
Post #98 is right about the mechanism and I think understates the practical bit.
Attrition is the failure mode most likely to invalidate a result and the least likely to be discussed. Differential attrition between arms is the specific thing to look for.
Coming back to post #96, because the follow-up matters more than the original answer.
Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.
A guess, clearly labelled as one.
That is a fair summary of where the discussion has got to.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
The arithmetic in post #102 is right; the assumption feeding it is the part to check.
The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.
Answering the question post #102 raises rather than the one it answers.
Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.
I am reporting what happened, not recommending it.
Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.
That is the version I use. It may not be the version that is correct.
Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.
That is my reading. Someone else read the same page differently and was reasonable.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
Post #106 describes the usual case. This is about the unusual one.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
Post #108 is right about the mechanism and I think understates the practical bit.
Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.
Scoping that to what I have actually seen rather than what I have read.
Coming back to post #106, because the follow-up matters more than the original answer.
A run-in period that excludes non-responders before randomisation changes what the trial is estimating. It is legitimate design and it must be stated in any summary.
Second-hand, so weight it accordingly.
A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.
Happy to be corrected if someone holds better data than mine.
Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.
Happy to expand any of that if it is the useful part.
Attrition is the failure mode most likely to invalidate a result and the least likely to be discussed. Differential attrition between arms is the specific thing to look for.
That much is documented. The rest is how I have interpreted it.
Picking up post #113: that is the part I would want checked first.
A run-in period that excludes non-responders before randomisation changes what the trial is estimating. It is legitimate design and it must be stated in any summary.
I am not the right person to answer the follow-up to this.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
Nothing above should be read as advice about what anyone else should do.
Post #113 describes the usual case. This is about the unusual one.
A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
A well-designed study with a badly written abstract
Posting this under the heading it deserves: A well-designed study with a badly written abstract Everything below is what sits behind that. Reading back through what has been written here about well-designed…
|
+117 | 128 | 7.2k | 14mo |
|
Coming back to: Measurement error in a self-reported exposure
Measurement error in a self-reported exposure — setting out what I have, and where I think it stops being reliable. Reading back through what has been written here about Measurement error, three questions…
|
+23 | 27 | 49k | 3mo |
|
A structured critique template this community uses — a second dataset
On the subject in the title: A structured critique template this community uses — a second dataset Working notes rather than a conclusion. Posting a small dataset on structured critique template. It is mine,…
|
+47 | 51 | 16k | 15h |
|
Coming back to: A critique that turned out to be unfair, retracted by its author
A critique that turned out to be unfair, retracted by its author — setting out what I have, and where I think it stops being reliable. Reading back through what has been written here about critique, three…
|
2 | 1.3k | 14mo | |
|
Immortal time bias in a claims-database study — one year on
Posting this under the heading it deserves: Immortal time bias in a claims-database study — one year on Everything below is what sits behind that. I would like to disagree carefully with the settled view on…
|
+41 | 46 | 3.7k | 22h |
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Journal club: SUSTAIN 6 and its retinopathy signal
On the subject in the title: Journal club: SUSTAIN 6 and its retinopathy signal Working notes rather than a conclusion. Reading SUSTAIN 6 ( N Engl J Med , 2016) for the population rather than the effect,…
|
+110 | 122 | 46k | 1d |
|
The "under review" status pill and when a moderator applies it — a second dataset
The "under review" status pill and when a moderator applies it — a second dataset Writing it up because I had to work it out twice and would rather nobody else did. A question about how this site should cite…
|
+50 | 54 | 1.4k | 11mo |
|
Secretagogues and glucose tolerance: the mechanistic concern — what changed since
Secretagogues and glucose tolerance: the mechanistic concern — what changed since Writing it up because I had to work it out twice and would rather nobody else did. What changes if the standard account of…
|
+2 | 6 | 31k | 2mo |
|
Journal club: SUSTAIN 6 and its retinopathy signal — does this still hold?
The question in the title: Journal club: SUSTAIN 6 and its retinopathy signal — does this still hold? I will give what I have already checked below so nobody repeats it. Reading SUSTAIN 6 ( N Engl J Med ,…
|
+60 | 66 | 12k | 16mo |
|
Revisiting: Filing a first vendor report: a worked example
On the subject in the title: Revisiting: Filing a first vendor report: a worked example Working notes rather than a conclusion. A question about Filing a first vendor that I think has a definite answer,…
|
2 | 60k | 11mo |