A well-designed study with a badly written abstract posts 91–120
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.
Collapsed as off-topic by two members at trust level 3 or above
Taking post #89 at face value and following it one step further.
What would change my mind on well-designed study is a second dataset collected by someone with no stake in the first. Until then I hold it loosely and I would rather say so than pretend to more.
Worth separating two things that post #93 runs together.
Well-designed study has been discussed here with more heat than it deserves, mostly because two definitions have been in play the whole time.
This follows post #93 rather than contradicting it.
A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.
I am reporting what happened, not recommending it.
Reporting bias within a paper — a result in a figure that is absent from the abstract — is checkable and is worth checking when the claim matters.
Flagging that the sources on this are thinner than the confidence in the thread suggests.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
I am aware this is the third time this month I have made this point.
Post #100 answers the question as asked. The question underneath it is different.
The bit of well-designed study that nobody enjoys is that the answer changes depending on what you are trying to decide with it. Say what the decision is and the thread will converge.
Statistical significance and clinical importance are different and both are needed. A significant difference below the minimal important difference is a real finding of no practical consequence.
This is the sort of thing the wiki should carry and currently does not.
Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.
Reading it again, the caveat matters more than the finding.
Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.
Adding a source would improve this post and I do not have one to hand.
Adding thanks rather than a view. I do not have a view worth the space.
On post #105 — agreed on the reasoning, with one qualification.
The arithmetic on well-designed study is the easy part and it is where the errors are, which is an uncomfortable combination. Show your working and someone will catch it.
My position on well-designed study is current rather than settled. I have revised it once already and I expect to again, so treat it accordingly.
I had written a reply contradicting post #105 and deleted it. Here is what survived.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
Scoping that to what I have actually seen rather than what I have read.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
Happy to expand any of that if it is the useful part.
That matches what I have seen, for whatever a single anecdote is worth.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
Picking up post #116: that is the part I would want checked first.
The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.
Marking that as an opinion rather than a finding.
On post #114 — agreed on the reasoning, with one qualification.
Well-designed study looks different depending on whether you are reading the primary literature or the summaries of it, and the difference is not in our favour.