Criticising the method without criticising the authors Writing it up because I had to work it out twice and would rather nobody else did.
I was wrong about criticising the method without criticising in a thread last spring and I would like to correct it publicly rather than quietly.
The error was in the units, which changed the conclusion by an order of magnitude. Setting out the corrected version, and the way I now check for that class of mistake.
Worth separating two things that the opening post runs together.
I would be cautious about generalising from the criticising the method without criticising example above. It is a good example. It is one example.
Attrition is the failure mode most likely to invalidate a result and the least likely to be discussed. Differential attrition between arms is the specific thing to look for.
It is a small point and it changes the answer, which is an awkward combination.
Narrowing post #5, because the general version has more than one answer.
The claim about criticising the method without criticising upthread is stronger than its source supports. I have read the source. The source says "associated with" and the post says "causes".
A critique should end with what would change your mind. Without that it is a position rather than an argument.
An observation about criticising the method without criticising that I cannot explain and am posting anyway, on the principle that unexplained observations are more useful public than private.
I had written a reply contradicting post #5 and deleted it. Here is what survived.
The strongest critiques cite the section they rely on. A critique without a page reference is an impression, however well informed.
Reframing criticising the method without criticising slightly, because I think the disagreement is about the question rather than the answer. If the question is "does it happen", yes. If it is "how often", nobody here knows.
Post #9 answers the question as asked. The question underneath it is different.
Answering the criticising the method without criticising question as asked, then the question I think is meant. As asked: yes, with the qualification below. As meant: it depends on how the first measurement was taken.
I read post #8 twice before replying, because I had assumed the opposite.
Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.
I have no interest in any supplier named above.
Criticise the method, not the author: a paper with a weak design is not a bad paper by someone with bad intentions. It is a paper that answers a limited question. Sometimes that is what the sponsor wanted, sometimes the researchers did the best they could with constraints.
Small correction to my own earlier position on criticising the method without criticising. I had the units the wrong way round, which changes the conclusion by an order of magnitude and therefore changes it entirely.
Good question, well framed, and I would like to see it answered properly.
The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.
Defending a design that is being criticised unfairly is as useful as criticising one that deserves it, and it happens far less often.
If someone has run criticising the method without criticising properly I would rather read that than my own reconstruction of it. Posting mine only because the thread has gone quiet.
Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.
Distinguishing three things in the criticising the method without criticising discussion that keep getting used interchangeably: the observation, the proposed mechanism, and the recommendation that gets attached to both.
Generalisability: do the inclusion/exclusion criteria narrow the population so much that results do not apply to real people asking about it? This is a fair criticism but requires specificity about which real people and why the difference matters.
Written from notes rather than memory, which is why the numbers are specific.
Where I would push back on the criticising the method without criticising consensus is the confidence, not the direction. The direction looks right. The confidence is borrowed.
I had written a reply contradicting post #25 and deleted it. Here is what survived.
Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.
Worth one more sentence than it usually gets.
Confirming post #25 from a second method, which matters more than confirming it from a second person.
An honest declaration on criticising the method without criticising: I have a prior here and it is strong enough that you should weight what I say downward. Stating it rather than hiding it.
Narrowing post #29, because the general version has more than one answer.
The practical version of criticising the method without criticising is three sentences long. The rigorous version is three pages and reaches the same conclusion with the conditions attached.