Posting this under the heading it deserves: A critique that turned out to be unfair, retracted by its author Everything below is what sits behind that.
Critique, and specifically the version of it that the documentation does not cover. The maintained page handles the general case well and stops exactly where my question starts.
Setting out the gap in case it is a gap in the page rather than a gap in what is known.
On the opening post — agreed on the reasoning, with one qualification.
Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.
Confirming the opening post from a second method, which matters more than confirming it from a second person.
Where a critique rests on a subgroup, check whether the interaction was tested. A difference between subgroups that is not statistically supported is not a difference.
When you change your mind: if a reply convinces you that your criticism was not well-founded, say so plainly. The critique might still be real but smaller than you originally thought. That is not a failure — it is how discussion works.
Posting it because the silence on this was starting to look like agreement.
That is a cleaner way of putting what I was circling around.
I read the earlier replies on critique twice before writing this, because I had assumed the opposite and wanted to be sure I was disagreeing with what was said rather than what I expected.
Worth stating the null on critique before we explain it: the observation may be nothing. That possibility deserves a sentence and usually does not get one.
Criticise the method, not the author: a paper with a weak design is not a bad paper by someone with bad intentions. It is a paper that answers a limited question. Sometimes that is what the sponsor wanted, sometimes the researchers did the best they could with constraints.
Adding a source would improve this post and I do not have one to hand.
Adding a small correction to the critique summary above rather than a disagreement with it. The substance holds; one of the figures is out by a factor that matters.
Post #9 and I disagree about the size of the effect, not about the direction.
Posting my critique numbers with the method attached so they can be discounted properly. Uncontrolled, unblinded, and collected by someone who wanted a particular answer.
Taking post #13 at face value and following it one step further.
Where a critique changes your position, saying so in the thread is worth more than being right, and it is the thing this subcategory does better than most places.
The step people skip is the one I have spelled out.
On critique the community has more anecdote than the confidence in this thread implies, and I include my own contribution in that.
Building on post #17 rather than restating it.
Two questions I would want answered before drawing anything from the critique data above: how were the cases selected, and what happened to the ones that dropped out.
The arithmetic in post #17 is right; the assumption feeding it is the part to check.
Blinding, duration and comparator choice trade against each other. Improving one usually costs another, and a critique that ignores the trade is incomplete.
The strength of my opinion here exceeds the strength of my evidence.
Confirming post #18 from a second method, which matters more than confirming it from a second person.
Where a critique changes your position, saying so in the thread is worth more than being right, and it is the thing this subcategory does better than most places.
If anyone has run this properly I would rather read that than my own guess.
Where I part company with post #20, and it is a narrow parting.
Adding what did not work for me on critique, since the failures never get written up and they are half the useful information.
The claim about critique upthread is stronger than its source supports. I have read the source. The source says "associated with" and the post says "causes".
Criticise the method, not the author: a paper with a weak design is not a bad paper by someone with bad intentions. It is a paper that answers a limited question. Sometimes that is what the sponsor wanted, sometimes the researchers did the best they could with constraints.
Not a conclusion. A place to stand while looking for one.
Seconded. It reads as careful rather than confident, which is the right register.
I had written a reply contradicting post #24 and deleted it. Here is what survived.
Generalisability and validity are separate axes. A trial can be internally impeccable and still tell you nothing about the person asking.
Picking up post #26: that is the part I would want checked first.
Where a critique rests on a subgroup, check whether the interaction was tested. A difference between subgroups that is not statistically supported is not a difference.
One of those cases where knowing the mechanism does not help the decision.
When you change your mind: if a reply convinces you that your criticism was not well-founded, say so plainly. The critique might still be real but smaller than you originally thought. That is not a failure — it is how discussion works.
Two people can read the same figure differently here and both be reasonable.
I would be cautious about generalising from the critique example above. It is a good example. It is one example.