What I would want before treating structured critique template as settled: the method, the sample, and whether anyone tried to find the opposite result. Two of the three are usually missing.
On the subject in the title: A structured critique template this community uses — a second dataset Working notes rather than a conclusion.
Posting a small dataset on structured critique template. It is mine, it is uncontrolled, and the method is stated so it can be discounted appropriately.
What I would like is not agreement but a second dataset collected by someone with no stake in mine. If one exists I would rather read it than argue for this one.
Acknowledging rather than arguing. The reasoning holds as far as I can follow it.
I had written a reply contradicting the opening post and deleted it. Here is what survived.
Structured critique template is well covered in the tag pages, and the older discussions are better than the recent ones because they were argued out properly. Worth twenty minutes before adding to this one.
The arithmetic in post #2 is right; the assumption feeding it is the part to check.
Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.
The reasoning is more useful than the number, which is why I have shown it.
Answering the question post #4 raises rather than the one it answers.
On structured critique template: the maintained page in the documentation commons covers the general case with citations and a review date, which is more reliable than any reply here including this one.
Answering the structured critique template question as asked, then the question I think is meant. As asked: yes, with the qualification below. As meant: it depends on how the first measurement was taken.
Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.
Agreed on structured critique template, with one qualification that I think matters. The reasoning holds for the case as described. Change the starting assumption and it does not, and the starting assumption is the part nobody states.
Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.
The strength of my opinion here exceeds the strength of my evidence.
On post #11 — agreed on the reasoning, with one qualification.
Where a critique changes your position, saying so in the thread is worth more than being right, and it is the thing this subcategory does better than most places.
Picking up post #11: that is the part I would want checked first.
My experience of structured critique template contradicts the reply above. I am posting it as a data point rather than as a refutation, because one person's experience is exactly that.
Nobody has said the unglamorous part of structured critique template yet, so: most of the variation is explained by things that are boring to write about and easy to check.
Confirming post #15 from a second method, which matters more than confirming it from a second person.
Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.
Post #15 describes the usual case. This is about the unusual one.
Reading back through the structured critique template threads from last year, the same three questions come up every time and only one of them has ever been answered properly. That seems like a documentation gap rather than a knowledge gap.
The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.
If this contradicts something upthread, the upthread version may well be the better one.
Post #18 is the version of this I will quote in future. One addition.
The strongest critiques cite the section they rely on. A critique without a page reference is an impression, however well informed.
Picking up post #20: that is the part I would want checked first.
Where a critique rests on a subgroup, check whether the interaction was tested. A difference between subgroups that is not statistically supported is not a difference.
I would hold that lightly until someone with a larger sample weighs in.
This settles it for me, at least until somebody posts a reason it should not.
A critique should end with what would change your mind. Without that it is a position rather than an argument.
For what it is worth, the same held on the two occasions I checked.
Fair, and the limits you put on it are the part I will remember.
Worth separating two things that post #26 runs together.
What I want from this structured critique template thread is the list of things that would need to be true for the claim to hold. If we can write that list, we can check it.
Narrowing post #28, because the general version has more than one answer.
Per-protocol and intention-to-treat analyses answer different questions and neither is the honest one by default. Reporting both is the practice worth insisting on.
I would want to see it done twice before believing it once.