Topic summary

Criticising the method without criticising the authors

Evidence › Study critique disputed observational data confounding effect size risk of bias

This is a generated summary. It shows the 9 most-liked posts from a topic of 62, in their original order, with the accepted answer included where one exists. It is a reading aid and it will miss nuance — the full topic is the record.

ka.batistaTL212 Aug 2024#1

Criticising the method without criticising the authors Writing it up because I had to work it out twice and would rather nobody else did.

I was wrong about criticising the method without criticising in a thread last spring and I would like to correct it publicly rather than quietly.

The error was in the units, which changed the conclusion by an order of magnitude. Setting out the corrected version, and the way I now check for that class of mistake.

30 likes 2y

j.moreauTL27 Sep 2024#4

Multiple comparisons without adjustment is a defensible choice when declared in advance and a problem when discovered afterwards. The registry entry settles which happened.

31 likes 23mo

FConsidineTL1Member31 Oct 2024#14

Criticise the method, not the author: a paper with a weak design is not a bad paper by someone with bad intentions. It is a paper that answers a limited question. Sometimes that is what the sponsor wanted, sometimes the researchers did the best they could with constraints.

25 likes 21mo

j.rasmussenTL2Regular19 Nov 2024#18

The pre-specified endpoint being a weaker proxy than you would like is a real criticism. It is a smaller one than saying the result was chosen after the fact.

33 likes 20mo

n.boatengTL26 Dec 2024#22

Seconded. It reads as careful rather than confident, which is the right register.

33 likes 20mo

ms_hollowayTL4Mass spectrometrist5 Feb 2025#37

Post #35 is right about the mechanism and I think understates the practical bit.

Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.

25 likes 18mo

f.danquahTL224 Feb 2025#42

The arithmetic in post #39 is right; the assumption feeding it is the part to check.

Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.

29 likes 17mo

h.almeidaTL2Member22 Mar 2025#49

dexa_twice_yearly, post #9: I had written a reply contradicting post #5 and deleted it. Here is what survived. The strongest critiques cite the section they rely on. A critique without a page reference is an impression, however well informed. Go to post

Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.

31 likes in reply to #9 16mo

j.mwangiTL4 Moderator22 Apr 2025#58

Post #56 and I disagree about the size of the effect, not about the direction.

A critique should end with what would change your mind. Without that it is a position rather than an argument.

31 likes 15mo

Read the full topic (62 posts)

Suggested topics

Topic	Participants	Replies	Views	Activity
About the Study critique category Evidence › Study critique risk of bias confounding observational data Pinned Wiki Structured criticism of a specific published study. Criticise the method, not the author. This post is a community wiki: any member at trust level 3 or above can edit it, and every edit is recorded with its…		4	114	16mo
A well-designed study with a badly written abstract Evidence › Study critique disputed effect size risk of bias observational data Posting this under the heading it deserves: A well-designed study with a badly written abstract Everything below is what sits behind that. Reading back through what has been written here about well-designed…	+117	128	7.2k	14mo
A structured critique template this community uses — a second dataset Evidence › Study critique observational data effect size confounding Solved On the subject in the title: A structured critique template this community uses — a second dataset Working notes rather than a conclusion. Posting a small dataset on structured critique template. It is mine,…	+47	51	16k	15h
Criticising the method without criticising the authors — one year on Evidence › Study critique confounding effect size disputed risk of bias Solved Closed On the subject in the title: Criticising the method without criticising the authors — one year on Working notes rather than a conclusion. A methods question rather than a substantive one, about Criticising…	+16	20	61k	11mo
A critique that turned out to be unfair, retracted by its author Evidence › Study critique confounding risk of bias effect size observational data Wiki Posting this under the heading it deserves: A critique that turned out to be unfair, retracted by its author Everything below is what sits behind that. Critique, and specifically the version of it that the…	+34	38	20k	17h

Related topics — sharing the tags disputed, observational data, confounding

Topic	Participants	Replies	Views	Activity
Aggregated purity results across four services, with caveats Data & Tools › Datasets observational data site feedback data table Posting this under the heading it deserves: Aggregated purity results across four services, with caveats Everything below is what sits behind that. Aggregated purity results across four, from the point of…		3	24k	9mo
Selection into a registry and what it does to the estimate Evidence › Study critique risk of bias disputed observational data Selection into a registry and what it does to the estimate — setting out what I have, and where I think it stops being reliable. Working notes on selection rather than a conclusion. I would rather post the…		2	55k	7mo
Journal club: STEP 3 and the intensive behavioural therapy floor Evidence › Journal club randomised trial journal club effect size risk of bias Posting this under the heading it deserves: Journal club: STEP 3 and the intensive behavioural therapy floor Everything below is what sits behind that. Two things I would like separated before anyone answers…	+6	10	13k	28d
What I would record differently if I started again Outcomes › Progress reports DXA long read data table effect size Closed Asking directly, because I could not find a straight answer: What I would record differently if I started again Asking about the phase almost nobody writes about. The material here on losing is good. The…		2	12k	5mo
Tirzepatide in obstructive sleep apnoea: reading SURMOUNT-OSA Compounds › Tirzepatide SURMOUNT programme tirzepatide randomised trial effect size Posting this under the heading it deserves: Tirzepatide in obstructive sleep apnoea: reading SURMOUNT-OSA Everything below is what sits behind that. Reading back through what has been written here about…	+4	8	14k	8mo