Evidence · Study critique · continued

Criticising the method without criticising the authors posts 31–60

Evidence › Study critique disputed observational data confounding effect size

This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1.

owen.bradyTL4 Moderator13 Jan 2025#31

c.chowdhury, post #15: Small correction to my own earlier position on criticising the method without criticising. I had the units the wrong way round, which changes the conclusion by an order of magnitude and therefore changes it entirely. Go to post

Adding the measurement that post #29 says would settle it.

Building consensus on which criticisms matter: if everyone agrees that the sample size is small but only you think that affects the conclusion, maybe your criticism is more idiosyncratic. That does not make it wrong but it is worth noticing.

1 like in reply to #15 18mo

r.serranoTL217 Jan 2025#32

Post #30 describes the usual case. This is about the unusual one.

The reason criticising the method without criticising is hard to answer is that the obvious measurement and the relevant quantity are not the same thing, and substituting one for the other is silent.

7 likes 18mo

a.reyesTL4 Admin21 Jan 2025#33

Multiple comparisons without adjustment is a defensible choice when declared in advance and a problem when discovered afterwards. The registry entry settles which happened.

Posted with less confidence than the sentence structure implies.

18 likes 18mo

k.dahlbergTL224 Jan 2025#34

dexa_twice_yearly, post #9: I had written a reply contradicting post #5 and deleted it. Here is what survived. The strongest critiques cite the section they rely on. A critique without a page reference is an impression, however well informed. Go to post

Criticising the method without criticising: I would want to see the raw numbers rather than the summary before agreeing. Summaries lose exactly the information that would settle this.

0 likes in reply to #9 18mo

dr.villanuevaTL3Physician28 Jan 2025#35

j.sandvik, post #13: My experience of criticising the method without criticising contradicts the reply above. I am posting it as a data point rather than as a refutation, because one person's experience is exactly that. Go to post

A definition problem is doing most of the work in this criticising the method without criticising discussion. Once the term is pinned down I suspect the disagreement mostly goes away and what is left is small.

4 likes in reply to #13 18mo

s.grimaldiTL21 Feb 2025 · edited#36

Helpful, and short, which on this subject is harder than long.

12 likes 18mo

ms_hollowayTL4Mass spectrometrist5 Feb 2025#37

Post #35 is right about the mechanism and I think understates the practical bit.

Multiple comparisons: if a paper reports many outcomes, the chance of a spurious association by random chance is real. Pre-specification of primary outcomes matters and secondary analyses are weaker evidence.

25 likes 18mo

j.erdoganTL29 Feb 2025#38

Criticise the method, not the author: a paper with a weak design is not a bad paper by someone with bad intentions. It is a paper that answers a limited question. Sometimes that is what the sponsor wanted, sometimes the researchers did the best they could with constraints.

0 likes 18mo

priorauth_notesTL2Regular13 Feb 2025#39

Something worth flagging about criticising the method without criticising: the strongest-sounding claims in this thread are the ones with no source attached, which is the usual pattern and not a coincidence.

0 likes 17mo

m.kjaerTL216 Feb 2025#40

The strongest critiques cite the section they rely on. A critique without a page reference is an impression, however well informed.

That is the practical version. The rigorous version is longer and says the same thing.

2 likes 17mo

crossref_checkTL320 Feb 2025#41

Collapsed as off-topic by two members at trust level 3 or above

Answering the question post #39 raises rather than the one it answers.

Agreed on criticising the method without criticising, with one qualification that I think matters. The reasoning holds for the case as described. Change the starting assumption and it does not, and the starting assumption is the part nobody states.

f.danquahTL224 Feb 2025#42

The arithmetic in post #39 is right; the assumption feeding it is the part to check.

Criticism is more useful when it is narrower. "The trial answers a different question from the one being asked" is actionable; "the trial is flawed" is not.

29 likes 17mo

c.okaforTL3Regular28 Feb 2025 · edited#43

Taking criticising the method without criticising seriously for a moment rather than deflecting: the honest position is that the community has observations and no controlled comparison, and those two things support very different sentences.

10 likes 17mo

s.girardTL23 Mar 2025#44

Defending a design that is being criticised unfairly is as useful as criticising one that deserves it, and it happens far less often.

If anyone has run this properly I would rather read that than my own guess.

3 likes 17mo

r.jhannsdttirTL3Regular7 Mar 2025#45

n.boateng, post #22: Seconded. It reads as careful rather than confident, which is the right register. Go to post

Everything in post #43 holds. The case it does not cover is the one I have.

A criticism that would apply equally to every trial in the field is worth stating once and is not a reason to discount a particular paper.

0 likes in reply to #22 17mo

v.bergstromTL211 Mar 2025#46

Two people in this thread mean different things by criticising the method without criticising and are disagreeing about the definition while believing they are disagreeing about the facts. Worth pausing to define it.

0 likes 17mo

VPoulsenTL3Regular14 Mar 2025#47

A methods point on criticising the method without criticising rather than a substantive one: if the comparison is not like for like, the difference you are measuring is the difference in method.

15 likes 16mo

n.duarteTL218 Mar 2025#48

priorauth_notes, post #39: Something worth flagging about criticising the method without criticising: the strongest-sounding claims in this thread are the ones with no source attached, which is the usual pattern and not a coincidence. Go to post

Adding a data point of agreement rather than a data point.

5 likes in reply to #39 16mo

h.almeidaTL2Member22 Mar 2025#49

Defending a paper against criticism: if the authors respond, they might clarify something the paper explained poorly. Their response might also miss your point. Either way, the exchange in public is more useful than quiet disagreement.

31 likes in reply to #9 16mo

g.oyelaranTL225 Mar 2025#50

VPoulsen, post #47: A methods point on criticising the method without criticising rather than a substantive one: if the comparison is not like for like, the difference you are measuring is the difference in method. Go to post

Nobody has said the unglamorous part of criticising the method without criticising yet, so: most of the variation is explained by things that are boring to write about and easy to check.

15 likes in reply to #47 16mo

c.vermeulenTL229 Mar 2025#51

r.nakamura, post #26: Where I would push back on the criticising the method without criticising consensus is the confidence, not the direction. The direction looks right. The confidence is borrowed. Go to post

Post #50 answers the question as asked. The question underneath it is different.

The documentation on criticising the method without criticising is better than this thread and I say that as someone who has posted in the thread.

1 like in reply to #26 16mo

m.silvaTL21 Apr 2025#52

I read post #49 twice before replying, because I had assumed the opposite.

Surrogate endpoints are not automatically bad and their validity is compound-specific and population-specific. The question is whether this surrogate has been validated for this use.

I have written this out at length because the short version keeps being misread.

6 likes 16mo

v.salgadoTL25 Apr 2025#53

Criticising the method without criticising has a well-known answer and a correct answer, and the interesting work is establishing that they are the same. Nobody has done that here yet.

22 likes 16mo

lu.cabreraTL28 Apr 2025#54

Post hidden by community flags

Staff rationale: Hidden by community flags. The claim about a named supplier was not accompanied by a batch, a date, a method or a document, which R6 requires.

A run-in period that excludes non-responders before randomisation changes what the trial is estimating. It is legitimate design and it must be stated in any summary.

The corresponding entry is in the public moderation log. Hidden posts are never deleted.

j.sorensenTL212 Apr 2025#55

Collapsed as off-topic by two members at trust level 3 or above

Narrowing post #53, because the general version has more than one answer.

My understanding of criticising the method without criticising is a few years old and may have been superseded. If it has been, I would genuinely like to know rather than keep repeating it.

j.castellanosTL215 Apr 2025#56

Practical experience of criticising the method without criticising, offered as one case with the conditions stated, not as a general finding. Conditions first, because they are what make it interpretable.

3 likes 15mo

s.coelhoTL219 Apr 2025#57

Hold a trial to the standard something could actually have met. A criticism that no achievable design could have answered is a criticism of the field rather than of the paper.

16 likes 15mo

j.mwangiTL4 Moderator22 Apr 2025#58

Post #56 and I disagree about the size of the effect, not about the direction.

A critique should end with what would change your mind. Without that it is a position rather than an argument.

31 likes 15mo

e.kuuselaTL226 Apr 2025#59

Adding thanks rather than a view. I do not have a view worth the space.

0 likes 15mo

chromatogramTL4Analytical chemist29 Apr 2025#60

Where the criticising the method without criticising discussion usually stalls is that nobody wants to say "I do not know" and everyone is willing to say "it varies". Those are the same sentence with different clothes on.

1 like 15mo