Posting this under the heading it deserves: Reading a meta-analysis you disagree with, fairly Everything below is what sits behind that.
I would like to disagree carefully with the settled view on reading a meta-analysis, and I would like to be argued out of it if the disagreement is bad.
The disagreement is about one step, not about the conclusion. If the step holds I withdraw it entirely.
Reading the pooled figure without reading the inclusion criteria is the equivalent of quoting a trial without its population. Both are common and both are avoidable.
That is a description of practice, not a recommendation of it.
Having read the whole reading a meta-analysis thread before replying: the question in the first post has not actually been answered yet, and three of us have answered a nearby one instead.
Reading a meta-analysis looks different depending on whether you are reading the primary literature or the summaries of it, and the difference is not in our favour.
Where the pooled result and the largest single trial disagree, that disagreement is the interesting thing rather than an inconvenience to be smoothed.
The interesting part of this is the exception, and I do not understand the exception.
A meta-analysis of four small trials is not stronger evidence than one adequately powered trial, whatever the summary statistic looks like.
I read post #6 twice before replying, because I had assumed the opposite.
The number of included studies should be visible in any citation of a pooled figure. "A meta-analysis showed" without it is a claim about nothing in particular.
Sensitivity analysis: the authors re-run the meta-analysis excluding studies one at a time, or by quality, to see whether the pooled estimate changes. Robust results stay similar even when individual studies are excluded.
If that reads as pedantic, it is, and it has saved me twice.
On reading a meta-analysis the community has more anecdote than the confidence in this thread implies, and I include my own contribution in that.
Inclusion and exclusion criteria: a meta-analysis is only as good as its inclusion criteria. If the criteria are too broad, apples and oranges get pooled. If they are too narrow, the meta-analysis answers a overly specific question.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
That much is documented. The rest is how I have interpreted it.
Staff rationale: Hidden by community flags. The substance was fine; the framing attributed motive rather than error, which is R1.
Where I part company with post #9, and it is a narrow parting.
Practical note on reading a meta-analysis: write down what you expect before you look. The number of times I have found what I went looking for is higher than chance would allow.
The corresponding entry is in the public moderation log. Hidden posts are never deleted.
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Publication bias detection and its low power
On the subject in the title: Publication bias detection and its low power Working notes rather than a conclusion. Working notes on publication bias detection rather than a conclusion. I would rather post the…
|
2 | 3.2k | 1d | |
|
Second pass at: A pooled estimate that changed when one trial was added
On the subject in the title: Second pass at: A pooled estimate that changed when one trial was added Working notes rather than a conclusion. Two things I would like separated before anyone answers on pooled…
|
+26 | 30 | 22k | 1mo |
|
Heterogeneity as information rather than as a nuisance
Posting this under the heading it deserves: Heterogeneity as information rather than as a nuisance Everything below is what sits behind that. An honest uncertainty about heterogeneity rather than a disguised…
|
+88 | 95 | 1.2k | 6mo |
|
Random versus fixed effects: choosing rather than defaulting
Random versus fixed effects: choosing rather than defaulting — setting out what I have, and where I think it stops being reliable. Random versus fixed effects, and specifically the version of it that the…
|
+16 | 20 | 22k | 13mo |
|
Risk-of-bias tools and the judgement they conceal — does this still hold?
Risk-of-bias tools and the judgement they conceal — does this still hold? — that is the question, and I have not found it answered plainly anywhere I have looked. Asking about Risk-of-bias tools on behalf of…
|
+26 | 30 | 5.8k | 23h |
| Topic | Participants | Replies | Views | Activity |
|---|---|---|---|---|
|
Follow-up: Individual participant data versus aggregate data
Individual participant data versus aggregate data — setting out what I have, and where I think it stops being reliable. A follow-up question about Individual participant data versus aggregate that I did not…
|
+101 | 111 | 23k | 18mo |
|
Second pass at: A pooled estimate that changed when one trial was added
On the subject in the title: Second pass at: A pooled estimate that changed when one trial was added Working notes rather than a conclusion. Two things I would like separated before anyone answers on pooled…
|
+26 | 30 | 22k | 1mo |
|
Journal club: semaglutide in MASH, and surrogate endpoints
Posting this under the heading it deserves: Journal club: semaglutide in MASH, and surrogate endpoints Everything below is what sits behind that. Working notes on semaglutide in MASH rather than a conclusion.…
|
+24 | 28 | 17k | 15h |
|
Sample size intuition for a personal experiment — does this still hold?
Sample size intuition for a personal experiment — does this still hold? I have a specific reason for asking rather than idle curiosity, and the context is below. Reading back through what has been written…
|
+33 | 38 | 26k | 23d |
|
How this community labels and discusses unrefereed work — the long version
Asking directly, because I could not find a straight answer: How this community labels and discusses unrefereed work — the long version Reading an unrefereed manuscript properly, and asking whether I am doing…
|
+18 | 23 | 21k | 2mo |