Revisiting: What pooling buys you and what it destroys posts 61–90
This is a continuation of a long topic, addressed by post number rather than by page. Start at post 1 · go to the accepted answer.
Where I part company with post #58, and it is a narrow parting.
Offering a way to settle pooling buys rather than another opinion about it. Two measurements, taken the same way, a fortnight apart. If the difference is within the noise, the question was not answerable at this precision.
The arithmetic on pooling buys is the easy part and it is where the errors are, which is an uncomfortable combination. Show your working and someone will catch it.
Individual participant data pooling is a much stronger design than aggregate pooling and is rare because it requires cooperation rather than a search.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Everything in post #66 holds. The case it does not cover is the one I have.
A note on how pooling buys gets discussed rather than on pooling buys itself: the confident posts get the replies and the careful ones get ignored, and the careful ones have been right more often.
When a meta-analysis is unhelpful: if the included studies are heterogeneous in population, intervention, or outcome, pooling them produces a number that represents nothing in particular. Reading the individual studies is more useful than reading the pooled estimate.
It cost nothing to check and would have cost something not to.
Second-hand on pooling buys, so weight it accordingly — someone whose method I trust told me this and I have not verified it myself.
Narrowing post #69, because the general version has more than one answer.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
Same experience here, different supplier, so it is at least not unique to one of them.
Collapsed as off-topic by two members at trust level 3 or above
The bit of pooling buys that nobody enjoys is that the answer changes depending on what you are trying to decide with it. Say what the decision is and the thread will converge.
Reframing pooling buys slightly, because I think the disagreement is about the question rather than the answer. If the question is "does it happen", yes. If it is "how often", nobody here knows.
The arithmetic in post #75 is right; the assumption feeding it is the part to check.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
It is the sort of thing that seems obvious in retrospect and was not at the time.
Pooling buys has a well-known answer and a correct answer, and the interesting work is establishing that they are the same. Nobody has done that here yet.
Post #77 describes the usual case. This is about the unusual one.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
That is what I would do. It may not be what is correct.
I read post #80 twice before replying, because I had assumed the opposite.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
For what it is worth, the same held on the two occasions I checked.
Taking post #82 at face value and following it one step further.
Where the included trials share a sponsor and a protocol template, their errors correlate and pooling does not average them out.
If the premise is wrong, everything after it is decoration.
Where the pooling buys discussion usually stalls is that nobody wants to say "I do not know" and everyone is willing to say "it varies". Those are the same sentence with different clothes on.
Worth separating two things that post #84 runs together.
The honest answer on pooling buys is that it depends, and the useful part is the list of what it depends on. Four items, in rough order of how much they matter.
Most people get the first two right and then argue about the fourth.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
The general case is well covered; this is the awkward specific one.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.