Posting this under the heading it deserves: Heterogeneity as information rather than as a nuisance Everything below is what sits behind that.
An honest uncertainty about heterogeneity rather than a disguised assertion.
I do not know the answer and I have not been able to find one. What I have is the shape of the question, which may be worth more than my guess at the answer.
A note on how heterogeneity gets discussed rather than on heterogeneity itself: the confident posts get the replies and the careful ones get ignored, and the careful ones have been right more often.
This follows the opening post rather than contradicting it.
The I-squared statistic describes the proportion of variability not attributable to chance and is frequently read as a threshold. It is a description rather than a test.
Worth separating two things that post #2 runs together.
Fixed-effects versus random-effects models: fixed-effects assumes all studies are estimating the same thing and variation is sampling error. Random-effects assumes studies are estimating effects from different distributions and allows between-study variance. Choice matters if heterogeneity is high.
A single observation, in a thread that deserves better than single observations.
Coming back to post #2, because the follow-up matters more than the original answer.
Checked the heterogeneity claim against the primary source this morning. It survives, with a narrower scope than the version quoted here. Posting the narrower scope.
Picking up post #4: that is the part I would want checked first.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
That is what the documentation says. What happens in practice is usually close.
On heterogeneity I would separate what is worth knowing from what is worth acting on. The first list is long and the second is short, and conflating them is how threads get heated.
The arithmetic in post #8 is right; the assumption feeding it is the part to check.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
I am not the right person to answer the follow-up to this.
Quality assessment of included trials should change the analysis rather than sit beside it. A sensitivity analysis excluding the weakest studies is the minimum.
The mechanism is plausible, which is not the same as established.
Seconded. It reads as careful rather than confident, which is the right register.
Where I part company with post #9, and it is a narrow parting.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Post #13 is the version of this I will quote in future. One addition.
Answering the heterogeneity question as asked, then the question I think is meant. As asked: yes, with the qualification below. As meant: it depends on how the first measurement was taken.
High heterogeneity is the finding rather than a nuisance to be minimised. If the effect genuinely differs across settings, an average of those settings is an average of things that should not have been averaged.
That is the shape of it. The detail is where I would expect to be corrected.
Where I have landed on heterogeneity, having got it wrong once in public: the direction is clear, the magnitude is not, and anyone quoting a precise magnitude has borrowed it from somewhere that did not measure it.
Post #13 put the caveat in the right place and I want to underline it.
The question underneath heterogeneity is usually "how would I tell?" rather than "what is true?", and that one has a method attached to it.
Write down what you would expect to see under each hypothesis before you collect anything. If they predict the same observation, collecting it will not help.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
Not a conclusion. A place to stand while looking for one.
Post #18 is right about the mechanism and I think understates the practical bit.
Marking my uncertainty on heterogeneity explicitly. I am confident about the direction, much less confident about the size, and not confident at all that it generalises past the case in the first post.
Coming back to post #20, because the follow-up matters more than the original answer.
If someone has run heterogeneity properly I would rather read that than my own reconstruction of it. Posting mine only because the thread has gone quiet.
A methods point on heterogeneity rather than a substantive one: if the comparison is not like for like, the difference you are measuring is the difference in method.
Confirming post #22 from a second method, which matters more than confirming it from a second person.
Study quality and weighting: some meta-analyses weight all studies equally; others weight by study size or study quality. The choice affects the result and should be stated and justified.
Heterogeneity is well covered in the tag pages, and the older discussions are better than the recent ones because they were argued out properly. Worth twenty minutes before adding to this one.
Bookmarking this. I will come back when I have something worth adding.
Double extraction with disagreement resolution is standard and is worth checking for, because single extraction errors are common and non-random.
Nobody has said the unglamorous part of heterogeneity yet, so: most of the variation is explained by things that are boring to write about and easy to check.