Second pass at: Regression to the mean in progress reports Writing it up because I had to work it out twice and would rather nobody else did.
Posting a small dataset on Regression. It is mine, it is uncontrolled, and the method is stated so it can be discounted appropriately.
What I would like is not agreement but a second dataset collected by someone with no stake in mine. If one exists I would rather read it than argue for this one.
On Regression I would separate what is worth knowing from what is worth acting on. The first list is long and the second is short, and conflating them is how threads get heated.
Confirming the opening post from a second method, which matters more than confirming it from a second person.
P-values and significance: p<0.05 means the data would be surprising if the null hypothesis were true, not that the null hypothesis is false. A non-significant p-value does not mean "no effect".
The opening post describes the usual case. This is about the unusual one.
Checked the Regression claim against the primary source this morning. It survives, with a narrower scope than the version quoted here. Posting the narrower scope.
Adding the measurement that post #5 says would settle it.
Regression to the mean explains a large share of apparent improvements in anything selected for being extreme. It is not a statistical curiosity; it is the default explanation.
The honest answer on Regression is that it depends, and the useful part is the list of what it depends on. Four items, in rough order of how much they matter.
Most people get the first two right and then argue about the fourth.
Where an analysis was changed after seeing the data, the honest thing is to report both and say which was pre-specified.
Reporting the observation and leaving the explanation open deliberately.
Taking post #9 at face value and following it one step further.
Regression came up in a thread eighteen months ago and was answered well. I cannot find it, which is itself the problem, so here is the reconstruction.
Post #8 and I disagree about the size of the effect, not about the direction.
A standard deviation describes the spread of individuals and a standard error describes the precision of the mean. Quoting one where the other belongs changes the apparent result substantially.
Fine by me. I had wanted a stronger conclusion and there is not one available.
I would call the community position on Regression likely rather than established, and I would be comfortable defending that hedge.
Building on post #14 rather than restating it.
Medians and means diverge for skewed distributions, and most of the quantities discussed here are skewed. Which one a paper reports is a choice worth noticing.
What I can speak to on Regression is narrow, so I will keep it narrow rather than generalising from it. Beyond that boundary I do not know.
Everything in post #16 holds. The case it does not cover is the one I have.
A distribution shown is worth ten summary statistics. Where a paper shows individual data points, read those first.
Counterpoint on Regression, offered without confidence: the same observation is consistent with a much duller explanation, and nobody has ruled the dull one out.
The useful distinction on Regression is between what was measured and what was inferred from it. Both end up in the same sentence and only one of them has error bars.
Taking post #21 at face value and following it one step further.
Percentages of small denominators should be reported with the denominator. Two out of three is not sixty-seven per cent in any useful sense.
Understood, and I withdraw the assumption I opened with.
Coming back to post #25, because the follow-up matters more than the original answer.
Regression is one of those subjects where the general answer and the answer for a specific case diverge, and the thread will go in circles until someone says which one is being asked for.
Post #25 is right about the mechanism and I think understates the practical bit.
Practical note on Regression: write down what you expect before you look. The number of times I have found what I went looking for is higher than chance would allow.
Confounding: a third variable explains an apparent association. In randomised data, randomisation balances confounders. In observational data, confounders can be adjusted for but unknown ones cannot.
Effect sizes: the magnitude of a difference, not just whether it is statistically significant. A difference that is significant (p<0.05) might be too small to matter. A large effect might not be significant if sample size is small.
Reading it again, the caveat matters more than the finding.