Where I have landed on Sample size intuition, having got it wrong once in public: the direction is clear, the magnitude is not, and anyone quoting a precise magnitude has borrowed it from somewhere that did not measure it.
Sample size intuition for a personal experiment — does this still hold? I have a specific reason for asking rather than idle curiosity, and the context is below.
Reading back through what has been written here about Sample size intuition, three questions come up every time and only one has ever been answered properly.
Listing all three, with what I think the state of the answer is for each, so the thread can start further along than the last one did.
Building on the opening post rather than restating it.
Worth separating Sample size intuition as a question about the compound from Sample size intuition as a question about the documentation. They get answered by different people and only one of them is answerable here.
The most common statistical error in this community is not technical: it is treating a self-selected collection of reports as a sample from a population.
None of the above is medical advice and I am not qualified to give any.
The question underneath Sample size intuition is usually "how would I tell?" rather than "what is true?", and that one has a method attached to it.
Write down what you would expect to see under each hypothesis before you collect anything. If they predict the same observation, collecting it will not help.
Where I part company with the opening post, and it is a narrow parting.
Adding the boring version of Sample size intuition, because the interesting version keeps getting posted and the boring one is usually right.
Check the ordinary explanations, in order, and stop when one of them accounts for what you are seeing. Most of the time the second one does.
Where I have landed on Sample size intuition, having got it wrong once in public: the direction is clear, the magnitude is not, and anyone quoting a precise magnitude has borrowed it from somewhere that did not measure it.
Confirming post #10 from a second method, which matters more than confirming it from a second person.
Power and sample size: a study might be too small to detect a real effect (low power). Sample size calculations help determine how many participants are needed to detect an effect of a given magnitude.
Two sources, same conclusion, and I could not rule out that one copied the other.
I had written a reply contradicting post #9 and deleted it. Here is what survived.
Whatever the answer on Sample size intuition turns out to be, the method for getting there is the same: state the assumption, do the arithmetic in public, invite the correction.
The version of Sample size intuition that I was taught turned out to be a teaching simplification. Useful, and not true in the way I had assumed it was.
Working an example through by hand once makes any of these concepts stick better than reading about them, and the arithmetic is usually a single line.
Filing a mild objection to the consensus on Sample size intuition. Mild because I might be wrong; an objection because nobody has addressed the case that does not fit.
Where I part company with post #12, and it is a narrow parting.
Effect size with an interval is more informative than a significance verdict in essentially every case discussed here.
I would want to see it done twice before believing it once.
A p-value is the probability of data at least this extreme given the null hypothesis. It is not the probability the hypothesis is false, and almost every plain-language gloss gets that backwards.
Answering the question post #18 raises rather than the one it answers.
Sample size determines what a study could have detected, not what it did. Reporting the detectable effect size makes a null result interpretable.
Survivorship in a self-reporting population biases every aggregate produced from it, and the bias is in the flattering direction.
Distinguishing three things in the Sample size intuition discussion that keep getting used interchangeably: the observation, the proposed mechanism, and the recommendation that gets attached to both.
An update on my earlier Sample size intuition post: the pattern held for another six weeks and then stopped, which I did not predict and cannot explain.
The most common statistical error in this community is not technical: it is treating a self-selected collection of reports as a sample from a population.
Stating my assumptions rather than smuggling them in.
Quietly grateful for the plain phrasing. Not every thread gets that.
Post #25 is the version of this I will quote in future. One addition.
An honest declaration on Sample size intuition: I have a prior here and it is strong enough that you should weight what I say downward. Stating it rather than hiding it.
I would keep Sample size intuition and the decision it usually gets used for separate in this thread. They are related and they are not the same question, and merging them is why the last one went badly.
Number needed to treat: how many people need to be treated to prevent one bad outcome or achieve one good outcome. More intuitive than relative risk reduction.
Not the whole picture, but the part of it I can speak to.