Imagine listening for a friend who is calling your name across a noisy café. Whether you actually hear them depends on three things: how loudly they call, how much background noise drowns them out, and how many times they call. Statistical power is exactly this — the chance that a study "hears" a real effect instead of missing it.
Researchers give these pieces names. The effect size $d$ is how big the real difference is, measured in standard deviations — a shout versus a whisper. The significance level $\alpha$ is how careful we are about false alarms, usually $0.05$. The power $1-\beta$ is the chance we do hear a real call, and the sample size $n$ is how long we listen. The surprising part is that these four are locked together by a single equation. To reach 80% power for a medium effect ($d=0.5$) at $\alpha=0.05$, you need
$n=\left(\dfrac{z_{\alpha/2}+z_\beta}{d}\right)^2=\dfrac{(1.96+0.84)^2}{0.5^2}\approx 32$ observations (for a one-sample test).
Cut the effect in half to $d=0.25$ and you need four times as many — about 126 — because $n$ grows with $1/d^2$. Faint signals are expensive to detect.
More precisely, when the effect is real the test statistic is shifted: $Z\sim N(\delta,1)$ with non-centrality $\delta=d\sqrt{n}$. Power is the slice of that shifted bell beyond the critical value, $\text{Power}=\Phi\!\left(d\sqrt{n}-z_{\alpha/2}\right)$. This is why power climbs with $\sqrt{n}$, not $n$: to double the signal $\delta$ you must quadruple the sample. Each slider maps straight onto a symbol — d sets the true effect, α the false-alarm rate, n the sample size, and Target power the line you are trying to clear.
Try this in the sim above. Set $d=0.5$ and drag n — watch the green power curve rise past the 0.80 line right around $n=32$. Now shrink $d$ to $0.2$ and see that same curve flatten, now needing hundreds of samples. Finally open the H₀ vs H₁ tab and raise n: the two bells narrow and pull apart until the green "power" area swallows the rejection region — that growing overlap is the power.
Statistical Power Analysis — Cohen 1988 / Neyman-Pearson Framework
$$\boxed{\text{Power}=1-\beta=\Phi\!\left(d\sqrt{n}-z_{\alpha/2}\right)}$$
Required n for target power: $n=\left(\dfrac{z_{\alpha/2}+z_\beta}{d}\right)^2$
| Symbol | Meaning | Cohen convention |
|---|---|---|
| $d$ | Effect size $|\mu_1-\mu_0|/\sigma$ | 0.2=small, 0.5=medium, 0.8=large |
| $\alpha$ | Type I error (false positive rate) | Typically 0.05 |
| $\beta$ | Type II error (false negative rate) | Typically 0.20 (power=0.80) |
| $1-\beta$ | Power | P(reject H₀ | H₁ true) |
| $n$ | Sample size | Increases power as $\sqrt{n}$ |
Under $H_1$: test statistic $Z\sim N(d\sqrt{n},1)$ (non-central). Power = $P(Z>z_{\alpha/2}|H_1)=P(N(d\sqrt{n},1)>z_{\alpha/2})=\Phi(d\sqrt{n}-z_{\alpha/2})$.
Set $\Phi(d\sqrt{n}-z_{\alpha/2})=1-\beta$, solve: $d\sqrt{n}=z_{\alpha/2}+z_\beta\Rightarrow n=(z_{\alpha/2}+z_\beta)^2/d^2$.
For $\alpha=0.05$ (two-sided) and 80% power: $n=(1.960+0.842)^2/d^2=7.85/d^2$.
Detect d=0.5 (medium) with 80% power at α=0.05:
$n=(1.960+0.842)^2/0.5^2=(2.802/0.5)^2=5.604^2\approx\mathbf{32}$
This n=32 is the total for a one-sample test. For a two-sample comparison (two independent groups) you need roughly twice as many per group: n=2(zα/2+zβ)²/d² ≈ 63 per group (≈126 total) — the familiar "≈64 per group for a medium effect" rule of thumb. With n=20 instead (one-sample): Power=Φ(0.5√20-1.96)=Φ(0.277)≈0.61 — only a 61% chance of detecting the effect.
"Non-significant result proves the null hypothesis is true."
p>α only means insufficient evidence to reject H₀ — not proof of H₀. An underpowered study (30% power) misses real effects 70% of the time. Report the confidence interval: wide CI including the null is more informative than just p>0.05. "Absence of evidence is not evidence of absence."
📖 Cohen — Statistical Power Analysis, Ch. 1
"Post-hoc power analysis explains why a study was non-significant."
Computing power after seeing p>0.05 using the observed effect size is circular — observed power is mathematically tied to the p-value. It adds no information. Instead: report CI, equivalence tests, or meta-analysis with prior studies. Pre-registration and pre-study power analysis is the correct approach.
📖 Casella & Berger — Ch. 8.3
Using one-sided formula for a planned two-sided test
✅ Two-sided: n=(z_{α/2}+z_β)²/d². One-sided: n=(z_α+z_β)²/d². Two-sided requires larger n. Always specify tailedness before computing sample size.
🔍 Using z_α=1.645 instead of z_{α/2}=1.96 for a planned two-sided α=0.05 test.
Not inflating n for expected dropout
✅ The calculated n is required COMPLETERS. Inflate by expected dropout: n_recruit = n_needed/(1-dropout_rate). For 20% dropout and n=100 needed: recruit 125. Always account for missingness upfront.
🔍 Recruiting exactly n_needed without a dropout buffer.
Using d from pilot study directly in power analysis
✅ Pilot study effect sizes are highly variable (often inflated by winner's curse). Use literature-based estimates or the minimum meaningful effect size (MDE), not the pilot estimate. Small pilot studies (n=10-20) give extremely unreliable d estimates.
🔍 Getting d=1.2 from a pilot n=8 study and computing n=9 for main study.