From a gut feeling to a number
Suppose a friend flips a coin ten times and gets nine heads. You start to doubt the coin is fair — not because nine heads is impossible, but because it would be surprising if the coin really were fair. A hypothesis test is just a careful, numerical version of that reaction: assume nothing special is going on, then measure how surprising the data would be if that assumption were true.
The boring default is the null hypothesis $H_0$ — "the coin is fair," or "these bolts really do average 10 mm." Our data give a sample mean $\bar{x}$. We measure surprise by counting how many standard errors $\bar{x}$ sits away from the claimed value $\mu_0$:
$$z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}$$
A worked number: a factory claims its bolts average $\mu_0 = 10$ mm. We measure $n = 25$ bolts and find $\bar{x} = 10.4$ mm, with known spread $\sigma = 1$ mm. The standard error is $\sigma/\sqrt{n} = 1/\sqrt{25} = 0.2$ mm, so $z = 0.4 / 0.2 = 2$. Our sample sits two standard errors above the claim — fairly far out.
Next we turn that distance into a probability. The p-value is the chance, if $H_0$ were true, of landing at least this far out: $p = 2\,P(Z > |z|) = 2\bigl(1 - \Phi(2)\bigr) \approx 0.046$. We compare it to a threshold $\alpha$ chosen in advance (often $0.05$). If $p \le \alpha$ we reject $H_0$; otherwise we fail to reject it. Crucially, we never prove $H_0$ true — we only decide whether the data are surprising enough to abandon it. Two ways to slip up: a Type I error (rejecting a true $H_0$, which happens at rate $\alpha$) and a Type II error (missing a real effect, rate $\beta$). The power $1-\beta$ is the chance of catching a real effect, and it climbs as $n$ grows, because the $\sqrt{n}$ in the standard error squeezes the curve tighter.
Each slider maps onto this picture: $\mu_0$ shifts the null curve left or right, $\bar{x}$ moves the red test-statistic line, $\alpha$ widens or narrows the red rejection region, and $n$ and $\sigma$ together set the curve's width through $\sigma/\sqrt{n}$.
1. Set μ₀=10, x̄=10.4, σ=1, n=25, α=0.05 and watch the test-statistic line land just inside the rejection region, with $p \approx 0.046$ — a borderline "reject."
2. Now drag n up toward 100. The curve narrows, the very same 0.4 mm gap becomes overwhelming, and the p-value collapses — that is statistical power growing with sample size.
3. Pull α down toward 0.001 and watch the red rejection region shrink until even a far-out x̄ no longer rejects — the Type I versus Type II tradeoff made visible.
Neyman-Pearson Hypothesis Testing Framework — Jerzy Neyman & Egon Pearson, 1933
One-sample z-test statistic (known σ):
$$\boxed{Z = \frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}} \overset{H_0}{\sim} \Normal{0}{1}}$$
Decision rule (two-sided at level $\alpha$): Reject $H_0$ if $|Z| > z_{\alpha/2}$, where $z_{\alpha/2} = \Phi^{-1}(1-\alpha/2)$.
| Symbol | Meaning | Type | Range | Interpretation |
|---|---|---|---|---|
| $H_0$ | Null hypothesis | Statement | — | The default assumption to be tested (e.g. $\mu=\mu_0$) |
| $H_1$ | Alternative hypothesis | Statement | — | What we conclude if $H_0$ is rejected |
| $\alpha$ | Significance level | Parameter | $(0,1)$ | P(Type I error) = P(reject $H_0$ | $H_0$ true) |
| $\beta$ | Type II error rate | Derived | $(0,1)$ | P(fail to reject $H_0$ | $H_1$ true) |
| $1-\beta$ | Power | Derived | $(0,1)$ | P(reject $H_0$ | $H_1$ true) — correctly detecting effect |
| $p$-value | p-value | Statistic | $[0,1]$ | $P(\text{data this extreme} | H_0 \text{ true})$ — NOT $P(H_0|\text{data})$ |
| $z_{\alpha/2}$ | Critical value | Derived | $(0,\infty)$ | Boundary of rejection region; $\Phi^{-1}(1-\alpha/2)$ |
| $n$ | Sample size | Fixed | $\mathbb{N}$ | Larger $n$ increases power for fixed effect size |
We want to decide if an observed sample mean $\bar{x}$ is "too far" from a hypothesized value $\mu_0$ to be explained by chance. "Too far" is quantified by standardizing — how many standard errors away is $\bar{x}$ from $\mu_0$?
If $H_0: \mu = \mu_0$ is true and $X_i \overset{iid}{\sim} \Normal{\mu_0}{\sigma^2}$, then by properties of the normal distribution:
$$\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i \sim \Normal{\mu_0}{\sigma^2/n}$$
Subtracting the mean and dividing by the standard error gives a standard normal under $H_0$:
$$Z = \frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}} \overset{H_0}{\sim} \Normal{0}{1}$$
The observed value $z_{obs} = \frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}}$ measures evidence against $H_0$.
We want $P(\text{reject }H_0 | H_0\text{ true}) = \alpha$. For a two-sided test:
$$P(|Z| > z_{\alpha/2} | H_0) = P(Z > z_{\alpha/2}) + P(Z < -z_{\alpha/2}) = \frac{\alpha}{2} + \frac{\alpha}{2} = \alpha$$
At $\alpha=0.05$: $z_{0.025} = \Phi^{-1}(0.975) \approx 1.960$.
The p-value is the probability, under $H_0$, of observing a test statistic at least as extreme as what we got:
$$p\text{-value (two-sided)} = 2\cdot P(Z > |z_{obs}|) = 2\cdot(1-\Phi(|z_{obs}|))$$
Reject $H_0$ if $p \le \alpha$ (equivalently, if $|z_{obs}| > z_{\alpha/2}$). The two error types are:
$$\text{Type I error: } \alpha = P(\text{reject }H_0|H_0\text{ true})$$
$$\text{Type II error: } \beta = P(\text{fail to reject }H_0|H_1\text{ true}) = \Phi\!\left(z_{\alpha/2} - \frac{|\mu_1-\mu_0|}{\sigma/\sqrt{n}}\right)$$
The power $\pi(\mu) = 1-\beta(\mu)$ is the probability of correctly rejecting for each true $\mu$:
$$\pi(\mu) = 1 - \Phi\!\left(z_{\alpha/2} - \frac{\mu-\mu_0}{\sigma/\sqrt{n}}\right) + \Phi\!\left(-z_{\alpha/2} - \frac{\mu-\mu_0}{\sigma/\sqrt{n}}\right)$$
This creates a U-shaped power curve — deepest at $\mu=\mu_0$ (where power = $\alpha$) and rising as $\mu$ departs from $\mu_0$.
A $(1-\alpha)$-level confidence interval for $\mu$ is the set of all $\mu_0$ values that would not be rejected:
$$\text{CI} = \left[\bar{x} - z_{\alpha/2}\frac{\sigma}{\sqrt{n}},\; \bar{x} + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right]$$
Reject $H_0: \mu=\mu_0$ at level $\alpha$ $\Leftrightarrow$ $\mu_0 \notin$ the $(1-\alpha)$ CI.
The μ₀ slider sets the null hypothesis value $\mu_0$ — this shifts the null distribution left/right. The red rejection region stays fixed relative to this curve at proportion $\alpha$.
The x̄ slider is the sample mean — watch the red vertical line (test statistic) move. When it enters the red rejection region, the banner shows "Reject H₀". The p-value (purple shaded area) changes continuously.
The α slider directly controls the critical value $z_{\alpha/2}$ — watch the rejection region expand (larger α → easier to reject → more Type I errors) or shrink (smaller α → harder to reject → more Type II errors). This is the fundamental tradeoff.
The n slider changes the standard error $\sigma/\sqrt{n}$ — larger $n$ narrows the null distribution, making the same $|z_{obs}|$ correspond to a smaller p-value (more power to detect the same effect).
Problem: A manufacturer claims their bolts have mean diameter $\mu_0 = 10$ mm. A quality inspector takes $n=25$ bolts and finds $\bar{x} = 10.4$ mm. Known population std dev $\sigma = 1$ mm. Test at $\alpha = 0.05$ (two-sided).
Step 1 — Hypotheses: $H_0: \mu = 10$ vs $H_1: \mu \neq 10$
Step 2 — Test statistic:
$$z_{obs} = \frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}} = \frac{10.4 - 10}{1/\sqrt{25}} = \frac{0.4}{0.2} = 2.00$$
Step 3 — Critical value: $z_{\alpha/2} = z_{0.025} = 1.960$
Step 4 — p-value:
$$p = 2\cdot P(Z > 2.00) = 2\cdot(1-\Phi(2.00)) = 2\cdot 0.0228 = \mathbf{0.0456}$$
Step 5 — Decision: Since $p = 0.0456 < \alpha = 0.05$, we Reject $H_0$. There is statistically significant evidence that the true diameter differs from 10 mm.
Set μ₀=10, x̄=10.4, σ=1, n=25, α=0.05 in the simulation to verify!
"p = 0.03 means there is a 3% probability that H₀ is true."
The p-value is $P(\text{data this extreme} | H_0 \text{ true})$ — a probability about the data given the hypothesis, not about the hypothesis given the data. $P(H_0|\text{data})$ is a Bayesian posterior probability and requires a prior. The frequentist p-value makes no statement about whether $H_0$ is true or false; it only says: "If $H_0$ were true, results this extreme would occur 3% of the time by chance." This is a conditional probability that is frequently inverted in interpretation.
📖 ASA Statement on p-values — Wasserstein & Lazar, The American Statistician, 2016
"Failing to reject H₀ proves H₀ is true."
"Fail to reject" is not the same as "accept." Absence of evidence is not evidence of absence. With low power (small n, small effect), we frequently fail to detect real effects. The correct statement is: "We do not have sufficient evidence to reject $H_0$ at level $\alpha$." Equivalently, $\mu_0$ lies within the confidence interval — but the CI may be very wide (uninformative).
📖 Casella & Berger — Statistical Inference, Ch. 8.3
"p < 0.05 means the result is practically important."
Statistical significance ≠ practical significance. With $n=100{,}000$, a difference of 0.001 mm in bolt diameter could give $p=0.0001$ — highly significant statistically but completely irrelevant practically. Always report effect size (Cohen's $d$, $r^2$, etc.) alongside the p-value. Conversely, with small $n$, a clinically important effect might give $p=0.2$ — nonsignificant but potentially important. Power is the right metric for detection.
📖 Freedman, Pisani & Purves — Statistics, Ch. 26
Using z-test when σ is unknown and n is small
✅ When σ is unknown (the usual case), use the t-test with $t = (\bar{x}-\mu_0)/(s/\sqrt{n})$ and $n-1$ degrees of freedom. The z-test assumes $\sigma$ is known. With large $n$ (>100), the difference is negligible, but for $n<30$ it matters.
🔍 Students plug in the sample std dev $s$ but use the z-table instead of the t-table.
Dividing the p-value by 2 for a two-sided test when the table gives one-tail values
✅ If your table/software gives the one-tail p-value and you're doing a two-sided test, multiply by 2: $p_{\text{two-sided}} = 2 \cdot p_{\text{one-tail}}$. Dividing gives a p-value too small, making it easier to reject — an anticonservative error.
🔍 Confusion between "divide α by 2 for the critical value" and "multiply the one-tail p by 2".
Choosing the tail direction after seeing the data (HARKing)
✅ The alternative hypothesis $H_1$ (one-sided or two-sided) must be specified before collecting data. Choosing one-sided after seeing that $\bar{x} > \mu_0$ effectively halves the p-value and doubles the Type I error rate. Pre-registration addresses this. Use two-sided tests by default unless there is a strong theoretical reason for directionality.
🔍 Students pick the "better looking" tail to get p<0.05 — this is p-hacking.