From "did it really help?" to a number you can trust
Two classes take the same test. One studied with flashcards, the other re-read the textbook, and the flashcard class scored a few points higher on average. Real effect — or did that group just happen to contain a couple of strong students? Comparing the averages of two separate groups, and deciding whether the gap is bigger than ordinary luck, is exactly what the two-sample t-test does.
Give the pieces names. Group 1 has sample mean $\bar{X}_1$ and group 2 has sample mean $\bar{X}_2$; the quantity we care about is their difference $\bar{X}_1-\bar{X}_2$. But every sample wobbles, so we also estimate how much that difference could jiggle by chance alone — its standard error (SE). The test is just a signal-to-noise ratio:
$$t=\frac{\text{difference in group means}}{\text{typical wobble of that difference}}=\frac{\bar{X}_1-\bar{X}_2}{\text{SE}}$$
A worked number: if class A averages $82$, class B averages $78$, and the standard error of that $4$-point gap is $2$ points, then $t=4/2=2$ — the gap is twice the size of the noise. We then compare that $t$ against a reference curve to get a $p$-value: the probability of seeing a gap this large if the two groups were really the same.
The precise Welch form does not assume the two groups are equally spread. It builds the standard error from each group's own variance, $\;\text{SE}=\sqrt{S_1^2/n_1+S_2^2/n_2}\;$, so that $t=(\bar{X}_1-\bar{X}_2)/\sqrt{S_1^2/n_1+S_2^2/n_2}$, and it reads the $p$-value off the $t_\nu$ curve using the (usually non-integer) Welch-Satterthwaite degrees of freedom $\nu$ shown in the next section. Significance is a separate question from size: Cohen's d $=(\bar{X}_1-\bar{X}_2)/S_p$ reports the gap in standard-deviation units. In the simulation above, the $\mu_1,\mu_2$ sliders set the true group means, $\sigma_1,\sigma_2$ set their spreads, $n_1,n_2$ set the sample sizes, and $\alpha$ is the cutoff the $p$-value is compared against.
1) Set $\mu_1=\mu_2$ so there is no real difference, then press "New Sample" over and over and watch the $p$-value bounce around, dipping below $0.05$ now and then (each of those is a false positive). 2) Keep the means a fixed distance apart and drag $n_1$ and $n_2$ from $5$ up to $200$: the same true gap turns "significant" as the samples grow. 3) Push $\sigma_2$ far above $\sigma_1$ and open the Sampling Demo tab to see the extra spread that makes Welch's unequal-variance correction worth using.
Two-Sample Welch t-Test — Bernard Lewis Welch, 1947
$$\boxed{t=\frac{\bar{X}_1-\bar{X}_2}{\sqrt{S_1^2/n_1+S_2^2/n_2}}\approx t_{\nu}}$$
Welch-Satterthwaite degrees of freedom: $\nu=\frac{(S_1^2/n_1+S_2^2/n_2)^2}{(S_1^2/n_1)^2/(n_1-1)+(S_2^2/n_2)^2/(n_2-1)}$
| Symbol | Meaning | Note |
|---|---|---|
| $\bar{X}_1,\bar{X}_2$ | Group sample means | Estimators of μ₁, μ₂ |
| $S_1^2,S_2^2$ | Group sample variances | Estimated from data, not assumed equal |
| $n_1,n_2$ | Group sample sizes | Need not be equal |
| $\nu$ | Welch df | Usually non-integer; Satterthwaite approximation |
Student's pooled t-test assumes equal variances: $S_p^2=\frac{(n_1-1)S_1^2+(n_2-1)S_2^2}{n_1+n_2-2}$. Welch's test makes no such assumption. Simulation studies (Zimmermann 2004) show Welch maintains Type I error ≤ α even when variances differ; Student's test inflates Type I error up to 20% with unequal variances. Use Welch's test by default — it has virtually no cost when variances are equal and protects when they're not.
$$d=\frac{\bar{X}_1-\bar{X}_2}{S_p},\quad S_p=\sqrt{\frac{(n_1-1)S_1^2+(n_2-1)S_2^2}{n_1+n_2-2}}$$
Interpretation: d=0.2 small, d=0.5 medium, d=0.8 large (Cohen 1988).
Problem: Drug vs placebo blood pressure reduction. Drug: n₁=25, x̄₁=12.3, s₁=3.2 mmHg. Placebo: n₂=25, x̄₂=8.7, s₂=4.1 mmHg.
SE$=\sqrt{3.2^2/25+4.1^2/25}=\sqrt{0.4096+0.6724}=\sqrt{1.082}\approx1.040$
$t=(12.3-8.7)/1.040=3.6/1.040\approx3.46$, Welch df≈45, p≈0.0012
Since p=0.0012<0.05, reject H₀. Drug significantly reduces BP more than placebo.
"Always use pooled (Student's) t-test; Welch is only for unequal variances."
Welch's test performs as well as Student's when variances are equal, and much better when they're not. Modern statistical practice recommends always using Welch's test. Never assume equal variances without testing with Levene's or Bartlett's test.
📖 Zimmermann (2004), Journal of Modern Applied Statistics
"Independent and paired t-tests are the same — use either for any two-group comparison."
Independent: two different groups (treatment vs control, men vs women). Paired: same subjects measured twice (before/after), or matched pairs. Paired is more powerful when within-subject correlation is high. Using independent t-test on paired data ignores correlation and reduces power.
📖 Casella & Berger — Ch. 11.1
Using incorrect pooled SE formula for Welch's test
✅ Welch SE = √(S₁²/n₁ + S₂²/n₂). Student's pooled SE = Sp·√(1/n₁+1/n₂). These are different. If variances are unequal, the pooled formula underestimates SE → inflated t → false positives.
🔍 Students memorize one formula and use it regardless of whether variances are equal.
Using wrong df for Welch's test (using n₁+n₂-2)
✅ Welch df = (S₁²/n₁+S₂²/n₂)²/[(S₁²/n₁)²/(n₁-1)+(S₂²/n₂)²/(n₂-1)] — Satterthwaite formula. Using n₁+n₂-2 (Student's df) with unequal variances gives incorrect critical values.
🔍 Plugging into Student's t-table when Welch df is needed.
Not checking independence assumption between groups
✅ Two-sample t-test requires independent observations WITHIN each group and independence BETWEEN groups. Paired data requires the paired t-test. Time-series data requires accounting for autocorrelation.
🔍 Using two-sample t-test for before/after measurements on the same subjects.