From a few lucky flips to an iron-clad long-run average
Start — the everyday picture. Flip a fair coin 10 times and you might get 7 heads. That looks lopsided, and it is — for now. Flip it 10,000 times and the share of heads sits almost exactly on one-half. The Law of Large Numbers (LLN) is just that promise: the more times you repeat a random experiment, the closer the average of your results creeps to its true, settled value. Early luck gets diluted, never "corrected."
Call each draw $X_i$ and the true long-run value $\mu$ (the "expected value"). The number you actually track is the running average $\bar{X}_n=\frac{1}{n}\sum_{i=1}^{n}X_i$. The LLN in one line is simply $\bar{X}_n\to\mu$ as $n$ grows. Roll a fair die: $\mu=3.5$. After 6 rolls your average might be $4.2$; after 6,000 rolls it will sit within a few hundredths of $3.5$.
The running average doesn't lock onto $\mu$ instantly — it wobbles inside a shrinking band of width about $\sigma/\sqrt{n}$ (the standard error). Quadruple $n$ and you only halve the typical error: convergence is real but slow. There is one fine-print condition — $\mu$ must actually exist and be finite. The Cauchy distribution has heavy tails with no finite mean, so its sample average never settles; it keeps jumping no matter how large $n$ gets. In the panel, the Distribution selector chooses which $X_i$ you draw (and therefore $\mu$), Speed sets draws per second, and the +100 / +1K / +10K buttons leap $n$ forward.
(1) Keep Normal and hold +1K a few times — watch the green running mean hug the yellow $\mu$ line while the $\pm\sigma/\sqrt{n}$ shading squeezes inward. (2) Switch to Cauchy and add +10K: the average lurches up and down forever, never converging — the LLN's textbook counterexample. (3) Open the 1/√n Rate tab and watch the error envelope fall steeply at first, then crawl — proof that more data always helps, but with diminishing returns.
Law of Large Numbers — Jacob Bernoulli 1713 (Weak) / Borel 1909 (Strong)
Weak LLN: $\bar{X}_n\overset{p}{\to}\mu$ — i.e., $P(|\bar{X}_n-\mu|>\varepsilon)\to0$ for all $\varepsilon>0$
$$\boxed{\textbf{Strong LLN:}\quad P\!\left(\lim_{n\to\infty}\bar{X}_n=\mu\right)=1}$$
Requires: $E[X]=\mu$ exists and is finite. The Cauchy distribution fails — it has no finite mean.
$P(|\bar{X}_n-\mu|\ge\varepsilon)\le\frac{\text{Var}(\bar{X}_n)}{\varepsilon^2}=\frac{\sigma^2}{n\varepsilon^2}\to0$ as $n\to\infty$ (requires finite $\sigma^2$).
Weak LLN: for each ε, the probability of being far from μ goes to 0. Strong LLN: with probability 1, every path of $\bar{X}_n$ eventually stays within ε of μ forever. Strong implies weak, but not vice versa.
Fair coin, P(H)=0.5. LLN says empirical frequency of heads → 0.5 as n→∞. After 10 flips you might see 7/10=0.7. After 10,000: typically within 0.01 of 0.5. Chebyshev bound: P(|f_n-0.5|>0.01) ≤ 0.25/(n·0.0001) = 2500/n. At n=50,000: ≤ 5%.
"LLN says after 7 heads, tails is 'due' to restore the average."
LLN describes LONG-RUN averages over infinitely many trials — not corrections within a finite sequence. After 7 heads, P(next = tails) is still exactly 0.5. The coin has no memory. The LLN works because additional data eventually swamps early deviations, not because later outcomes "compensate."
📖 Ross — A First Course in Probability, Ch. 3.3
"LLN works for all distributions including the Cauchy."
LLN requires finite E[X]. The Cauchy distribution has no finite mean — its sample mean does NOT converge to any value. The sample mean of Cauchy(0,1) data has the same Cauchy(0,1) distribution regardless of n! The simulation shows this: Cauchy paths wander erratically even with 10,000 samples.
📖 Casella & Berger — Statistical Inference, Ch. 5.5
"LLN and CLT say the same thing."
LLN (convergence in probability/almost sure): x̄ₙ → μ (a constant). CLT (convergence in distribution): √n(x̄ₙ−μ) → N(0,σ²). LLN tells us WHERE the mean goes; CLT describes the SHAPE of fluctuations around that limit. Both are true simultaneously for finite-variance distributions.
📖 DeGroot & Schervish — Ch. 6.2–6.3
Confusing "sample mean converges" with "each sample value converges"
✅ Individual X_i are always drawn from the same distribution — they don't converge. Only the AVERAGE x̄ₙ converges to μ. Individual observations remain spread with variance σ² forever.
🔍 Thinking LLN means individual data points become less variable over time.
Applying LLN-based reasoning to small n
✅ LLN is an asymptotic statement (n→∞). For small n, the sample mean can be far from μ. The standard error σ/√n quantifies how far: ~68% chance |x̄ₙ−μ| < σ/√n. LLN says this → 0, but for n=10 it's σ/3.16 — possibly large.
🔍 Assuming "large enough" means n=20 for heavily skewed distributions.