Suppose you measured 10 plants and found their average height. How sure are you about that average? If you could grow 10 brand-new plants you'd get a slightly different average — but you only have the 10 you measured. The bootstrap is a clever trick: pretend your sample is the whole world and "regrow" fake datasets from it. Pull a value out of your data, write it down, and put it back — then pull again, until you have a fresh set of 10. Because you put each one back, some values get picked twice and some not at all, so every new set comes out a little different.
Your data are $x_1,\dots,x_n$ and the number you care about is a statistic such as the sample mean $\bar{x}$. One resample draws $n$ values with replacement and gives a new mean $\bar{x}^{*}$. Do this $B$ times and you get $B$ slightly different means; how widely they spread tells you how jumpy $\bar{x}$ really is. Tiny example: from $[2,4,4,9]$ one resample might be $[4,4,9,9]\to 6.5$ and another $[2,2,4,4]\to 3.0$. Collect a thousand of these and the histogram of $\bar{x}^{*}$ is an estimate of the sampling distribution — with no formula required.
Formally, resampling draws from the empirical CDF $\hat{F}_n(x)=\frac{1}{n}\sum_{i=1}^n\mathbf{1}[x_i\le x]$, which converges to the true $F$ (Glivenko–Cantelli). This is the plug-in principle: use the data's own distribution in place of the unknown population one. A 95% percentile confidence interval is then just the middle 95% of the sorted bootstrap means, $[\hat\theta^{*}_{(0.025B)},\,\hat\theta^{*}_{(0.975B)}]$. The n slider sets how much data you have (more data → a genuinely narrower interval), while B sets how many resamples you take (more → a smoother histogram, but not a narrower interval — $B$ only sharpens the estimate, it cannot add information that is not in your sample).
Switch the distribution to Exponential, run the bootstrap, and open the "vs Normal CI" tab — the bootstrap interval leans to one side while the normal one stays symmetric. Next, crank B from 100 up to 5000 and watch the histogram smooth out without changing its overall width. Then drag n down to 5 and back up to 200 to see the interval widen with little data and tighten as the sample grows.
Bootstrap — Bradley Efron, 1979
$$\boxed{\hat{F}_n(x)=\frac{1}{n}\sum_{i=1}^n\mathbf{1}[X_i\le x]\to F(x)\quad\text{(empirical CDF approximates true CDF)}}$$
Bootstrap CI for statistic $\hat\theta$: draw $B$ bootstrap samples $X^{*b}=(X_{i_1}^*,...,X_{i_n}^*)$ with replacement from data, compute $\hat\theta^{*b}$ for each, then $\text{CI}_{0.95}=\left[\hat\theta^*_{(0.025B)},\;\hat\theta^*_{(0.975B)}\right]$
Replace the unknown population distribution $F$ with the empirical distribution $\hat{F}_n$. Then bootstrap sampling from $\hat{F}_n$ mimics real sampling from $F$. The variability of $\hat\theta^*$ around $\hat\theta$ (bootstrap variation) approximates the variability of $\hat\theta$ around $\theta$ (sampling variation).
1. Compute $\hat\theta$ on original data. 2. For $b=1,...,B$: resample $n$ observations WITH replacement → compute $\hat\theta^{*b}$. 3. Sort $\hat\theta^{*1},...,\hat\theta^{*B}$. 4. CI = [$\hat\theta^*_{(0.025)}$, $\hat\theta^*_{(0.975)}$]. No normality assumption needed!
When the statistic's sampling distribution is unknown or non-normal: median, correlation coefficient, regression coefficient, ratio estimators. When sample is too small for CLT approximation. When parametric assumptions are violated.
Data: Incomes (skewed): [25,28,30,35,45,50,55,60,80,200] (n=10, in $thousands)
$\hat\mu=\bar{x}=60.8$, sample SD $s\approx 51.8$. Normal CI: 60.8 ± 1.96×51.8/√10 = [28.7, 92.9]
Bootstrap (B=10,000): percentile CI ≈ [32.5, 96.4] — shifted and asymmetric because the distribution is right-skewed.
The bootstrap CI is more accurate here because it captures the skewness of the sampling distribution directly.
"Bootstrap creates new data — it's cheating because we're generating fake observations."
Bootstrap does NOT create new information. It uses the existing data to approximate the sampling distribution of a statistic. Resampling with replacement is mathematically equivalent to computing the statistic's distribution assuming the true distribution is the empirical distribution F̂ₙ. By Glivenko-Cantelli theorem, F̂ₙ → F uniformly, so this approximation is valid.
📖 Efron & Tibshirani — An Introduction to the Bootstrap, Ch. 2
"Bootstrap always works — it's a universal solution."
Bootstrap fails for: statistics that depend on extreme values (minimum, maximum), non-smooth statistics, situations where n is too small relative to the complexity of the distribution, and heavy-tailed distributions (Pareto with α≤2). Bootstrap also fails for extremes of the distribution (tail probabilities require more sophisticated methods like importance sampling).
📖 Casella & Berger — Ch. 10
Sampling WITHOUT replacement for bootstrap
✅ Bootstrap resamples WITH replacement. Sampling without replacement always gives the same dataset. The key is that with replacement, each resample is a different random subset, simulating drawing new data from the population.
🔍 Students implement bootstrap without replacement, getting identical resamples.
Not using enough bootstrap resamples B
✅ B=1000 for rough estimates; B=10,000 for accurate 95% CIs; B=100,000 for 99% CIs or very small tail probabilities. The Monte Carlo error of the CI endpoint estimate is O(1/√B). Rule of thumb: B ≥ 1000 for 95% CI, B ≥ 9999 for 99% CI.
🔍 Using B=100 bootstrap resamples — too few for stable CI estimates.
Using bootstrap when parametric method is available and valid
✅ When normality holds and the statistic's exact distribution is known (e.g., t-test for mean), use the parametric method — it's more efficient (narrower CIs for same coverage). Bootstrap is valuable when no parametric method exists or when assumptions are violated.
🔍 Using bootstrap for a normal sample mean when the t-interval is exact and more efficient.