From "trust one noisy school less" to shrinkage estimators
Two restaurants both show five stars. One has 1,000 reviews, the other has 3. You instinctively trust the busy one and treat the three-review score as barely better than a guess — quietly nudging it toward "probably an average place." Hierarchical models make that instinct precise: when a group carries only a little data, lean on what groups typically look like; when it carries a lot, trust the group's own number. That nudge of a shaky estimate toward the overall average is called shrinkage, or more gently partial pooling.
Suppose a small school's 4 students average $\bar y_j = 70$ on a test, while the average across all schools is $\mu = 50$. Believing the 70 outright ("no pooling") over-reacts to four possibly-lucky students; forcing every school to 50 ("complete pooling") pretends schools don't really differ. Partial pooling splits the difference with a weight $\lambda$ between 0 and 1: $\hat\theta_j=\lambda\,\bar y_j+(1-\lambda)\mu$. With $\lambda=0.5$ this school's estimate becomes $0.5(70)+0.5(50)=60$ — pulled halfway home toward the crowd.
The weight reflects how much trustworthy signal each group carries: $\lambda_j=\dfrac{n_j\tau^2}{n_j\tau^2+\sigma^2}$, where $\sigma$ is the noise within a group and $\tau$ is the genuine spread between groups. More observations ($n_j\!\uparrow$) or larger true between-group differences ($\tau\!\uparrow$) push $\lambda\to1$ (trust the group mean); noisier data ($\sigma\!\uparrow$) or near-identical groups ($\tau\to0$) push $\lambda\to0$ (shrink to $\mu$). The same two variances define the intraclass correlation $\text{ICC}=\tau^2/(\tau^2+\sigma^2)$ — the share of all variation that lives between groups rather than within them. The four sliders set the number of groups $J$, the per-group sample size $n$, and these two standard deviations $\tau$ and $\sigma$.
On the Shrinkage Effect tab, drag $\tau$ down toward zero and watch every group's purple dot collapse onto the grand-mean line — that is $\lambda\to0$. Then drag $n$ upward and see the dots spring back toward their own group means as $\lambda\to1$. On the Pooling Comparison tab, set $\tau$ small and confirm complete pooling wins on MSE, then set $\tau$ large and watch no pooling take over — with partial pooling rarely the worst in either regime. Finally, the ICC & Variance tab traces how raising $\tau$ relative to $\sigma$ lifts the ICC.
Hierarchical Normal Model — Gelman et al.
$y_{ij}\mid\theta_j\sim N(\theta_j,\sigma^2)$, $\quad\theta_j\mid\mu,\tau\sim N(\mu,\tau^2)$, $\quad\mu\sim\text{flat}$
$$\boxed{\hat\theta_j^{\text{partial pool}}=\underbrace{\lambda_j}_{\text{weight}}\bar y_j+(1-\lambda_j)\hat\mu,\quad\lambda_j=\frac{n_j/\sigma^2}{n_j/\sigma^2+1/\tau^2}}$$
| Method | Estimate of theta_j | When to use |
|---|---|---|
| Complete pooling | Grand mean ȳ (ignore groups) | Groups are identical (tau=0) |
| No pooling | Group mean ȳ_j (ignore hierarchy) | Groups are completely independent |
| Partial pooling | Weighted average (shrinkage) | Groups share info — best in most cases |
$\lambda_j = n_j\tau^2/(n_j\tau^2+\sigma^2)\in[0,1]$. When $n_j\to\infty$: $\lambda_j\to1$ (trust group mean fully). When $\tau^2\to0$ (groups identical): $\lambda_j\to0$ (shrink to grand mean). When $\sigma^2\to\infty$ (noisy data): $\lambda_j\to0$ (shrink more).
ICC $=\tau^2/(\tau^2+\sigma^2)$. Measures the proportion of total variance explained by group membership. ICC=0: groups are identical (complete pooling is appropriate). ICC=1: within-group variance=0 (no pooling appropriate). Most real data: ICC between 0.05 and 0.3.
"Hierarchical models are only for Bayesian analysis."
Hierarchical models = multilevel models = mixed effects models. They exist in both frequentist (lme4 in R, mixed effects in STATA) and Bayesian frameworks. The key concept — partial pooling — is the same in both. Bayesian implementations make priors explicit; frequentist estimates variance components by REML.
📖 Gelman & Hill — Data Analysis Using Regression and Multilevel Models
Using complete pooling when data has group structure
✅ Ignoring group structure when groups exist causes Simpson's paradox — individual group trends can be masked by aggregate data. Always check if there is meaningful between-group variation (tau > 0) before pooling. Use ICC to quantify group structure.
🔍 Analyzing test scores by pooling all students without accounting for school effects.
Using no pooling (separate models) for each group
✅ No pooling is inefficient: groups with few observations have unreliable estimates. Partial pooling borrows strength from other groups. For groups with n_j=2, the no-pooling estimate is essentially noise. Hierarchical models regularize small-group estimates by shrinking toward the grand mean.
🔍 Fitting separate regression for each school with n=3 students.