From one cause to many, untangled
Start — the everyday version. Guess a used car's price. Mileage matters, but so does age — and older cars usually have more miles. If you only look at mileage, you end up blaming it for wear that age really caused. Multiple regression asks a sharper question: how much does each thing matter on its own, while the others are held steady?
Build — name the pieces. Call the outcome $Y$ (say, a house price) and the predictors $X_1$ (size) and $X_2$ (bedrooms). The model simply adds up weighted contributions: $\hat{Y}=\hat\beta_0+\hat\beta_1 X_1+\hat\beta_2 X_2$. Each slope is a partial effect — the change in $Y$ for a one-unit change in that predictor while the others stay fixed. With $\hat{Y}=1.2+0.8X_1+0.3X_2$ (price in units of \$100K), a 1500 sqft ($X_1=15$), 3-bedroom ($X_2=3$) house predicts $\hat{Y}=1.2+12+0.9=14.1$, i.e. about \$1.41M. The value $\hat\beta_1=0.8$ says each extra 100 sqft adds \$80K — holding bedrooms fixed.
Deepen — the precise form. Stack every house into a design matrix and the model becomes $\mathbf{Y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\varepsilon$. The least-squares fit that minimizes $\|\mathbf{Y}-\mathbf{X}\boldsymbol\beta\|^2$ is $\hat{\boldsymbol\beta}=(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{Y}$. That matrix inverse is what "holding the others fixed" actually does — it strips out the variation predictors share. When two predictors are nearly the same ($X_1$ and $X_2$ strongly correlated), $\mathbf{X}^\top\mathbf{X}$ becomes near-singular, the inverse blows up, and the coefficient standard errors explode. The variance inflation factor $\text{VIF}=1/(1-r_{12}^2)$ measures this: $\text{VIF}=1$ means independent predictors, $\text{VIF}>10$ signals trouble. The sliders map straight onto these symbols: True β₁/β₂ set the real partial effects the data is built from, Noise σ adds scatter, Corr(X₁,X₂) dials up multicollinearity, and n is the sample size.
① Set Corr(X₁,X₂)=0 and watch $\hat\beta_1$ and $\hat\beta_2$ land near the true values; now slide it to 0.95 and click New Data a few times — the estimates lurch around even though R²adj barely moves.
② Open the Multicollinearity tab and drag Corr upward: the point cloud collapses toward a line as VIF climbs from 1 past 10.
③ Push Noise σ high, then raise n from 50 to 300 — more data pulls the estimates back toward the truth.
Multiple Linear Regression — OLS — Matrix Form
$$\mathbf{Y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon},\quad\varepsilon\sim\mathcal{N}(\mathbf{0},\sigma^2\mathbf{I})$$
$$\boxed{\hat{\boldsymbol{\beta}}=(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{Y}}$$
| Symbol | Meaning | Interpretation |
|---|---|---|
| $\mathbf{X}$ | Design matrix (n×p) | First column all 1s (intercept), rest are predictors |
| $\hat{\boldsymbol{\beta}}$ | OLS estimates | Partial effects: β̂₁ = effect of X₁ holding X₂ fixed |
| $R^2_{adj}$ | Adjusted R² | $1-(1-R^2)(n-1)/(n-p)$ — penalizes extra predictors |
| VIF | Variance Inflation Factor | $1/(1-r_{12}^2)$ — measures multicollinearity; VIF>10 problematic |
Minimize $\|\mathbf{Y}-\mathbf{X}\boldsymbol{\beta}\|^2$. Setting gradient to zero: $\mathbf{X}^\top\mathbf{X}\hat{\boldsymbol{\beta}}=\mathbf{X}^\top\mathbf{Y}$. Unique solution when $\mathbf{X}^\top\mathbf{X}$ is invertible (no perfect multicollinearity).
$\hat{\beta}_1$ is the change in $E[Y]$ for a one-unit increase in $X_1$ HOLDING $X_2$ CONSTANT. This differs from simple regression of Y on X₁ alone when $X_1$ and $X_2$ are correlated (omitted variable bias).
When $r_{12}\approx\pm1$: $(\mathbf{X}^\top\mathbf{X})$ near-singular → $\text{Var}(\hat{\beta}_j)=\sigma^2[(\mathbf{X}^\top\mathbf{X})^{-1}]_{jj}$ explodes → unstable estimates. VIF$_j=1/(1-R^2_j)$ where $R^2_j$ = R² from regressing $X_j$ on all others.
House price Y ($100K): X₁=size (100sqft), X₂=bedrooms
$\hat{Y}=1.2+0.8X_1+0.3X_2$. House with X₁=15, X₂=3: $\hat{Y}=1.2+12+0.9=14.1$ (in units of \$100K) $\approx\$1.41$M
β̂₁=0.8: holding bedrooms fixed, each 100sqft adds $80K on average.
"Adding more predictors always improves MLR."
R² always increases with more predictors — even useless ones. Adjusted R² penalizes complexity. Use AIC, BIC, or cross-validation for model selection. More predictors also increase multicollinearity risk and reduce interpretability.
📖 Casella & Berger — Ch. 11
"β̂₁ in MLR has the same interpretation as in simple regression of Y on X₁."
In MLR, β̂₁ is the PARTIAL effect of X₁ holding all other predictors fixed. In simple regression, it's the total (marginal) effect. They differ when predictors are correlated — omitting a correlated predictor causes omitted variable bias.
📖 Hogg, McKean & Craig — Ch. 9
Not checking multicollinearity before interpreting coefficients
✅ Compute VIF for all predictors. VIF>10: multicollinearity is severe, estimates unstable. Solutions: drop redundant predictors, use Ridge regression, or combine via PCA.
🔍 Interpreting wildly different coefficients on correlated predictors without diagnosing multicollinearity.
Using R² instead of adjusted R² to compare models with different numbers of predictors
✅ R²_adj = 1-(1-R²)(n-1)/(n-p) — it can decrease when adding an unhelpful predictor. Always use adjusted R² for model comparison.
🔍 Preferring a 5-variable model (R²=0.82) over a 2-variable model (R²=0.80) without checking adjusted R².
Not checking residual plots in MLR
✅ Plot residuals vs each predictor and vs fitted values. Fan shapes = heteroskedasticity. Curves = nonlinearity (add polynomial term). Always verify OLS assumptions visually before reporting inference.
🔍 Reporting p-values from MLR without checking any residual diagnostics.