From a yes/no question to a smooth probability
Start with a plain question with only two answers: will it rain tomorrow — yes or no? Will this student pass or fail? Will an email be spam or not? We don't just want a guess; we want a chance, like "about a 70% chance." And a chance has to stay between 0% and 100%. If you tried to draw a straight line through the data the way you do in ordinary regression, the line would happily shoot above 100% or below 0% — which is nonsense for a probability. So we bend the straight line into a gentle S-shape that hugs 0 at the bottom and 1 at the top. That S is the whole trick.
Now give the S a name and a formula. Let $x$ be something we can measure (say, hours studied) and let $p$ be the probability of a "yes" (passing). The S-curve is the sigmoid: $p=\sigma(\beta_0+\beta_1 x)=\dfrac{1}{1+e^{-(\beta_0+\beta_1 x)}}$. The number $\beta_1$ controls how steep the S is, and $\beta_0$ slides it left or right. Try a quick number: suppose $\beta_0=-4$ and $\beta_1=1$, and a student studies $x=5$ hours. Then $\beta_0+\beta_1 x=-4+5=1$, and $p=\sigma(1)=1/(1+e^{-1})\approx 0.73$ — about a 73% chance of passing.
To see why the line is straight again underneath, talk about odds instead of probability. Odds are $p/(1-p)$: at $p=0.73$ the odds are about $0.73/0.27\approx 2.7$ to 1 in favor. The magic is that the log of the odds is exactly the straight line $\log\frac{p}{1-p}=\beta_0+\beta_1 x$. So $\beta_1$ is the change in log-odds for each extra unit of $x$, and $e^{\beta_1}$ is the odds ratio — the constant factor your odds get multiplied by per unit. With $\beta_1=1$, studying one more hour multiplies the odds of passing by $e^1\approx 2.7$ every time. The sim fits $\beta_0,\beta_1$ to the dots by maximum likelihood (it searches for the curve that makes the observed yeses and noes most probable), and the sliders map straight onto these ideas: True β₀ shifts the curve and moves the decision boundary $x=-\beta_0/\beta_1$; True β₁ sets the steepness; n sets how many dots; threshold picks the cutoff above which you call a point "yes."
Try this in the sim above: (1) Drag True β₁ down toward 0 and watch the S flatten into a near-flat line — the predictor stops separating yeses from noes and the AUC falls toward 0.5 (pure guessing). (2) Crank True β₁ up high and watch the curve snap into a sharp step: classes separate cleanly and AUC climbs toward 1. (3) Open the ROC tab and slide the threshold — the yellow dot walks along the curve, trading more true positives for more false positives, showing there is no single "right" cutoff without knowing which mistake costs more.
Logistic Regression — Cox 1958 / Berkson 1944
$$\boxed{P(Y=1|X=x) = \sigma(\beta_0+\beta_1 x) = \frac{1}{1+e^{-(\beta_0+\beta_1 x)}}}$$
Log-odds: $\log\frac{P(Y=1)}{P(Y=0)} = \beta_0 + \beta_1 x$ — linear in x
| Symbol | Meaning | Interpretation |
|---|---|---|
| $\sigma(z)$ | Sigmoid function | $\sigma(z)=1/(1+e^{-z})$, maps $\mathbb{R}\to(0,1)$ |
| $\beta_1$ | Log-odds slope | One unit increase in X multiplies odds by $e^{\beta_1}$ |
| $e^{\beta_1}$ | Odds ratio | Effect of one-unit increase in X on the odds |
| Deviance | $-2\log L$ | Goodness-of-fit; lower = better |
Parameters estimated by maximizing the log-likelihood:
$$\ell(\beta)=\sum_{i=1}^n\left[y_i\log\hat{p}_i+(1-y_i)\log(1-\hat{p}_i)\right]$$
No closed form — solved iteratively via Newton-Raphson (IRLS: Iteratively Reweighted Least Squares).
$\beta_1=2$: one unit increase in X multiplies the odds by $e^2\approx7.4$. Decision boundary: $P(Y=1)=0.5$ at $x=-\beta_0/\beta_1$. The logistic curve is steeper when $|\beta_1|$ is large (sharper decision).
Credit default model: β₀=−3, β₁=0.05 (debt-to-income ratio). For DTI=50:
$\log\text{-odds}=-3+0.05\times50=-0.5$, $P(\text{default})=\sigma(-0.5)=1/(1+e^{0.5})\approx0.378$
Odds ratio: $e^{0.05}\approx1.051$ — each unit increase in DTI multiplies default odds by 5.1%.
"β₁ in logistic regression is the change in probability per unit X."
β₁ is the change in LOG-ODDS per unit X. The change in probability depends on where you are on the sigmoid — it's largest near the decision boundary (P≈0.5) and smallest at the extremes. The odds ratio exp(β₁) is the multiplicative effect on odds. To get probability change at a specific point: dP/dX = β₁·P(1-P).
📖 Casella & Berger — Ch. 12
"R² has the same interpretation in logistic regression as in linear regression."
R² is not defined for logistic regression. Pseudo-R² measures (McFadden's, Nagelkerke's) are used but don't directly measure explained variance. McFadden's R² = 1 − logL_model/logL_null. Values of 0.2–0.4 indicate excellent fit — unlike linear R² where 0.8+ is typical. Use AUC-ROC, Brier score, or calibration plots instead.
📖 Hogg, McKean & Craig — Ch. 9
Using linear regression for a binary outcome
✅ Linear regression of Y∈{0,1} on X gives the Linear Probability Model — predictions can fall outside [0,1] and has heteroskedastic errors. Use logistic regression for binary outcomes to ensure P∈(0,1) and correct likelihood.
🔍 Regressing 0/1 outcome on X with ordinary linear regression.
Interpreting logistic coefficients as probability changes instead of log-odds changes
✅ β₁ = log-odds change. Effect on probability: Δp ≈ β₁·p̂·(1-p̂). For rare events (p≈0): Δp ≈ β₁·p̂. Always exponentiate to get odds ratios: exp(β₁).
🔍 Stating "β₁=0.5 means probability increases by 50%" — wrong.
Always using threshold 0.5 for classification
✅ The optimal threshold depends on the cost ratio of false positives vs false negatives. For medical screening (costly false negatives), use lower threshold (e.g., 0.2). For spam filtering (costly false positives), use higher threshold. The ROC curve shows all threshold options — choose based on the cost-benefit tradeoff.
🔍 Using threshold=0.5 for imbalanced classes where P(Y=1)=0.05.