From "where are my keys?" to the Cramer-Rao bound
You lose your keys somewhere in the house. A friend asks where you think they are, and you point to the one spot that best fits everything you remember — the most believable place. Maximum likelihood estimation (MLE) does exactly this with numbers: given the data you actually collected, it picks the parameter value that makes that data the least surprising.
Flip a bent coin 10 times and get 7 heads. What is its true heads-probability? Try a value and ask: "how probable is 7-out-of-10 heads if the coin were like that?" For the coin this chance is proportional to $p^7(1-p)^3$. A fair coin ($p=0.5$) makes that smaller than $p=0.7$ does, and maximizing lands exactly on $\hat p = 7/10 = 0.7$ — the observed fraction. In general the likelihood $L(\theta)=\prod_{i=1}^n f(x_i;\theta)$ is the chance of seeing your whole dataset if the parameter were $\theta$, and the MLE is the $\theta$ that maximizes it.
Multiplying many small probabilities underflows to zero on a computer, so we maximize the log-likelihood $\ell(\theta)=\sum_{i=1}^n\log f(x_i;\theta)$ instead — same maximizer, because $\log$ is increasing. Setting its derivative (the "score") to zero, $\ell'(\theta)=0$, gives the clean Normal answers $\hat\mu=\bar x$ and $\hat\sigma^2=\frac1n\sum(x_i-\bar x)^2$. How sharply $\ell$ peaks is the Fisher information $I(\theta)=E[-\ell''(\theta)]$: a sharp peak means the data pin $\theta$ down tightly. As $n$ grows, $\sqrt{n}(\hat\theta-\theta_0)\overset{d}{\to}N(0,I(\theta_0)^{-1})$, and the variance settles onto the Cramer-Rao floor $1/(nI)$ — no fair estimator beats it. The sliders set the true parameters and $n$; each "New Data" press refits.
On the Log-Likelihood Surface tab, slide $n$ upward and watch the peak grow narrower — that is Fisher information increasing. On the Cramer-Rao Bound tab, raise $n$ and see the yellow empirical-variance dots drop onto the red $1/(nI)$ line. Finally, set sigma small versus large and notice how a tighter true spread sharpens the likelihood peak and shrinks the sampling distribution of the estimate.
Maximum Likelihood Estimation — R.A. Fisher, 1912
$$\boxed{\hat{\theta}_{MLE}=\arg\max_\theta\ell(\theta)=\arg\max_\theta\sum_{i=1}^n\log f(x_i;\theta)}$$
Asymptotic normality: $\sqrt{n}(\hat\theta-\theta_0)\overset{d}{\to}N(0,I(\theta_0)^{-1})$
| Distribution | MLE | Fisher Info I(theta) |
|---|---|---|
| Normal(mu, sigma²) | $\hat\mu=\bar x$, $\hat\sigma^2=(1/n)\sum(x_i-\bar x)^2$ | $1/\sigma^2$ (for mu) |
| Exponential(lambda) | $\hat\lambda=1/\bar x$ | $1/\lambda^2$ |
| Poisson(lambda) | $\hat\lambda=\bar x$ | $1/\lambda$ |
$I(\theta)=E\!\left[-\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\right]$. For any unbiased $\hat\theta$: $\text{Var}(\hat\theta)\ge1/(nI(\theta))$. MLE achieves this bound asymptotically — it is efficient.
$\hat\sigma^2_{MLE}=(1/n)\sum(x_i-\bar x)^2$ is biased downward. The unbiased estimator divides by $n-1$. MLE trades small bias for minimum variance — often acceptable for large $n$.
"MLE is always unbiased."
MLE is asymptotically unbiased but can be biased in finite samples. Classic: sigma²_MLE = (1/n)sum(xi-xbar)² divides by n, not n-1, giving a downward-biased estimate of population variance. The unbiased S² divides by n-1.
📖 Casella & Berger — Ch. 10
Maximizing likelihood instead of log-likelihood numerically
✅ Always maximize log-likelihood. Product of 100 small probabilities numerically underflows to exactly 0. Log converts products to sums, avoiding underflow. argmax L(theta) = argmax log L(theta) since log is monotone increasing.
Treating likelihood as a probability distribution over theta
✅ L(theta|x) = f(x|theta) is a function of theta, not a probability distribution over theta. It does NOT integrate to 1 over theta. The posterior P(theta|x) = L(theta|x)*P(theta)/P(x) requires a prior P(theta) — that's Bayesian inference.