How much does one measurement actually tell you?
Imagine guessing how heavy a backpack is. Lift it once and you get a rough feeling; lift it ten times and your guess gets noticeably sharper. Each lift carries a little information about the true weight. Fisher information is just a careful way of counting how much a single measurement tells you about the hidden number you are trying to pin down.
Call the hidden number $\theta$ (the true backpack weight, a coin's bias, an average rate). You collect $n$ independent measurements and form an estimate $\hat\theta$. The key relationship is simple in words: the more information each measurement carries, the less your estimate bounces around. Each observation contributes $I(\theta)$ units of information, and because they are independent the total is just $n\,I(\theta)$. The best any unbiased estimate can possibly do is a variance of $1/(nI(\theta))$.
One worked number: for a Normal measurement with $I(\theta)=1$, taking $n=100$ readings gives a best-possible variance of $1/(100\times1)=0.01$, so the typical error is $\sqrt{0.01}=0.1$. Want to halve that error? You need four times the data, because the bound shrinks like $1/n$.
The score is the slope of the log-likelihood, $s(x;\theta)=\partial_\theta\log f(x;\theta)$. It has mean zero, and its variance is the Fisher information, which also equals the expected curvature, $I(\theta)=-E[\partial_\theta^2\log f]$. A log-likelihood that peaks sharply (high curvature) means the data strongly disagrees with nearby values of $\theta$ — lots of information, tight estimate. The Cramér-Rao bound makes this exact: $\text{Var}(\hat\theta)\ge 1/(nI(\theta))$. In the sim, the distribution dropdown changes the shape of $I(\theta)$, the True theta slider moves where you sit on that curve, and the n slider scales the bound as $1/n$.
Pick Normal and slide $\theta$: the Fisher Info curve stays flat at $I=1$ — for a Normal mean, every value is equally easy to estimate. Switch to Bernoulli and push $\theta$ toward 0 or 1: watch $I(\theta)=1/(\theta(1-\theta))$ spike — a near-certain coin is pinned down with very few flips. Finally raise n, open the CRB vs Empirical tab, and click Run 500 MLEs: the gold dots fall right onto the red Cramér-Rao line, showing the maximum-likelihood estimator actually reaches the theoretical floor.
Fisher Information & Cramér-Rao Bound — Fisher 1922 / Cramér 1946 / Rao 1945
$$I(\theta)=E\!\left[\left(\frac{\partial}{\partial\theta}\log f(X;\theta)\right)^2\right]=-E\!\left[\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\right]$$
$$\boxed{\text{Cramér-Rao:}\quad\text{Var}(\hat\theta)\ge\frac{1}{nI(\theta)}}\quad\text{for any unbiased }\hat\theta$$
| Distribution | I(theta) | CRB for n obs | MLE achieves CRB? |
|---|---|---|---|
| Normal(theta, sigma²) | $1/\sigma^2$ | $\sigma^2/n$ | Yes — exactly |
| Bernoulli(theta) | $1/(\theta(1-\theta))$ | $\theta(1-\theta)/n$ | Yes — exactly |
| Poisson(theta) | $1/\theta$ | $\theta/n$ | Yes — exactly |
| Exponential(theta) | $1/\theta^2$ | $\theta^2/n$ | Yes — exactly |
Fisher information = expected curvature of the log-likelihood. High $I(\theta)$: log-likelihood is sharply peaked → data is very informative → estimates are precise. Low $I(\theta)$: log-likelihood is flat → data is uninformative → estimates are imprecise. The score function $s(x;\theta)=\partial\log f/\partial\theta$ has mean 0 and variance $I(\theta)$.
An estimator is efficient if it achieves the CRB: $\text{Var}(\hat\theta)=1/(nI(\theta))$. Efficiency $= 1/(nI(\theta)\cdot\text{Var}(\hat\theta))\in(0,1]$. MLE is asymptotically efficient for exponential family distributions. The sample median has efficiency $2/\pi\approx0.637$ relative to sample mean for Normal — it wastes 36% of information.
Normal(theta,1): I(theta)=1 always. For n=100 observations:
CRB = 1/(100×1) = 0.01. MLE = xbar with Var(xbar) = 1/100 = 0.01. Efficiency = 100%.
Compare: sample median has Var ≈ pi/(2n) = 0.01571. Efficiency = 0.01/0.01571 = 63.7%.
"The CRB always gives the minimum achievable variance."
CRB is only a LOWER BOUND for UNBIASED estimators. Biased estimators can have lower MSE than the CRB by trading variance for bias. The CRB applies only within the class of unbiased estimators. Also: CRB can be unachievable even among unbiased estimators (when no UMVUE exists).
📖 Casella & Berger — Ch. 7.3
Computing Fisher information without checking regularity conditions
✅ CRB requires: (1) support doesn't depend on theta, (2) derivative and integral can be exchanged, (3) log-likelihood is twice differentiable. The Uniform(0,theta) distribution violates condition (1) — CRB doesn't apply, and the MLE achieves a faster 1/n² rate instead of 1/n.
🔍 Applying CRB to Uniform(0,theta) — support boundary moves with theta, violating regularity.
Confusing Fisher information matrix (FIM) with Hessian of log-likelihood
✅ FIM = E[-Hessian of log-likelihood] = E[score * score']. For a single sample, -d²logf/dtheta² is the observed Fisher information. The FIM is the expected value. For large n, the observed FIM ≈ expected FIM.
🔍 Using observed curvature from one sample as Fisher information without taking expectation.