Imagine you pull a coin from your pocket and wonder whether it is fair. Before flipping, you would probably guess it lands heads about half the time — that hunch is your starting point. Then you flip it ten times and get seven heads, so you lean toward "maybe it favors heads," but you don't throw your original hunch away completely. Bayesian inference is just this everyday habit, written down precisely: start with a belief, look at evidence, update.
The thing we are unsure about is the coin's true chance of heads, call it $p$. Our starting belief is the prior; the evidence is the data ($k$ heads in $n$ flips); the updated belief is the posterior. Bayes' rule says the posterior is proportional to the prior times the likelihood — "what you believed" combined with "how well each value of $p$ explains the data." For a coin there is a tidy shortcut: if your prior is $\text{Beta}(\alpha,\beta)$ and you observe $k$ heads in $n$ flips, the posterior is simply $\text{Beta}(\alpha+k,\ \beta+n-k)$. You literally add your observed successes and failures onto the prior's counts. With a weak prior $\text{Beta}(2,2)$ and 7 heads in 10 flips you get $\text{Beta}(9,5)$, whose mean $9/14\approx0.64$ sits between the prior guess (0.5) and the raw data rate ($7/10=0.7$).
Going further, the prior's total $\alpha+\beta$ behaves like a "virtual sample size." The posterior mean is the weighted average $\frac{\alpha+\beta}{\alpha+\beta+n}\cdot\frac{\alpha}{\alpha+\beta}+\frac{n}{\alpha+\beta+n}\cdot\frac{k}{n}$, blending prior mean and data rate. A large $\alpha+\beta$ resists the data; a flat prior lets the data take over almost immediately, and as $n\to\infty$ the posterior concentrates on the true $p$ no matter where you started. The sliders map directly onto this: Prior α and Prior β set your starting curve, "True p" controls the hidden coin the flips are drawn from, and every flip nudges the green posterior.
Try this in the sim above: (1) Set Prior α=1 and Prior β=1 for a totally flat prior, then click "Flip 100" and watch the posterior snap onto the true $p$. (2) Now set α=20, β=20 for a stubborn prior and flip just 5 — see how little it moves. (3) Switch to the "Update Sequence" tab and toggle "Show likelihood" to watch the posterior ride between the prior and the data as evidence builds.
Bayes' Theorem — Thomas Bayes 1763 / Richard Price / Laplace 1812
$$\boxed{p(\theta|\mathbf{x}) = \frac{p(\mathbf{x}|\theta)\,p(\theta)}{p(\mathbf{x})} \propto p(\mathbf{x}|\theta)\,p(\theta)}$$
Beta-Binomial Conjugate: Prior $\theta\sim\text{Beta}(\alpha,\beta)$, observe $k$ heads in $n$ flips, then:
$$\theta|\mathbf{x}\sim\text{Beta}(\alpha+k,\;\beta+n-k)$$
| Symbol | Meaning | Interpretation |
|---|---|---|
| $\theta$ | Parameter (coin bias p) | What we want to learn about; range [0,1] |
| $p(\theta)$ | Prior distribution | Belief about $\theta$ BEFORE seeing data |
| $p(\mathbf{x}|\theta)$ | Likelihood | Probability of observed data given $\theta$ |
| $p(\theta|\mathbf{x})$ | Posterior distribution | Updated belief about $\theta$ AFTER seeing data |
| $p(\mathbf{x})$ | Marginal likelihood | Normalizing constant; ensures posterior integrates to 1 |
| $\text{Beta}(\alpha,\beta)$ | Beta distribution | Conjugate prior for binomial likelihood |
Start with $\theta\sim\text{Beta}(\alpha,\beta)$: $p(\theta)\propto\theta^{\alpha-1}(1-\theta)^{\beta-1}$. The Beta distribution on $[0,1]$ is the natural prior for a probability. $\alpha-1$ represents "pseudo successes", $\beta-1$ represents "pseudo failures" in prior experience.
Observe $k$ heads in $n$ flips: $p(k|\theta,n)=\binom{n}{k}\theta^k(1-\theta)^{n-k}$. As a function of $\theta$ with $k,n$ fixed, the likelihood is $L(\theta)=\theta^k(1-\theta)^{n-k}$.
$$p(\theta|k)\propto L(\theta)\cdot p(\theta)=\theta^k(1-\theta)^{n-k}\cdot\theta^{\alpha-1}(1-\theta)^{\beta-1}=\theta^{(\alpha+k)-1}(1-\theta)^{(\beta+n-k)-1}$$
This is exactly the kernel of $\text{Beta}(\alpha+k,\beta+n-k)$! Conjugacy means prior and posterior have the same functional form.
$$E[\theta|k] = \frac{\alpha+k}{\alpha+\beta+n}, \qquad \text{Mode} = \frac{\alpha+k-1}{\alpha+\beta+n-2}$$
The posterior mean is a weighted average: $\frac{\alpha+k}{\alpha+\beta+n} = w\cdot\frac{\alpha}{\alpha+\beta} + (1-w)\cdot\frac{k}{n}$ where $w=\frac{\alpha+\beta}{\alpha+\beta+n}$. As $n\to\infty$, the data (MLE $k/n$) dominates; with small $n$, the prior dominates.
The 95% highest density interval (HDI) $[l,u]$ satisfies $\int_l^u p(\theta|k)\,d\theta = 0.95$. Unlike frequentist CI, the Bayesian credible interval HAS the interpretation: "There is 95% posterior probability that $\theta$ lies in $[l,u]$" — given the prior and data.
Problem: Prior belief: Beta(2,2) (weak prior centered at 0.5). Observe 7 heads in 10 flips.
Posterior: $\theta|k\sim\text{Beta}(2+7, 2+3)=\text{Beta}(9,5)$
$$E[\theta|k] = \frac{9}{14} \approx 0.643, \qquad \text{Mode} = \frac{8}{12} \approx 0.667$$
The posterior mean 0.643 is between the prior mean (0.5) and the MLE (7/10=0.7) — the prior pulled the estimate toward 0.5 slightly. The 95% credible interval is approximately [0.37, 0.88].
Try this: set Prior α=2, Prior β=2, flip 10 coins and let 7 be heads. Watch the posterior shift!
"The posterior is the same as the likelihood."
The posterior $p(\theta|x)$ combines likelihood AND prior: $p(\theta|x)\propto p(x|\theta)\cdot p(\theta)$. The likelihood $L(\theta)=p(x|\theta)$ is an unnormalized function of $\theta$ for fixed data — it is not a probability distribution and doesn't integrate to 1 over $\theta$. The posterior is the properly normalized product. With a flat (uniform) prior, posterior ∝ likelihood, but they are still conceptually distinct.
📖 Gelman et al. — BDA, Ch. 2
"With enough data, the prior doesn't matter."
This is approximately true for proper priors and well-identified models: the likelihood eventually dominates. But for improper priors (like Jeffreys' prior), the posterior may be improper. For high-dimensional parameters or small data, the prior can heavily influence results. And in hierarchical models, priors on hyperparameters can matter even with large datasets.
📖 Gelman et al. — BDA, Ch. 3
"Bayesian credible intervals and frequentist confidence intervals have the same interpretation."
A 95% Bayesian credible interval [l,u] means P(θ∈[l,u]|data)=0.95 — this IS a direct probability statement about θ. A 95% frequentist CI does NOT — it means the procedure has 95% coverage. They are constructed differently and give different intervals for the same data (except in special cases with flat priors).
📖 Casella & Berger — Statistical Inference, Ch. 9.2
Forgetting the normalizing constant and treating unnormalized posterior as a distribution
✅ The posterior $p(\theta|x)=p(x|\theta)p(\theta)/p(x)$. The denominator $p(x)=\int p(x|\theta)p(\theta)d\theta$ is the evidence. For conjugate families it cancels analytically, but in general MCMC is needed. Never use $p(x|\theta)p(\theta)$ directly as a distribution without normalizing.
🔍 Students write $p(\theta|x)\propto p(x|\theta)p(\theta)$ and then compute probabilities from it without normalizing.
Confusing the prior parameters with data counts in Beta-Binomial
✅ In Beta(α,β), α and β are hyperparameters (pseudo-counts), not actual observed data. After observing k successes in n trials: posterior is Beta(α+k, β+n-k). The data adds to, not replaces, the prior parameters.
🔍 Students write Beta(k, n-k) ignoring the prior, or Beta(α+k, β+n) without subtracting k.
Using an improper prior without checking posterior properness
✅ Improper priors (like uniform on [0,∞)) can lead to improper posteriors (don't integrate to 1). Always verify $\int p(x|\theta)p(\theta)d\theta < \infty$ before using an improper prior for inference.
🔍 Students use flat priors everywhere without checking properness conditions.