From "how surprised am I?" to bits of information
Play a guessing game. If a friend flips a fair coin, you genuinely cannot call it — every flip is a little jolt of surprise. If instead they flip a trick coin that lands heads $99$ times out of $100$, "heads" is boring; you already knew. Entropy is simply the average amount of surprise in a situation. Predictable things carry low entropy; wild, unpredictable things carry high entropy.
We measure surprise in bits. The surprise of an outcome with probability $p$ is $\log_2(1/p)$: a one-in-two event costs $1$ bit, a one-in-eight event costs $3$ bits, a near-certain event costs almost nothing. Average that surprise over every outcome and you get Shannon entropy:
$$H(X) = -\sum_x p(x)\,\log_2 p(x).$$
A fair coin: $H = -\left(\tfrac12\log_2\tfrac12 + \tfrac12\log_2\tfrac12\right) = 1$ bit. A lopsided $90/10$ coin: $H \approx 0.47$ bits — less uncertainty, so on average fewer bits are needed to record each flip.
Now suppose your data truly come from a distribution $P$ but you encode them with the wrong model $Q$. The wasted bits are the Kullback–Leibler divergence:
$$D_{KL}(P\,\|\,Q) = \sum_x p(x)\,\log\frac{p(x)}{q(x)} \ge 0,$$
which equals zero only when $Q=P$ and is not symmetric: in general $D_{KL}(P\|Q)\neq D_{KL}(Q\|P)$. For continuous variables the sum becomes an integral. The simulation above draws a Gaussian, and a Gaussian's (differential) entropy is $H=\tfrac12\log\!\left(2\pi e\,\sigma^2\right)$ — it depends only on the spread $\sigma$, never on where the peak sits. So here the p1 slider sets the mean (location) and p2 sets $\sigma$: p2 is effectively your entropy dial.
① Drag p2 down to a narrow spike, then up wide. Narrow means low spread and low entropy (outcomes are predictable); wide means high entropy, because $\tfrac12\log(2\pi e\,\sigma^2)$ grows with $\sigma$. ② Slide p1 (the mean) back and forth with p2 held fixed: the bell glides sideways but its width — and therefore its entropy — never changes. Entropy ignores where a distribution sits and cares only about how spread out it is. ③ Hit +1000 and watch the histogram fill the curve: the tighter the bell, the more confidently you could have guessed each new sample in advance.
Entropy & KL Divergence — Information theory fundamentals
$$H(X) = -\sum_x p(x)\log p(x) \quad\text{(Shannon entropy)}$$
$D_{KL}(P\|Q) = \sum_x p(x)\log\frac{p(x)}{q(x)} \geq 0$
This distribution has important theoretical and applied properties. See the simulation tabs to explore how shape changes with parameters.
Machine learning: cross-entropy loss (= log loss for classification). Information theory: data compression, channel capacity. Bayesian inference: KL divergence between posterior and prior. Variational inference: ELBO = E[log p] - KL(q | p). Natural language processing: perplexity. KL divergence measures how much information is lost when approximating P with Q.
Use the interactive simulation above to explore how the PDF shape responds to parameter changes. The Live Sampling tab demonstrates convergence of empirical frequencies to theoretical probabilities.
Always verify parameter definitions before computing probabilities.
Different textbooks and software packages use different parameterizations. Always check which convention is being used (e.g., rate vs scale for exponential/gamma; shape vs rate for beta). Plug in extreme values and check that the PDF shape matches your expectation.
See the simulation to visualize how parameters affect the distribution shape.