Mutual Information & Neural Decoding

A friend secretly picks one of a few flashcards and you only get to watch how a single neuron reacts — say, how fast it fires. Sometimes the firing rate tells you the card almost for sure; sometimes the neuron is so noisy that you are still guessing. Mutual information is just a fair score for that game: it measures how much watching the neuron shrinks your uncertainty about which card was shown. A useless, random neuron scores zero; a perfectly reliable one scores the full uncertainty of the deck.

First we need a ruler for uncertainty. Entropy $H$ counts the average number of yes/no questions needed to pin down an outcome, in bits. One fair coin flip is exactly $1$ bit; four equally likely cards take $2$ bits. Mutual information is the drop in that count once you have seen the response: $I(S;R)=H(R)-H(R|S)$ — the response's total uncertainty minus the uncertainty that survives after you know the stimulus $S$.

Suppose the neuron's response could land in one of four equally likely bins, so $H(R)=\log_2 4 = 2$ bits. If knowing which stimulus was shown narrows that down to two still-equal possibilities, then $H(R|S)=\log_2 2 = 1$ bit. The spike train therefore carries $I = 2-1 = 1$ bit about the stimulus — exactly one well-chosen yes/no question's worth.

Written out over the joint distribution, $I(S;R)=\sum_{s,r}p(s,r)\,\log_2\frac{p(r|s)}{p(r)}$. It is symmetric in $S$ and $R$, never negative, and equals zero exactly when stimulus and response are statistically independent. Treat the neuron as a noisy communication channel: its channel capacity $C=\max_{p(s)}I(S;R)$ is the most information it could ever convey, found by choosing the best stimulus distribution. The controls map onto this picture: Parameter 1 sharpens the tuning (more signal), Parameter 2 adds response noise (more $H(R|S)$, less $I$), Input I sets stimulus strength, and T sim sets how many trials are sampled — short runs over-estimate $I$ because of sampling bias.

① Turn Parameter 2 (noise) to its maximum and watch the response histogram blur until the stimulus and response look independent — mutual information collapses toward $0$ bits. ② Drop the noise and raise Parameter 1: the per-stimulus histograms pull apart, $H(R|S)$ shrinks, and $I$ climbs toward the channel capacity. ③ Shorten T sim and note how the estimated information creeps upward with too few trials — the classic small-sample bias the derivation below corrects for.

Estimate H(R) from response histogram, H(R|S) from per-stimulus histograms. Bias correction required (Miller–Madow): the naive estimate is biased low, so H_true ≈ H_obs + (|alphabet|-1)/(2N ln2). Requires many trials. Net systematic bias overestimates I for small N.

Wiener filter K(f) = C_sr(f)/C_rr(f). Reconstruction SNR = |K|²C_ss/[C_ss - |C_sr|²/C_rr]. Lower bound: I ≥ ½∫log(1+SNR(f))df. Upper bound from coding theorem.

Barlow (1961): neural codes maximise I subject to metabolic constraints. Predictions: response distributions maximally informative (whitened), filters matched to stimulus statistics. Explains center-surround (retina), orientation tuning (V1), sparse coding (IT cortex).