From "lucky flukes" to controlling false discoveries

Start — the lucky-coin problem. Flip a single coin ten times and getting eight heads feels surprising. But hand out twenty coins and look for a "lucky" one, and a couple will land 8-out-of-10 heads purely by chance. Nothing about those coins is special — you just gave luck many tries. A "psychic" who guesses the card of a hundred strangers will get a handful right, and a lab that screens thousands of genes will find some that look important even when none truly are. The more questions you ask, the more flukes pile up.

Build — counting the flukes. Each test has a small false-alarm rate, the significance level $\alpha$ (commonly $0.05$). Run $m$ independent tests and the expected number of false alarms is just $m\alpha$. With $m=100$ tests at $\alpha=0.05$ you expect about $5$ false positives before any real effect exists. The chance of getting at least one false alarm — the family-wise error rate — is $\text{FWER}=1-(1-\alpha)^m$. For $m=100$ that is $1-0.95^{100}\approx 0.994$: a false discovery is nearly guaranteed. The simplest fix is Bonferroni: shrink each test's threshold to $\alpha/m$, so the whole family of tests together stays below $\alpha$.

Deepen — discoveries instead of guarantees. Bonferroni controls the FWER but is strict: with $m=20{,}000$ genes, a threshold of $0.05/20000$ rejects almost nothing and you lose real findings. Benjamini-Hochberg instead controls the False Discovery Rate, $\text{FDR}=E\!\left[V/R\right]$, where $V$ is false positives and $R$ is total rejections — the expected fraction of your "discoveries" that are wrong. The recipe: sort the $m$ p-values, find the largest rank $k$ with $p_{(k)}\le \frac{k}{m}q$, and reject every test up to $k$. You accept a controlled sprinkle of false positives ($q$, say $10\%$) in exchange for far more true discoveries.

Try this in the sim above. The plot is the null sampling distribution — what a test statistic looks like when nothing is really going on. (1) Move Query x out toward the tail and watch the Query prob card: that tail probability is like a single test's p-value. (2) Press +1000 a few times and watch the green histogram fill in the bell — each sample is one "experiment," and a few always land far out in the tails by pure chance. Those stragglers are your false positives. (3) Notice that roughly $5\%$ of samples fall beyond about $1.64$ standard deviations — exactly the per-test false-alarm rate that multiplies into many false discoveries once you run $m$ tests at once.

Multiple Testing — Bonferroni, FDR — Controlling error rates when testing many hypotheses

$$\text{FWER} = P(\text{any false positive}) \leq \alpha \quad\text{(Bonferroni: use } \alpha/m)$$

$\text{FDR} = E\!\left[\frac{\text{false positives}}{\text{total rejections}}\right] \leq q \quad\text{(Benjamini-Hochberg)}$

Use the interactive simulation above to explore how the PDF shape responds to parameter changes. The Live Sampling tab demonstrates convergence of empirical frequencies to theoretical probabilities.