Spiking Neural Networks
& Neuromorphic Computing

The Network Activity tab animates a population of N LIF neurons as a 2D grid. Each neuron is drawn as a circle whose brightness represents its current membrane potential — dim blue when near rest, bright cyan when near threshold, flashing white/yellow when it fires. Connections between neurons are drawn as faint lines when a spike propagates. The Synchrony Index (SI) is computed each timestep as the variance of the population-averaged voltage divided by the mean of individual voltage variances: SI = Var(mean_V) / mean(Var_V). SI near 0: neurons fire asynchronously (normal cortical state — each neuron has its own rhythm). SI near 1: neurons fire synchronously (epilepsy-like; also seen in gamma oscillations). Drag the "Synaptic weight w" slider up — watch the network transition from asynchronous (random-looking activity) to synchronised (waves of firing sweeping across the grid).

The major neuromorphic platforms are: Intel Loihi 2 (2021) — 1 million LIF neurons, 120 million synapses, 31 mm² TSMC 4 nm, ~1 mW for typical workloads, programmable STDP on-chip, used for robotics and sparse signal processing. IBM TrueNorth (2014) — 4096 neurosynaptic cores, 256 LIF neurons each (1M total), 4096 spike inputs/outputs per core, consumes 70 mW at full utilisation — 400 billion synaptic operations per second per watt. Intel BrainScaleS-2 (Heidelberg) — analogue mixed-signal, runs 1000× faster than biological real-time. SpiNNaker 2 (Manchester) — digital ARM-core architecture, 10 million neurons. Each chip implements LIF as: voltage register U, threshold register θ, decay coefficient β (bit-shifted multiply), synaptic weight memory, and an output spike bus. One LIF update = 1 clock cycle = ~2 ns on Loihi 2.

Rate coding represents information in the mean firing frequency over a time window (~50–100 ms). Advantages: robust to noise (averaging), simple to decode (count spikes). Disadvantages: slow, requires long time window, ignores millisecond-scale timing. Used by: primary visual cortex (orientation tuning curves), motor cortex (force encoding). Temporal coding represents information in precise spike times, ISI patterns, or phase relative to ongoing oscillations. Advantages: fast (single spike), high capacity. Disadvantages: sensitive to noise, complex decoder needed. Used by: auditory system (sound localisation via microsecond ITD), place cells (theta phase precession), cerebellar timing. Population coding represents information in the pattern of activity across many neurons simultaneously — combines rate and temporal information. The brain almost certainly uses all three codes in different areas; current SNN research in AI predominantly uses rate coding (easier to train) but temporal coding offers 10-100× efficiency gains if training methods mature.

With random input spike trains (no correlation), a synapse experiences roughly equal numbers of LTP (pre before post) and LTD (post before pre) events. If A+ = A−, the random walk of the weight is neutral — weights drift randomly. If A+ > A−, random input causes net potentiation — all weights drift toward their maximum, eliminating selectivity and causing "runaway" potentiation (the network loses its ability to distinguish inputs). Setting A− slightly greater than A+ (by ~10–20%) creates a slight net depressive bias for uncorrelated inputs. Only when pre-post correlations are strong (the presynaptic neuron reliably predicts postsynaptic firing) does LTP win and the weight increase. This makes STDP a correlation detector: it strengthens synapses that carry predictive information and depresses those that do not. This is the biological implementation of Bayesian inference and causal learning — STDP preferentially strengthens "causal" synapses.

The spike emission function S(U) = Θ(U−θ) (Heaviside step) has a gradient that is zero everywhere except at U = θ, where it is undefined (technically a Dirac delta). During backpropagation, the chain rule requires \(\partial \mathcal{L}/\partial W = (\partial \mathcal{L}/\partial S)(\partial S/\partial U)(\partial U/\partial W)\). The middle term \(\partial S/\partial U\) is zero almost everywhere, giving zero gradients — the "dead neuron" problem. The surrogate gradient replaces \(\partial S/\partial U\) with a smooth proxy during the backward pass only: the forward pass still uses the hard Heaviside. A good surrogate should: (1) be non-zero near threshold to pass gradients; (2) fall to zero far from threshold (to preserve sparsity); (3) be computationally cheap. Common choices: fast sigmoid ≈ 1/π(1+πU)², piecewise linear ≈ max(0,1−|U|), arctan ≈ 1/π(1+U²). The choice matters: too wide → dense gradients, too narrow → vanishing gradients.

In a standard ANN, every neuron in every layer computes a multiply-accumulate (MAC) operation for every input sample — energy scales as O(N_neurons × N_weights). In an SNN, a synapse only needs to be updated when a presynaptic spike arrives — energy scales as O(N_spikes × N_weights/neuron). Since neurons fire at ~1–5% of their maximum rate (sparsity ~95–99%), SNN energy ≈ sparsity × ANN energy. Theoretical advantage: if sparsity = 1% and energy per accumulate (AC, no multiply needed for binary spikes) = 0.1 × MAC energy, then SNN energy ≈ 0.01 × 0.1 = 0.001 × ANN energy = 1000× advantage. In practice, overhead (routing, memory, control logic) reduces this to 10–100× on current chips. The "multiply" is eliminated because spike weights are summed, not multiplied with activations — this is the key arithmetic simplification. However, SNNs require multiple timesteps per inference (typically 10–100), which partially offsets the per-spike efficiency.

Current SNNs typically achieve 80–85% accuracy on ImageNet vs 85–90% for comparable ANNs — a gap of 5–10%. The main challenges are: (1) Temporal credit assignment — errors must be propagated not just across layers (spatial) but also across timesteps (temporal), requiring backpropagation through time (BPTT), which is memory-intensive. (2) Surrogate gradient approximation error — the surrogate introduces gradient noise; optimising the surrogate shape is still an active research area. (3) Multi-timestep inference — SNNs require 10–100 timesteps per inference to accumulate enough spikes, adding latency. (4) Training instability — threshold annealing, batch normalisation for SNNs (tdBN), and weight initialisation remain challenging. Current frontiers: ANN-to-SNN conversion (convert pretrained ANN by mapping activations to firing rates), online learning with STDP+neuromodulation (reward-modulated STDP), and training with exact spike timing (SLAYER, STBP). As of 2025, the SNN-ANN accuracy gap is closing rapidly.