Neuromorphic Computing Mathematics

Your brain runs on about 20 watts — less than a light bulb — yet it outthinks data centers that gulp megawatts. The trick isn't faster parts. Brain cells stay quiet and only send a tiny electrical "spike" when they actually have something to say. A chip built to copy this — staying dark until a spike arrives — is called neuromorphic ("brain-shaped"). Neuromorphic computing is really the math of paying for thinking only when something happens.

Each artificial neuron keeps a running tally called its membrane potential $U$. Inputs nudge $U$ upward; when $U$ crosses a threshold $\theta$, the neuron fires a spike ($S=1$) and the tally resets. Between spikes $U$ slowly leaks away by a decay factor $\beta$ (a number just under 1). In words: $U_{\text{new}}=\beta\,U_{\text{old}}+(\text{input})$, with the rule "fire if $U>\theta$." Because a neuron costs energy only on the step it fires, and typically only ~1 in 100 fire on any given step, the chip does a small fraction of the arithmetic a conventional AI chip does.

Precisely, the leaky integrate-and-fire update is $U[t]=\beta U[t-1](1-S[t-1])+W_{in}X[t]$ with $S[t]=\Theta(U[t]-\theta)$, where $\Theta$ is the step function and the $(1-S[t-1])$ factor resets the neuron right after it spikes. The catch for learning: $\Theta$ has zero slope everywhere (and a spike at $\theta$), so ordinary backpropagation can't flow through it. Surrogate-gradient training swaps in a smooth stand-in derivative during the backward pass only, letting deep spiking networks train like ordinary ones. In the sim above, Parameter 1 and 2 set the decay $\beta$ and threshold $\theta$, Input $I$ drives the $W_{in}X[t]$ term, and T sim sets how many timesteps $t$ run.

(1) Push Input $I$ to its maximum and Parameter 2 (threshold) to its minimum — the "always firing" regime, the worst case for energy. (2) Reverse it: tiny input, high threshold, and the neuron stays silent — a real chip would then burn almost nothing. (3) Sweep T sim and keep the takeaway in mind: in neuromorphic hardware you pay per spike, so fewer, sparser spikes mean lower energy.

Intel Loihi 2 (2021): 128 neuromorphic cores, up to ~1 million LIF neurons and ~120M synapses, on the Intel 4 process — roughly 8× the neuron capacity of first-gen Loihi (128 cores, ~131K neurons, 14nm). The original Loihi measured ~23.6 pJ per synaptic operation (SOP) — comparable to, not below, a GPU MAC (~1 pJ). The advantage is therefore architectural, not per-operation: compute is event-driven (work happens only when spikes occur), so 1–5% activity sparsity yields ~20–100× fewer operations per inference. Neuron update: 1 cycle (~6 ns); chip power on the order of milliwatts for sparse workloads.

SNN threshold Θ is non-differentiable: dS/dU = δ(U-θ). Surrogate: replace with σ'(U-θ) = (1/π)/(1+πU²). BPTT: unroll T timesteps, compute gradients backward. Latest: SNNs achieve 77% ImageNet top-1 accuracy (ANN: 80%) at 10–20% of ANN energy on Loihi. SpikGPT, SpikeFormer: transformer-scale SNNs.

Total synaptic energy E = (number of synaptic operations) × E_SOP. The decisive lever is the operation count, not the per-op cost: a Loihi SOP (~23.6 pJ) is actually a touch pricier than one GPU MAC (~1 pJ), so the win must come from doing far fewer operations. A dense ANN layer with N=1000 inputs and K=1000 outputs costs N×K = 10⁶ MACs ≈ 1 μJ per forward pass. An SNN run for T=100 steps with ~1% of neurons active each step performs T × N_active × K_fanin = 100×10×1000 = 10⁶ SOPs ≈ 24 μJ — at this density the SNN is not cheaper. Push sparsity to ~0.1% and T to ~10 steps (10×1×1000 = 10⁴ SOPs ≈ 0.24 μJ) and it beats the GPU by ~4×. Documented end-to-end advantages on genuinely sparse, event-driven tasks (keyword spotting, optical flow, sparse coding/LASSO) span a few× up to ~100×; the gain scales with how sparse and event-driven the workload is, not with the hardware alone.