📊 Section 1 — Interactive Simulation
Controls
Function
Display
💡 Section 2 — The Idea, Step by Step
Imagine standing on a grassy hill. Look down at your feet: which way is straight uphill? Walk that way and you climb fastest; walk along the slope and you stay level; walk the opposite way and you go down. A hill's height depends on two things — how far east you are and how far north — and at every spot there is exactly one steepest-uphill direction. The gradient is just an arrow that always points that way.
Call the hill's height $f(x,y)$ — the elevation when you are $x$ steps east and $y$ steps north. Face east and take a tiny step: the height changes at a rate we call the partial derivative $\partial f/\partial x$ (say "partial f, partial x"). The trick is simple — freeze $y$ as if it were a constant, and find the ordinary slope in $x$. Face north instead and you get $\partial f/\partial y$. These two numbers tell the whole story, because any heading is just a blend of east and north. Stack them into one vector, the gradient $\nabla f=(\partial f/\partial x,\;\partial f/\partial y)$. On the bowl $f=x^2+y^2$ at the point $(1,1)$, the east-slope is $2x=2$ and the north-slope is $2y=2$, so $\nabla f=(2,2)$ — an arrow pointing straight outward, away from the bowl's bottom: exactly "uphill".
To get the slope along any heading — a unit arrow $\mathbf{u}$ at angle $\theta$ — take the dot product $D_\mathbf{u}f=\nabla f\cdot\mathbf{u}=|\nabla f|\cos\alpha$, where $\alpha$ is the angle between your heading and the gradient. It is largest, equal to $|\nabla f|$, when you walk straight along $\nabla f$ ($\cos\alpha=1$), and it is zero when you walk at right angles to it ($\cos\alpha=0$). That perpendicular case is why the gradient always crosses the contour (equal-height) lines at $90^\circ$. In the sim the sliders Point x0 and Point y0 move where you are standing, and Direction θ swings your heading while the readout $D_\mathbf{u}f$ rises and falls like a cosine.
(1) Pick "x²+y²" and drag the point to the center — watch the gradient arrow shrink to nothing at the flat bottom; that vanishing point is a critical point. (2) Switch to "saddle" ($x^2-y^2$) and notice the gradient points uphill along one axis and downhill along the other. (3) Spin the Direction θ slider and watch the lower graph of $D_\mathbf{u}f$ trace a cosine curve that peaks exactly when your arrow lines up with the gradient.
📐 Section 3 — Partial Derivatives & Gradient
$$\nabla f(x_0,y_0)=\left(\frac{\partial f}{\partial x},\frac{\partial f}{\partial y}\right),\qquad D_{\mathbf{u}}f=\nabla f\cdot\mathbf{u}=|\nabla f|\cos\alpha.$$
Maximum $D_\mathbf{u}f=|\nabla f|$ when $\mathbf{u}=\nabla f/|\nabla f|$. Gradient is perpendicular to level curves.
| Symbol | Meaning | Formula |
|---|---|---|
| $\partial f/\partial x$ | Hold $y$ fixed, differentiate in $x$ | $\lim_{h\to0}[f(x+h,y)-f(x,y)]/h$ |
| $\nabla f$ | Gradient — steepest ascent direction | $(f_x,f_y)$ |
| $D_\mathbf{u}f$ | Rate of change in unit direction $\mathbf{u}$ | $f_x\cos\theta+f_y\sin\theta$ |
| $D=f_{xx}f_{yy}-f_{xy}^2$ | Hessian determinant | Second-order test |
Along a level curve $f(x(t),y(t))=c$: $df/dt=\nabla f\cdot(x'(t),y'(t))=0$. So $\nabla f\perp$ tangent to level curve.
At critical point $(x_0,y_0)$ where $\nabla f=0$: $D=f_{xx}f_{yy}-f_{xy}^2$. $D>0,f_{xx}>0$: min. $D>0,f_{xx}<0$: max. $D<0$: saddle. $D=0$: inconclusive.
$f_x=2,f_y=2$, $|\nabla f|=2\sqrt2$. At $\theta=45°$: $D_\mathbf{u}f=(2+2)/\sqrt2=2\sqrt2$ — maximum. At $(0,0)$: $\nabla f=0$ (critical). $D=4>0,f_{xx}=2>0$ — global minimum.
If $f_{xy}$ and $f_{yx}$ are both continuous: $f_{xy}=f_{yx}$. This is used to verify the Hessian matrix is symmetric.
Update $\mathbf{x}\leftarrow\mathbf{x}-\eta\nabla f(\mathbf{x})$. Each step moves in the direction of steepest descent. Step size $\eta$ (learning rate) controls convergence speed. Used in every neural network training.
If $z=f(x,y)$, $x=x(t)$, $y=y(t)$: $dz/dt=f_x(dx/dt)+f_y(dy/dt)$. More generally for $z=f(x(s,t),y(s,t))$: $z_s=f_x x_s+f_y y_s$, $z_t=f_x x_t+f_y y_t$.
❓ Section 4 — FAQ
Yes. $f(x,y)=xy/(x^2+y^2)$ has $f_x(0,0)=f_y(0,0)=0$ but along $y=x$: $f=1/2\neq f(0,0)=0$. Not continuous, hence not differentiable. Differentiability requires the stronger condition that partial derivatives are continuous in a neighborhood.
Key takeaway: $f_x,f_y$ existing does NOT imply differentiability. Need continuous partials.Each arrow points in the direction of steepest ascent. Length proportional to $|\nabla f|$. Arrows are always perpendicular to the colored contour lines. Click/drag the canvas to move the evaluation point and see directional derivative update in real time.
Key takeaway: gradient arrows perpendicular to contours — a fundamental geometric property.ML: gradient descent trains every neural network. Computer vision: $|\nabla I|$ detects edges (Sobel filter). Physics: $\mathbf{E}=-\nabla V$ (electric field), $\mathbf{F}=-\nabla U$ (conservative force). Heat: $\mathbf{q}=-k\nabla T$ (Fourier law).
Key takeaway: gradient descent, edge detection, electric field, heat flow — all $\nabla f$ in application.At $\nabla f=0$, every direction gives $D_\mathbf{u}f=0$ — no unique steepest direction. The Hessian determines whether it is a min, max, or saddle. For the Rosenbrock function, the gradient is tiny along the "banana valley" making gradient descent very slow near the minimum.
Key takeaway: steepest ascent direction unique iff $\nabla f\neq 0$. At critical points, Hessian classifies the type.Treat $y$ as constant: by product rule, $\partial/\partial x[x^2\sin(xy)]=2x\sin(xy)+x^2\cos(xy)\cdot y=2x\sin(xy)+x^2y\cos(xy)$. For $\partial f/\partial y$: $x^2\cos(xy)\cdot x=x^3\cos(xy)$. Check Clairaut: $f_{xy}=f_{yx}$ for smooth $f$.
Key takeaway: freeze all other variables and apply single-variable rules. Clairaut guarantees $f_{xy}=f_{yx}$.On $(M,g)$: $\nabla^g f$ is the unique vector satisfying $g(\nabla^g f,v)=df(v)$ for all tangent vectors $v$. The metric $g$ changes the gradient. In ML, the "natural gradient" uses the Fisher information metric — gives faster convergence than the Euclidean gradient for probability distributions.
Key takeaway: gradient depends on metric. Natural gradient (Fisher metric) is used in modern ML for faster convergence.