Backpropagation

Gradient descent, from the machine learning track, says: compute the GradientThe list of slopes of a function, one per parameter. It points in the direction in which the function increases fastest, so its opposite points downhill.Open in glossary of the loss with respect to every parameter, then step each parameter a little way downhill. For a line with two parameters that is easy. A network has thousands to billions of weights, and the loss depends on each of them through a long chain of layers. BackpropagationThe algorithm that computes the gradient of the loss with respect to every weight in a network in one backward pass, by applying the chain rule from the loss back to the inputs.Open in glossary is the algorithm that gets all of those gradients efficiently. It is one idea from calculus, applied very systematically.

Break the calculation into small pieces

Here is a single neuron with two inputs, a tanh activation, and a squared-error loss against a target $y$ :

z = w_1 x_1 + w_2 x_2 + b, \qquad a = \tanh(z), \qquad L = (a - y)^2.

Write this as a Computational graphA diagram of a calculation as simple operations connected by the values flowing between them. Each operation knows its own local derivative, which makes backpropagation mechanical.Open in glossary: every multiplication, addition, tanh, and square becomes its own node, with arrows showing which values feed which. Each node does something simple, and for something simple we know the derivative. Those are local derivatives: how much a node’s output changes when one of its inputs changes slightly, ignoring everything else in the graph.

Node	Local derivative
$p = w \cdot x$	$\partial p / \partial w = x$ and $\partial p / \partial x = w$
$s = p + q$	$\partial s / \partial p = \partial s / \partial q = 1$
$a = \tanh z$	$\partial a / \partial z = 1 - a^2$
$d = a - y$	$\partial d / \partial a = 1$
$L = d^2$	$\partial L / \partial d = 2d$

Chain the pieces together

The Chain ruleThe calculus rule for nested functions: if L depends on a and a depends on z, then dL/dz = dL/da times da/dz. Backpropagation applies it over and over.Open in glossary says how to combine local derivatives. If $L$ depends on $a$ , and $a$ depends on $z$ , then

\frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z}.

Backpropagation applies this one node at a time, starting from the loss. The loss’s gradient with respect to itself is 1. Then each node takes the gradient arriving from above, multiplies it by each of its local derivatives, and passes the results down to its inputs. Nodes are visited in reverse order of the forward pass, so a node is processed only after everything that uses it.

Backpropagation, one node at a time

A single tanh neuron with squared-error loss. Run the forward pass, then send gradients backward with the chain rule.

Press Next step to run the forward pass: each node computes its value from its inputs.

Each box shows a value and, once the backward pass reaches it, its gradient ∂L/∂(that value) after the ∂ sign. Inputs x₁ = 1.000, x₂ = −0.500, target y = 1.000. Highlighted boxes are the parameters the network can change.

Weight w₁0.40

Weight w₂0.80

Bias b−0.30

Step0 of 15

Loss Lnot computed yet

Try this

Press Next step until the loss appears. That is the forward pass: values flow left to right (top to bottom on a phone).
Keep stepping. Gradients now flow backward from L. At each step, check that the new gradient is the one arriving from above times the local derivative shown in the explanation.
Notice the addition nodes pass their gradient through unchanged, while the multiplication nodes scale it by the other input. Compare the gradients of $w_1$ and $w_2$ : they differ by exactly the ratio of $x_1$ to $x_2$ .
When the backward pass finishes, press Update weights. Each weight moves against its gradient, and the next forward pass gives a smaller loss. That is one step of training.

Putting the whole chain together for $w_1$ :

\frac{\partial L}{\partial w_1} = \underbrace{2(a - y)}_{\partial L / \partial a} \cdot \underbrace{(1 - a^2)}_{\partial a / \partial z} \cdot \underbrace{x_1}_{\partial z / \partial w_1}.

With the demo’s starting values ( $x_1 = 1$ , $x_2 = -0.5$ , $w_1 = 0.4$ , $w_2 = 0.8$ , $b = -0.3$ , $y = 1$ ), the neuron outputs $a \approx -0.29$ , far from the target of 1, and the loss is about 1.67. The gradient for $w_1$ comes out negative, which says increasing $w_1$ would lower the loss, so gradient descent raises it. One update with learning rate 0.5 brings the loss down to nearly zero.

Three patterns worth remembering

Addition routes. A sum sends the same gradient to every input.
Multiplication swaps. The gradient for one factor is the upstream gradient times the other factor. That is why a weight’s gradient is proportional to the input it multiplies: an input of zero means the weight had no effect, so its gradient is zero.
Branches add up. If one value feeds several places, its gradient is the sum of the gradients coming back from each. The demo’s engine accumulates gradients this way, and the tests check it.

Why this is fast

There is a brute-force alternative: nudge one weight by a tiny amount, rerun the network, and see how much the loss moved. That needs a fresh forward pass for every weight. For a network with a million weights, that is a million forward passes for a single training step.

Backpropagation reuses the values saved during the one forward pass and sweeps backward once. The total cost is a small constant multiple of a forward pass, usually quoted as about two to three times, no matter how many weights there are. That efficiency is what makes training large networks possible at all.

Backpropagation through a whole layer

For a layer $\mathbf{z} = W\mathbf{x} + \mathbf{b}$ , write $\boldsymbol{\delta} = \partial L / \partial \mathbf{z}$ for the gradient arriving from above. The same chain rule, done for every entry at once, gives

\frac{\partial L}{\partial W} = \boldsymbol{\delta}\,\mathbf{x}^{\top}, \qquad \frac{\partial L}{\partial \mathbf{b}} = \boldsymbol{\delta}, \qquad \frac{\partial L}{\partial \mathbf{x}} = W^{\top} \boldsymbol{\delta}.

The weight gradient is an outer product of the upstream gradient and the input, which is the “multiplication swaps” rule for every weight at once. The input gradient, $W^{\top}\boldsymbol{\delta}$ , is what gets passed down to the previous layer. Through an activation $\mathbf{h} = \sigma(\mathbf{z})$ , the gradient is multiplied elementwise by $\sigma'(\mathbf{z})$ .

Where backpropagation came fromOptional

The underlying technique, reverse-mode automatic differentiation, was described by Seppo Linnainmaa in 1970. Paul Werbos discussed applying it to neural networks in his 1974 thesis and published the first such application in 1982. It became widely known after David Rumelhart, Geoffrey Hinton, and Ronald Williams showed in a 1986 paper that networks trained this way learn useful internal representations.

Modern libraries such as PyTorch and JAX do exactly what the demo does, at scale: as you run the forward pass they record the computational graph, and calling for gradients walks it backward. You almost never write a backward pass by hand, but knowing what it computes explains a lot, for instance why gradients can shrink to nothing through many saturated tanh layers (each step multiplies by a slope near zero).

Key ideas

A network’s computation is a graph of simple operations, each with an easy local derivative.
The chain rule multiplies local derivatives along the path from the loss back to each parameter.
Backpropagation visits nodes in reverse order, so each gradient is complete before it is passed on.
Sums pass gradients through, products scale them by the other factor, and branches add their gradients.
One backward pass gives every gradient for about the cost of a few forward passes.