Gradient descent on a loss surface
Contour lines join points of equal loss; the stronger the tint, the lower the loss. Tap anywhere to start there.
One minimum, but steeper in one direction. Too large a step makes the path zig-zag.
Tap to choose a start
Tap the map to choose a starting point, or press Step. The arrow points downhill: the direction of the negative gradient.
Steps0
Loss11.22
Slope (gradient size)11.4978
Try this
- On the Stretched bowl, compare step sizes 0.1, 0.2, and 0.26.
- On Many valleys, tap several starting points and see which valley each one finds.
- On Banana valley, turn on Fast forward and watch the loss chart flatten into a long slide.
The same idea with a single parameter:
Downhill in one dimension
The ball shows the current parameter value. Drag it to choose a starting point, then step.
The slope is negative (uphill to the left), so the step goes right. The arrow along the bottom shows the next step: step size times slope, 0.3 times -2.5.
Steps0
Parameter x-2.5
Loss3.125
Slope-2.5