Step
Rate
Tap to drop the balls
How to play
- Pick a landscape: Bowl, Valley, Bumpy or Saddle.
- Tap anywhere on the map to drop three balls there. They race to the bottom.
- Slide the learning rate up until something blows up.
| Do this | Phone | Keyboard |
|---|---|---|
| Drop the balls | Tap or drag the map | Arrows, then Enter (on the map) |
| Play / Pause | Play | P |
| One step | Step | N |
| Race again | Restart | R |
| Learning rate | Slider | [ ] |
| Landscape | Bowl / Valley / Bumpy / Saddle | 1 to 4 |
| Racer on / off | Tap its card | Tab to it, Enter |
What's happening?
- The map is a landscape seen from above. Dark = low, light = high. Each line joins spots of the same height.
- A ball can't see the whole map. It only feels the slope right under it, then takes a step downhill. Then again.
- The learning rate is the size of each step. Too small: it crawls. Too big: it jumps over the bottom, and can bounce out for ever.
- Plain GD just steps downhill. Momentum is a heavy ball that keeps its speed. Adam takes about the same size step in every direction.
- Training an AI is the same game, in millions of directions at once. The height is the loss: how wrong the AI is.
- Want to see it for real? The Neural Net Playground runs this exact trick to teach a tiny network.
Try this: pick Bumpy. Can you find a learning rate where Plain GD gets stuck in a little dip, but Momentum rolls through to the real bottom in the middle?