Gradient Descent Playground

I reduce loss functions for a living. Here's one you can reduce yourself.

This is the entire mechanism underneath deep learning, stripped of everything else. A loss function (the curve), a parameter (the ball's position), and one rule repeated forever: measure the slope where you stand, take a step downhill, step size proportional to the slope times the learning rate.

Click anywhere on the curve to drop the ball, then press run. At a small learning rate it settles politely into the nearest valley, including the shallow local one on the left that isn't the best answer available. That's not a bug in this page, it's the actual fundamental limitation. Now raise the learning rate. First the ball starts overshooting the valley floor and ringing back and forth. Push further and each step lands higher than the last one started, and the whole thing leaves the visible universe. Every practitioner has produced that exact NaN in a training log. Now you can produce it on purpose.

The production version of this game is played with millions of parameters and a GPU bill. The rules don't change. Some stories from playing it for real: One Encoder, Three Jobs →