AI/Biz

The Boring Paper That Will Save You Money

Most of what gets written about machine learning is about capability. What can the machine do now that it couldn’t do last year.

I want to write about something much less exciting and considerably more useful to anyone with a budget: how much human effort it takes to get a model to train at all.

In December, Diederik Kingma and Jimmy Ba published a method called Adam. It will not appear in any newspaper. It is one of the most economically significant papers of the last two years.

The Problem Nobody Puts in the Press Release

Here’s what actually happens when a team trains a neural network.

Somebody picks a learning rate, which is the size of the step the model takes each time it adjusts itself. Too large and the model overshoots and never settles. Too small and it crawls, burning days of GPU time to get nowhere. There is no formula for the right value. You guess, you watch, you guess again.

Then you discover the right value changes over the course of training. So you add a schedule that decays it. Now you have more knobs.

Then you discover different parts of the network want different values. Parameters connected to common features get updated constantly and need small steps. Parameters connected to rare features are barely touched and need large ones. One global number cannot serve both.

“The published results always show the run that worked. They never show the eleven runs before it that a person had to sit and watch fail.” — Sameer Gupta

This is where the money goes. Not the final training run. The search for a configuration where the final training run is possible.

What Adam Does

Adam, short for adaptive moment estimation, gives every parameter its own learning rate, and works them out automatically as training proceeds.

It keeps two running averages for each parameter. The first is the average recent gradient, which tells it which direction things have been moving. The second is the average recent squared gradient, which tells it how volatile that parameter has been. Divide one by the square root of the other and you get a step size that is large where the signal is consistent and small where it’s noisy.

The analogy I keep using is cruise control. You can drive across the country working the throttle by hand, and an attentive driver will do a decent job. Cruise control doesn’t beat the best possible human driver. It beats a tired one, it beats an average one, and it means the driver can think about something else.

The paper suggests defaults of 0.001 for the step size, with the two decay rates at 0.9 and 0.999. The remarkable thing, and the entire reason this matters commercially, is those defaults just work across a wide range of problems.

Why This Is a Business Story

Think about where the cost of a machine learning project actually sits.

  • Engineer time is the largest line item, and a meaningful fraction of it is spent watching training curves and adjusting settings. That is expensive people doing work that produces no intellectual property.
  • Failed runs cost the same as successful ones. A GPU spends the same money converging as it does diverging. Every abandoned run is money burnt.
  • The tuning expertise doesn’t transfer. The intuition an engineer builds for one architecture on one dataset is only partly useful on the next one, so you pay for the learning curve repeatedly.

Adam attacks all three. Not by making models smarter, but by cutting the number of attempts between starting and finishing.

“Efficiency work never gets a headline, because the result is the absence of a problem. But the absence of that problem is what turns a research project into something a normal team can ship.” — Sameer Gupta

There’s a broader point here that I keep returning to. The barrier to using these techniques in a normal company has never really been the mathematics. It’s been that the mathematics required a specialist to nurse it. Every method that reduces the nursing widens the population of teams who can use it.

The Caveats

I’ve been reading the discussion around this paper and I’d flag three things:

  • Defaults are a starting point, not an answer. Adam gets you to a reasonable result quickly. Careful hand-tuning of plain stochastic gradient descent with momentum still wins on some problems, particularly in vision.
  • It uses more memory. Two running averages per parameter means roughly three times the optimiser state. On a large model and a small GPU, that’s a real constraint.
  • It is not a substitute for understanding your data. No optimiser rescues a model trained on the wrong features. The step size was never your biggest problem.

Final Thoughts

I’ve argued before that the useful question in this field isn’t what the frontier can do, it’s what an ordinary competent team can reliably ship. Those are very different questions and only one of them shows up on a balance sheet.

Adam is a good example of the second kind of progress. Nothing became possible that wasn’t possible before. It just became routine, and routine is what scales.

If you have a team experimenting with neural networks and they’re burning weeks on tuning, ask them whether they’ve tried this. It’s a few lines of code and it will most likely give you back a fortnight.