AI/Biz

152 Layers Deep, and Easier to Train Than 20

Three years ago I wrote about a neural network winning ImageNet with eight layers and calling it deep. In December, a team at Microsoft Research won the same competition with 152 layers and a top-5 error rate of 3.57%.

For context, the commonly cited human error rate on this task is somewhere around 5%. The machines have quietly gone past us on this particular problem, and hardly anyone outside the field noticed.

But the depth isn’t the interesting part. The interesting part is why nobody had done it sooner, because the answer was not what everyone assumed.

The Problem With Deep

Everyone knew deeper networks should be better. More layers means more capacity to build up complex features from simple ones. The field had been stacking layers steadily since 2012 and it had been working.

Then it stopped working. Past a certain depth, adding layers made networks worse.

The obvious explanation is overfitting: a bigger model memorises the training set and fails on new data. Kaiming He and his co-authors checked, and that isn’t what was happening. The deeper networks were worse on the training data too. They weren’t memorising too well. They were failing to learn at all.

“A 56-layer network performing worse than a 20-layer one on data it was trained on isn’t a capacity problem. It’s an admission that we couldn’t optimise the thing we’d built.” — Sameer Gupta

This is a subtle and slightly embarrassing finding. In principle, a deeper network can always match a shallower one, by having the extra layers pass their input through unchanged. If the deep network can’t even manage that, then gradient descent simply cannot find the solution we know exists.

The Fix Is Almost Insultingly Simple

The paper’s answer is the residual connection, and it is one of those ideas that seems obvious about four seconds after you read it.

Normally a layer takes an input and produces an output. In a residual network, the layer produces an output and then the original input is added back to it. The layer isn’t learning what the answer should be. It’s learning the difference between its input and the answer.

Why that helps: if the best thing a layer can do is nothing at all, it now achieves that by outputting zeros, which is trivially easy. The identity shortcut carries the signal past it unchanged. Doing nothing became the default rather than something the network had to laboriously learn.

Think of a building with express lifts running alongside the staircase. The gradient signal, which has to travel from the output all the way back to the early layers, no longer has to descend 152 flights. It takes the shortcut.

What This Means Commercially

Two things changed, and only one of them is about accuracy.

Off-the-shelf vision is now better than a person at narrow classification. If your business process involves someone looking at an image and putting it into one of a fixed set of buckets, the honest position is that this is now a solved problem, given data. Quality inspection. Document sorting. Damage triage. Defect classification. I’d be reviewing any process that currently spends salaried hours on that.

Transfer learning is the real prize. This is the part most business readers miss, and it’s the one I’d put in front of a budget holder.

You do not have to train a 152-layer network. Somebody already did, at enormous expense, on 1.2 million images. The weights get published. You take that trained network, cut off the final classification layer, bolt on a small new one for your categories, and train only that part on your own data.

  • You need hundreds of labelled examples, not millions.
  • It runs in hours on a single GPU, not weeks on a cluster.
  • The features the big network learned on photographs of dogs and cars turn out to transfer remarkably well to X-rays, circuit boards, and satellite images.

“The expensive part of computer vision has already been paid for by somebody else. Your job is to bring a few hundred labelled examples and the good sense to know what question you’re asking.” — Sameer Gupta

That is a fundamentally different economic proposition from the one I was writing about in 2013. Back then the entry ticket was a research team and a million labelled images. Now it’s a competent engineer and a weekend.

Where I’d Still Be Careful

  • Better than human on a benchmark is not better than human in your building. ImageNet photographs are well-lit, centred, and unambiguous. Your production line camera is none of those.
  • These models are large and slow to serve. 152 layers is a lot of arithmetic per image. If you need real-time inference on cheap hardware, you’ll be trading accuracy for speed, and that trade is a design decision nobody will make for you.
  • Explainability has not improved at all. We now have a network that’s more accurate and no more interpretable. In regulated settings that gap is getting wider, not narrower.

Final Thoughts

The result that gets the headline is 3.57% and beating humans. The result that will actually show up in company accounts is that a small team can now fine-tune somebody else’s enormous model on a modest dataset and get something that works.

I keep coming back to the same conclusion in this field. The frontier is interesting. The second-hand frontier, the part that gets published, packaged, and handed to everyone eighteen months later, is where the money is.