AIPre-Train Once, Fine-Tune Forever
Ask any team that has tried to build something useful with machine learning what stopped them, and the answer is almost never the algorithm. It’s the labels.
You need thousands of examples where a human has already written down the right answer. Labelling is slow, boring, expensive, and inconsistent. I’ve watched more projects die on this than on any modelling problem.
In June, a team at OpenAI published a paper called “Improving Language Understanding by Generative Pre-Training” that takes a serious run at the problem, and the approach is clean enough to explain over lunch.
Two Stages
Stage one: read everything, learn nothing in particular.
They took a Transformer, twelve layers of it, and trained it on BooksCorpus, a collection of about 7,000 unpublished books. The task was the simplest one available: given the text so far, predict the next word. Over and over, across millions of passages.
No labels. No annotation. Nobody marked anything up. Just a very large volume of ordinary prose and a very boring objective.
Stage two: point it at the actual job.
Then they took that pre-trained model and fine-tuned it on individual tasks, each with its own small labelled dataset. Question answering. Sentiment. Textual entailment. Semantic similarity.
The result: state of the art on nine of the twelve tasks they tried. Several of them by wide margins, against models that had been purpose-built for that single task by people who had thought about nothing else.
The Detail I Find Most Impressive
Here’s the part that reveals the quality of the thinking.
When you fine-tune a general model for a specific task, the usual approach is to bolt on task-specific machinery. Different architecture for question answering than for classification, and so on. Every task becomes an engineering project.
They refused to do that. Instead they left the model almost entirely alone and changed the shape of the input to fit the model. Comparing two sentences? Concatenate them with a delimiter token in between. Multiple-choice question? Run each option through separately and compare the scores.
One architecture. Different ways of arranging the text going in.
“They didn’t build twelve models for twelve tasks. They built one that had read a great deal, then explained each new job to it in a format it already understood.” — Sameer Gupta
That’s not a modelling insight, it’s an engineering one, and it’s what makes the approach practical rather than merely clever.
Why This Changes the Economics
I want to be direct about why I think this is the most commercially relevant paper I’ve read this year.
The expensive stage requires no labels, and the stage requiring labels is cheap. Pre-training is costly in compute and needs enormous text, but that text is unlabelled and therefore effectively free. Fine-tuning needs labelled data, but far less of it, and runs in hours.
Somebody else pays for stage one. This is the pattern that already transformed computer vision, where you download a network trained on millions of images and fine-tune it on a few hundred of your own. Language is now on the same path. The heavy lifting happens once, in a well-funded lab, and everybody else starts from the result.
Your unlabelled archive becomes an asset. Most companies are sitting on years of documents, tickets, contracts, and correspondence that they’ve never analysed because analysing it required labelling it. That constraint is loosening.
The analogy I’d use with a business audience: you’ve been hiring specialists and training each one from scratch for a single narrow job. This is hiring a graduate who has read very widely, then spending a week showing them your particular process. It’s not that the graduate is smarter. It’s that the general education transfers, and you only pay for it once.
The Honest Limits
- It’s a decoder reading left to right. It predicts the next word, so when interpreting a given word it only sees what came before, never what comes after. For a lot of comprehension tasks that’s a real handicap, and I’d expect someone to attack it.
- 7,000 books is not very much text. The pre-training corpus is modest. My guess is that this scales, and that the numbers in this paper are a floor rather than a ceiling.
- You still need labelled data. Less of it, but not none. Anyone selling you “no training data required” on the back of this paper is overselling.
- Nobody knows what it learned. The model absorbed the patterns, assumptions, and prejudices of 7,000 books and there is no way to inspect what came along for the ride. With GDPR live since May and regulators asking sharper questions about automated decisions, “we fine-tuned a model that read some books” is going to be an uncomfortable answer.
Final Thoughts
I’ve written before that the useful measure of progress in this field isn’t what the frontier can do, it’s what an ordinary competent team can ship. By that measure this is a bigger deal than most of what got more attention this year.
Labelled data has been the wall every practical language project runs into. This paper doesn’t remove the wall, but it lowers it substantially, and it does so with a method simple enough that it will be reproduced and packaged quickly.
If you have a text problem you shelved because labelling was unaffordable, take it off the shelf and look at it again.