AIThe Alignment Paper That Explains the Product
There’s a result in OpenAI’s InstructGPT paper from last month that should reframe how anyone thinks about the value of scale.
A 1.3 billion parameter model, tuned with their method, was preferred by human evaluators over the original 175 billion parameter GPT-3.
That’s a model over a hundred times smaller winning on the thing that actually matters, which is whether a person found the answer useful.
The Problem It Solves
I wrote when GPT-3 came out that it made a great many things easy to try and rather fewer safe to ship. This paper explains a large part of why.
A language model trained on internet text is optimised to predict the next word. That is not the same objective as being helpful. Ask it a question and it may continue with more questions, because that’s what a list of questions looks like on the internet. Ask for advice and it may produce something confident and wrong, because confident and wrong is well represented in the training data.
The model was never trained to be useful. It was trained to be plausible. Those diverge constantly, and every bit of prompt engineering everyone has been doing for two years is a workaround for that gap.
“We built a machine that predicts what text usually follows, then acted surprised that it didn’t do what we wanted. It was doing exactly what we asked. We asked for the wrong thing.” — Sameer Gupta
How It Works
Three stages, and the middle one is the clever bit.
Supervised fine-tuning. Human contractors write good answers to a set of prompts. The model is fine-tuned on those. This alone helps, and it’s limited by how many answers people can write.
Train a reward model. For a given prompt, generate several answers from the model and have a person rank them from best to worst. Then train a separate model to predict those rankings.
This is the insight worth pausing on. Writing a good answer is slow and requires expertise. Comparing four answers and saying which you prefer is fast and requires only judgement. Ranking scales in a way that writing does not, and the reward model turns a pile of comparisons into an automatic scorer.
Reinforcement learning against the reward model. Now the language model can be optimised directly for producing answers that score well, without a human in the loop for each attempt. The human preferences have been distilled into something you can run millions of times.
Why the Size Result Matters Commercially
Everyone has spent two years assuming that better output requires a bigger model, and that the frontier is defined by parameter count.
This says something different: a substantial fraction of what looks like a capability gap is actually an alignment gap. The smaller model knew enough. It just wasn’t pointed at the right objective.
The implications for anyone paying inference bills are direct:
- A well-tuned small model can beat a poorly-tuned large one on real tasks, at a fraction of the cost per request.
- The expensive resource shifts from compute to human judgement. You need people who can consistently rank answers in your domain, which is a different hiring problem and, notably, one where domain expertise beats machine learning expertise.
- Your preferences are specific to you. What counts as a good answer in a medical context, a legal one, and a customer support one are genuinely different. The generic tuning that ships with a public model encodes somebody else’s idea of good.
“The alignment step is where a general model becomes your model, and the input isn’t data in the usual sense. It’s your organisation’s judgement about what a good answer looks like.” — Sameer Gupta
What I’d Watch Carefully
I want to flag the parts that concern me, because the results are strong enough to obscure them.
Whose preferences? The paper is refreshingly direct about this. A specific, small group of contractors, with specific instructions, produced the rankings. Their judgement is now embedded in the model’s behaviour. That’s not a criticism of the method, it’s an inherent property of it, and anyone deploying such a model is inheriting a set of values they didn’t specify and can’t inspect.
Helpful and harmless pull against each other. A model optimised hard for helpfulness will help with things it shouldn’t. One optimised hard for caution refuses reasonable requests. There’s no setting that resolves this, only a trade-off somebody chose.
Sounding good is not being right. The reward model learns what people prefer, and people prefer confident, fluent, well-structured answers. That’s correlated with correctness and it is not the same thing. I’d expect this technique to make models more persuasive somewhat faster than it makes them more accurate, and that gap is where the risk lives.
Final Thoughts
This is the most commercially significant paper of the year so far, and I don’t think that’s obvious from the outside, because the headline result is about a benchmark of human preference rather than a capability.
But it changes the shape of what these systems are. A raw language model is a text predictor that you coax toward usefulness with careful prompting. A preference-tuned model is something you can hand to a person who has never heard of prompt engineering.
That’s the difference between a component and a product, and I’d expect the next twelve months to make that distinction very visible.