AI175 Billion Parameters and No Fine-Tuning
I’ve written about this line of work twice before, once when OpenAI showed that pre-training then fine-tuning beat purpose-built models, and again when Google’s BERT made the approach standard.
Both papers followed the same recipe. Pre-train a big model on unlabelled text, then fine-tune it on your labelled data for your specific task.
The paper OpenAI put out at the end of May throws away the second half of that recipe.
The Claim
GPT-3 is 175 billion parameters, which is more than a hundred times the size of the largest BERT. Ninety-six layers, a context window of 2,048 tokens, trained on roughly 300 billion tokens of text scraped from the web, books, and Wikipedia.
The size is not the interesting part. The interesting part is what they do with it.
You don’t fine-tune it. There’s no training step, no gradient updates, no labelled dataset. You write a prompt that describes what you want, include two or three examples inside the prompt itself, and the model does the task.
Translate English to French:
sea otter => loutre de mer
cheese => fromage
peppermint =>That’s the whole interface. They call it few-shot learning, and on a range of tasks it approaches or matches models that were specifically fine-tuned for them.
“For eight years the way to use these models has been to train them. This is the first time the interface is a paragraph of English written by someone who has never trained anything.” — Sameer Gupta
Why This Is a Different Kind of Announcement
I want to separate two things that are getting mixed together in the commentary.
The capability improvement is real but incremental. It’s a bigger model trained on more text and it performs better. That’s the expected result of scaling, and the paper is largely a careful measurement of exactly how performance improves with size. Useful, not shocking.
The change in who can use it is not incremental at all.
Consider what it previously took to get a text classifier into production. A machine learning engineer. A labelled dataset, which meant weeks of somebody’s time. A training pipeline. Evaluation infrastructure. A deployment story. Realistically, a quarter of work and a specialist you had to hire.
Now: a product manager writes three examples into a text box and sees whether it works before lunch.
That is not a modest efficiency gain. It’s a change in the number of people inside a company who can build something, and it’s roughly the shift that spreadsheets caused for financial modelling. The analyses didn’t get better. The population who could perform them got a hundred times larger.
What I’d Actually Do With It
- Prototype ruthlessly. The cost of testing whether a language model helps with a given workflow has collapsed. Test ten ideas in the time you used to spend scoping one.
- Attack the long tail. Every business has dozens of small text tasks that were never worth a modelling project. Reformatting supplier emails. Categorising free-text survey responses. Drafting first-pass replies. Each too small to justify an engineer, and collectively significant.
- Use it to build training data. If you eventually need a small, cheap, fast model in production, a large one can help you label the dataset to train it. That’s a sensible pipeline, and considerably cheaper to run than the large model.
The Parts That Worry Me
I’d be doing you a disservice if I stopped there, because I think the enthusiasm is running well ahead of the engineering.
It is confidently wrong. The model generates fluent, well-structured text that is sometimes simply false, and the prose gives no signal about which is which. Every previous generation of this technology produced obvious garbage when it failed. This one produces plausible garbage, which is far more dangerous in a business process.
You cannot run it. 175 billion parameters does not fit on hardware you own. This is an API, and that means your product roadmap now runs through somebody else’s pricing page, rate limits, availability, and terms of service. I’d want that written into a risk register before it’s written into an architecture diagram.
Your data leaves the building. Every prompt goes to a third party. For anyone in a regulated sector, that conversation with your compliance team should happen before the prototype, not after.
Prompts are brittle. Rewording a request slightly can change the output substantially. There’s no version control, no test suite, and no theory for why one phrasing beats another. Right now this is closer to a craft than an engineering discipline.
The cost model is unfamiliar. You pay per token, forever, on every request. Traditional machine learning has a large one-off training cost and cheap inference. This inverts that, and a service that’s cheap in a pilot can be expensive at volume. Model your unit economics before you scale, not after.
“A pilot that costs pennies and a production system that costs a fortune are the same system with more users. Do the arithmetic while the numbers are still small.” — Sameer Gupta
Final Thoughts
The honest summary is that GPT-3 makes an enormous number of things easy to try and a much smaller number of things safe to ship. Both halves of that sentence matter.
I expect the next two years to produce a great many impressive demonstrations and a considerably smaller number of systems that survive contact with real customers, real regulators, and real invoices. The gap between those two will be filled by unglamorous work: evaluation, guardrails, fallbacks, and knowing which problems genuinely need a model this large.
That’s usually where the actual value ends up. Start with the boring parts.
If you’re thinking through where this fits in your organisation, find me on LinkedIn.