AIHow Frontier Models Got Cheaper to Train
A year and a half ago I wrote Are We Over-Estimating AI’s Future? The argument was simple: GPT-3 changed the world for under $200M, GPT-4 cost more than $1B for a smaller leap, and GPT-5 might cost $10B without being ten times smarter. The cost of improvement was outrunning the improvement.
Then GPT-5 shipped, and by Epoch AI’s estimate it was trained on less compute than GPT-4.5 — the first GPT generation ever to use less than its predecessor. Eight months later DeepSeek released V4, a 1.6-trillion-parameter model that its own report says trails GPT-5.4 and Gemini 3.1 Pro by three to six months, and priced it at roughly a tenth of what OpenAI and Anthropic were charging per output token.
So this is the update. I need to separate two things the earlier essay ran together.
The cost of the biggest run kept climbing. Epoch AI’s trend data has frontier training compute growing about 5× a year and the cost of the run about 3.5× a year since 2020, and it estimates Grok 4 at roughly $490M and 246 million H100-hours. Labs are building gigawatt campuses. Nobody at the very top is spending less.
The cost of any fixed level of capability collapsed. Stanford’s AI Index found the cost to serve a GPT-3.5-class answer fell more than 280-fold between late 2022 and late 2024, and a16z’s “LLMflation” analysis pegs the decline at about 10× a year for equivalent performance. The cost to train a model to last year’s bar fell nearly as fast, for reasons I’ll walk through below.
“I said the price of progress was climbing. I should have said it was moving. Up at the very frontier, yes — but down, hard, in cost per unit of capability. Both curves are real. Only one of them shows up in the headlines.” — Sameer Gupta
The part I got right: brute-force dense scaling has punishing returns. The part I missed: efficiency is itself a frontier, and between 2024 and 2026 it moved faster than raw capability did. Here is what changed, layer by layer.
Pre-training: quality is the new scaling law
Chinchilla told everyone to stop building oversized models and feed smaller ones more tokens. The 2024–26 refinement is that which tokens matter more than how many.
- Aggressive filtering. FineWeb-Edu and DataComp-LM showed that a model-scored, heavily pruned slice of Common Crawl beats the full firehose at the same token budget. You reach the same loss with less compute because you stop training on garbage.
- Synthetic and rephrased data. “Textbooks Are All You Need” and the Phi models made the case that curated and generated “textbook-quality” text lets a 3–4B model punch far above its weight. Rephrasing web pages into cleaner prose is now a standard pre-training step, not a trick.
- Denser training signal. Meta’s multi-token prediction — training the model to predict several future tokens at once — improves sample efficiency. Alibaba’s Qwen3-Next and DeepSeek’s V3 line both ship it, and as a bonus it gives you a built-in draft model for speculative decoding at serving time.
None of this makes the headline number smaller on its own. Together they mean a 2026 model reaches a 2024 capability bar with materially less compute.
Pre-training: fewer bits, better optimizers, no wasted sweeps
Three quieter wins, each worth roughly a factor of two on the parts it touches:
- FP8 is standard; FP4 is arriving. FP8-LM and DeepSeek-V3’s technical report showed that 8-bit mixed precision is stable for most of the network, roughly halving the bytes moved versus BF16. In September 2025 NVIDIA showed the next step: pre-training a 12B model on 10 trillion tokens in NVFP4, with a loss gap under 1.5% relative to FP8, using random Hadamard transforms, two-dimensional block scaling, and stochastic rounding to keep the run stable. AMD followed with MXFP4 on MI355X in 2026. DeepSeek V4 reportedly trains its expert weights with a mix of FP8 and quantization-aware FP4. Bandwidth, not arithmetic, is the bottleneck in these runs — so each halving of precision is close to a free 2×.
- A better optimizer. AdamW ran the field for a decade. Moonshot’s “Muon is Scalable for LLM Training” showed Muon matching AdamW at about 52% of the training FLOPs, then used the stabilized MuonClip variant to pre-train Kimi K2 — a 1-trillion-parameter MoE — on 15.5 trillion tokens with, in their words, zero loss spikes. DeepSeek V4 adopted Muon too. A 2× from the optimizer alone was not on anyone’s 2024 roadmap.
- Hyperparameter transfer. muP lets you tune learning rate and initialization on a small proxy and carry the settings to the large run. The large run is expensive precisely because you only get to do it once; muP means you don’t burn a fortune discovering you set the learning rate wrong.
MoE: capacity without the FLOPs bill
This is the architectural centrepiece, and it deserves its own section.
A dense transformer activates every parameter for every token. A Mixture-of-Experts model splits each feed-forward layer into many “experts” and routes each token to only a few of them. GShard and the Switch Transformer proved the idea; Mixtral made it mainstream open-weight; DeepSeek-V3 pushed it to 671B total parameters with about 37B active. The 2025–26 generation pushed the ratio further still:
| Model | Total params | Active per token | Active share |
|---|---|---|---|
| DeepSeek-V3 (Dec 2024) | 671B | 37B | 5.5% |
| Kimi K2 (Jul 2025) | 1T | 32B | 3.2% |
| gpt-oss-120b (Aug 2025) | 117B | 5.1B | 4.4% |
| Qwen3-Next (Sep 2025) | 80B | 3B | 3.8% |
| DeepSeek-V4-Pro (Apr 2026) | 1.6T | 49B | 3.1% |
| DeepSeek-V4-Flash (Apr 2026) | 284B | 13B | 4.6% |
Two footnotes to the table. gpt-oss-120b is OpenAI’s own open-weight release — 4 of 128 experts active, 2.1 million H100-hours to train, and the whole thing fits on one 80GB GPU. And Moonshot’s Kimi K3 (July 2026) pushes sparsity further still: 2.8 trillion parameters with 16 of 896 experts active per token, under 2% of the expert pool. On the closed side, Google’s Gemini 2.5 report says plainly that its models “are sparse mixture-of-experts (MoE) transformers.” The other closed labs don’t say, and nobody believes they are dense.
The economic point is the one that matters here:
“MoE decouples how much a model knows from how much each token costs to run. You pay training and inference compute for the active slice; you keep the knowledge of the whole thing.” — Sameer Gupta
DeepSeekMoE added the refinements that made it work at these ratios: many fine-grained experts instead of a few big ones, a shared expert that always fires for common patterns, and load balancing that doesn’t lean on a noisy auxiliary loss. Qwen3-Next’s configuration — 512 experts, 10 routed plus one shared — is what that recipe looks like when everyone has adopted it.
MoE is not free. It trades compute for memory — you still have to hold all 1.6T parameters somewhere — and the all-to-all routing traffic is an engineering problem that eats real months. It is harder to train stably and harder to serve. But for anyone whose goal is intelligence per dollar rather than parameters per press release, it is the most important structural change of the period.
Attention: the quadratic finally cracked
Attention cost grows with the square of context length, which is why long context used to be a luxury. Two 2025 developments made it cheap.
- Sparse attention that learns what to look at. DeepSeek-V3.2 introduced DeepSeek Sparse Attention: a lightweight “lightning indexer” scores every prior token and the model attends only to the top 2,048 of them, turning quadratic cost into linear. V4 extended this into Compressed Sparse Attention and Heavily Compressed Attention, and the result is a one-million-token context window that its report says needs 27% of the per-token inference FLOPs and 10% of the KV cache of V3.2.
- Hybrid linear attention. Qwen3-Next alternates three Gated DeltaNet layers — a linear-time recurrent form — with one conventional attention layer. Alibaba’s claim is the cleanest single sentence on the efficiency wave I have read: it beats the dense Qwen3-32B on downstream tasks at 10% of the training cost, with ten times the throughput past 32K tokens.
Alongside these, DeepSeek’s Multi-head Latent Attention compresses the KV cache into a low-rank latent, and their manifold-constrained hyper-connections (mHC, December 2025) widen the residual stream for better convergence at negligible overhead. Individually these are engineering. Together they are why a 1.6T-parameter model can be served at $3.48 per million output tokens.
Post-training: RLHF got a lot cheaper
The classic alignment recipe — collect human preference labels, train a reward model, run PPO against it — is finicky and expensive. Most of it has been replaced:
- Direct Preference Optimization and its relatives (KTO, ORPO, SimPO) skip the reward model and the RL loop entirely, optimizing preferences with a plain classification-style loss. Cheaper, more stable, more reproducible.
- Constitutional AI / RLAIF replaces most human labels with model-generated feedback against a written set of principles. The expensive input — human judgement — moves from “every comparison” to “the rules.”
Post-training: RL on things you can check, now at scale
The bigger shift is reinforcement learning with verifiable rewards. Where the answer can be graded automatically — math, code, formal proofs, whether an agent actually completed a task in a sandbox — you can run RL on correctness directly, with no reward model to game.
DeepSeek-R1 was the clean demonstration in January 2025: its “R1-Zero” variant was trained with pure RL on verifiable problems and no supervised fine-tuning, and it spontaneously developed long chains of thought, backtracking, and self-checking. When the work passed peer review in Nature that September, the supplementary material put a price on the RL stage: 512 H800s for a few hundred hours, about $294K on top of the $5.6M base model. MiniMax-M1 published a similar bill in June — its full RL run took 512 H800s for three weeks, $534,700. The reasoning breakthrough of the decade cost less than one enterprise’s annual licence renewal. What happened next is the part that matters for a budget conversation.
RL became the place labs put their marginal dollar. Epoch’s analysis of GPT-5 is blunt: until recently models spent roughly 100× more compute on pre-training than post-training; the new reasoning methods let you cut pre-training by about 10× and make it up with post-training, so OpenAI scaled post-training as hard as it could on a smaller base model. OpenAI has said o3 used more than 10× the RL compute of o1, and xAI described a similar jump from Grok 3 to Grok 4. The Information reported in September 2025 that Anthropic had discussed spending over $1B on RL environments — the sandboxes, tasks, and graders that RL trains against — over the following year.
RL got predictable. Pre-training has had scaling laws since 2020; RL had folklore. Meta’s “The Art of Scaling Reinforcement Learning Compute for LLMs” (October 2025) spent over 400,000 GPU-hours to show that RL performance follows sigmoidal compute curves you can fit on small runs and extrapolate, and that most recipe choices change efficiency rather than the ceiling. Kimi K2 and DeepSeek V3.2 both describe large joint RL stages across reasoning, agentic, and alignment tasks in one run. Once you can forecast it, you can budget it.
The relevant fact: this bought a large capability jump with post-training and inference compute, not with a ten-times-bigger pre-training run. Epoch also expects the trick to run out — post-training compute has been growing something like 10× every four months, which cannot continue — and predicts GPT-6 will resume the pre-training climb. That is the frontier curve reasserting itself. It does not undo the efficiency gains below it.
Distillation: frontier reasoning at small-model prices
DeepSeek distilled R1’s reasoning traces into dense 7B–70B models built on Qwen and Llama, and they beat much larger instruction-tuned models on reasoning benchmarks. Eighteen months on, this is simply how the ecosystem works: every frontier release is followed within a quarter by open-weight models that reproduce most of its capability at a fraction of the size. That dynamic is exactly the one behind The Case for SLMs and Classic ML. If you can wait a quarter, someone will hand you this year’s reasoning in something you can run on one GPU.
Test-time compute as a scaling axis
Finally, work on scaling test-time compute optimally showed that letting a smaller model think longer — more samples, search, longer chains — can match a much larger model on hard problems for a fraction of the training cost. Every major model now ships with an adjustable thinking budget; DeepSeek’s V3.2-Speciale variant is the same idea taken to the extreme. “Bigger model” is now one lever among several, and often not the cheapest one.
Hardware: the multiplier under everything
I have left hardware for last because it is the smallest part of the story. Stanford’s AI Index estimates hardware price-performance improving about 30% a year — real, but nowhere near the 3× a year Epoch attributes to algorithms, and a rounding error next to the 10× a year in equivalent-capability inference prices. Independent analyses of the price collapse put nearly all of it on software rather than chips.
The next hardware step is nonetheless designed for this new world. NVIDIA’s Vera Rubin, ramping in the second half of 2026, is built around NVFP4 and NVIDIA claims it needs a quarter as many GPUs as Blackwell to train a large MoE. And DeepSeek V4 shipped with day-one support for Huawei’s Ascend chips, with Huawei saying it ran V4-Pro’s post-training on its own silicon. Four-bit math, sparse experts, and a second supplier — all three push the same direction.
So — are we still over-estimating?
Both things are true at once, and the earlier essay only said the first:
Diminishing returns on dense brute force are real. Nobody is getting a 2029-level jump by renting ten times the cluster and changing nothing else. The biggest runs cost half a billion dollars and will cross a billion; that money buys the frontier, not a multiple of it.
“AI progress is stalling” was wrong. The efficient frontier kept moving, and it moved down in cost per unit of capability about as fast as it moved up in raw capability. Sparse experts, sparse attention, ruthless data curation, 8-bit and now 4-bit math, a 2× optimizer, preference tuning without the RL circus, RL on checkable answers that now scales predictably, and distillation each took a bite out of the bill. A team with a few thousand GPUs can train a model in 2026 that would have been the best in the world in 2024.
One honest caveat. A 2025 MIT study, “On the Origin of Algorithmic Progress in AI,” found that most measured algorithmic gains are scale-dependent — they pay off more the bigger you go — and that progress for small models has been slower than the headline numbers suggest. That is why the two curves diverge instead of converging: efficiency tricks compound with compute, so the frontier gets both, and everyone else gets the tricks a year later.
For an enterprise the conclusion is the same as it was in The Case Against Manhattan-Scale AI Farms, just with better evidence:
“The teams that win the next few years won’t be the ones who booked the biggest cluster. They’ll be the ones who rode the efficiency curve — distilled models, MoE-served capacity, cheap preference tuning, open weights a quarter behind the frontier — and spent the savings on getting it into production.” — Sameer Gupta
The cost of frontier AI didn’t run away from us. It split in two. The organizations that notice will pay a fraction of what the 2024 projections said they would.