AIStop Tuning the Model. Fix the Data.
Andrew Ng has been running a campaign this year with a simple thesis: the field has spent a decade improving models on fixed datasets, and in most real applications the dataset is where the remaining gains are.
He calls it data-centric AI. It’s not a new observation, and anyone who has run an actual project has felt it. But it’s worth stating plainly, because the incentives in this field push relentlessly the other way.
Why the Field Optimises the Wrong Half
Academic progress works by holding the data constant and varying the method. That’s a sound experimental design. If everyone benchmarks on the same dataset, improvements are comparable and attributable.
The consequence is that a decade of published work is about better models on fixed data, and every practitioner is trained in that habit.
Then they arrive in a company where the data is not fixed, is not clean, was assembled by accident over eleven years, and where nobody has ever checked whether two labellers agree.
“The benchmark holds the data still so you can measure the model. Then everyone forgets that in the real world, the data is the part you’re allowed to change.” — Sameer Gupta
The Example That Convinced Me
The demonstration Ng keeps using is a steel defect inspection problem, and the shape of it matches what I’ve seen repeatedly.
A team improves the model architecture extensively and gains almost nothing. Then somebody goes and looks at the labels, and discovers that inspectors disagree about what counts as a defect. Some mark the whole affected region. Some mark the centre. Some don’t mark borderline cases at all.
The model isn’t underperforming because it’s not clever enough. It’s being asked to learn a rule that the training data does not consistently express. No architecture fixes that, because the inconsistency is the ceiling.
Getting the labellers in a room to agree on a definition, then relabelling, produced a bigger gain than months of modelling work.
What This Looks Like in Practice
The data-centric approach is a set of unglamorous disciplines:
- Measure inter-annotator agreement. Have two people label the same five hundred examples independently and compare. If they agree eighty per cent of the time, your model cannot exceed eighty per cent, and you now know what to fix first. Almost nobody does this and it takes a day.
- Write the labelling guide, then argue about it. Every disagreement you resolve on paper is noise you remove from the training set. The edge cases are where the definition actually lives.
- Look at the errors individually. Not the aggregate metric. Print fifty misclassified examples and read them. In my experience a third of them turn out to be labelled wrong, which means your accuracy figure is understated and your model is better than you think.
- Fix systematically, not randomly. Errors cluster. If the model fails on a particular sub-population, the answer is more and better data for that sub-population, not more data overall.
- Improve consistency before quantity. A thousand consistently labelled examples beat ten thousand noisy ones. This is counterintuitive to anyone who has absorbed the “more data is always better” message.
Why This Suits Enterprises Specifically
Here’s the strategic argument, and it’s the reason I think this matters more for ordinary companies than for research labs.
You cannot win on models. The architectures come from a handful of well-funded labs, they’re published, and everyone gets them at the same time. I made this argument years ago about the talent market and it’s only become more true.
You can absolutely win on data, because your data is genuinely yours. Nobody else has your maintenance records, your claim histories, your inspection photographs, or your customers’ behaviour in your specific market.
Which means the effort you spend making that data cleaner, more consistent, and better labelled compounds into an asset nobody can copy. The effort you spend on architecture search produces something your competitor will have next quarter for free.
“Model improvements are rented from the research community. Data improvements are owned. Spend accordingly.” — Sameer Gupta
The Organisational Obstacle
I should be honest that this is hard to sell internally, for reasons that are entirely about status.
Improving a model architecture is technical work that looks like engineering. Relabelling a dataset looks like admin. The first gets presented at a conference. The second gets outsourced to the cheapest available bidder with no guidance, which is precisely how the labels got inconsistent in the first place.
If you want data-centric work to happen, somebody senior has to say out loud that it’s the valuable part, and it has to be done by people who understand the domain rather than by whoever is cheapest per label.
Final Thoughts
The pattern I keep returning to in this blog is that the unglamorous half of a problem is usually where the money is. Evaluation over modelling. Deployment over research. Data over architecture.
None of it makes an announcement. All of it determines whether a project works.
If you have a model stuck at some disappointing level of accuracy, and the team’s plan is to try another architecture, I’d ask a different question first. Get two people to label five hundred examples independently and see how often they agree.
The answer to that question tells you whether you have a modelling problem at all. Usually you don’t.