AIWhen the Model Is Biased, the Business Is Liable
In May, ProPublica published an investigation into a risk-assessment tool used in American courts to predict whether a defendant will reoffend. The scores inform bail and sentencing decisions.
Their finding, put simply: the tool was wrong in different directions for different racial groups. Black defendants who did not go on to reoffend were far more likely to have been labelled high risk. White defendants who did reoffend were far more likely to have been labelled low risk.
The subsequent argument between ProPublica and the vendor is technically fascinating and I’ll come to it. But I want to start with the part that should concern anyone deploying a model in a business.
The Bit That Should Worry You
Nobody put race into the model.
That’s the whole point and it’s the part that gets lost. The vendor didn’t use race as an input. They didn’t need to. In a society where postcode, employment history, prior contact with police, and family circumstances all correlate with race, a model built on those features will reproduce the pattern faithfully without ever being told about it.
“You cannot make a model blind to something by not showing it to the model. The correlations do the work you were trying to avoid.” — Sameer Gupta
Now translate that to a commercial setting. You build a credit model, a pricing model, an insurance model, a hiring screen. You carefully exclude protected characteristics, as your legal team advised. You feel you’ve addressed the problem.
You have not addressed the problem. You have only made it harder to detect.
The Argument Is Genuinely Hard
The vendor’s defence was not stupid, and understanding it matters.
They argued their tool was fair by a defensible definition: among defendants given a particular score, the actual reoffence rate was roughly the same across groups. A score of seven meant the same thing regardless of who received it. That’s calibration, and it’s a completely reasonable thing to mean by fairness.
ProPublica used a different definition: among people who did not reoffend, what fraction were wrongly labelled high risk? By that measure, error rate balance, the tool was clearly unequal.
Here is the uncomfortable mathematical fact that has been coming into focus this year: when the underlying base rates differ between groups, you cannot satisfy both definitions at once. It is not a matter of building a better model. The definitions are mutually exclusive except in special cases.
So there is no purely technical answer. Someone has to decide which kind of fairness matters for this application, and that is a values decision that gets made either explicitly by an accountable person, or implicitly by whichever engineer picked a loss function.
“There is no setting on the model that makes this go away. Somebody chooses which unfairness to accept, and right now that somebody is usually a developer who doesn’t know they’re choosing.” — Sameer Gupta
Why This Is a Business Risk, Not an Ethics Seminar
I want to make the commercial case, because in my experience that’s what gets a project funded.
Disparate impact is a legal standard in several jurisdictions, and intent is not a defence. If your process produces materially different outcomes across a protected group, “the algorithm did it and we didn’t include the variable” is not an answer that helps you. In some regimes it’s an aggravating factor, because it suggests you didn’t look.
You will be asked to explain it. By a regulator, a journalist, or a claimant’s lawyer. “It’s a proprietary model” and “the vendor built it” are both answers that make the situation worse.
The reputational asymmetry is brutal. A model that’s 3% more accurate and produces one well-documented discriminatory outcome is a net loss, and the accuracy gain will not be mentioned in any of the coverage.
What I’d Actually Require
Before any model that affects a person’s access to money, employment, housing, or a service:
- Measure outcomes by group, and write the numbers down. You cannot manage what you refuse to look at, and choosing not to look is itself a decision a court can examine.
- Decide explicitly which fairness definition applies, document the reasoning, and have a named person sign it. This is uncomfortable and it is the entire job.
- Audit the proxies. Which inputs correlate with protected characteristics? You will find some. Postcode is almost always one.
- Keep a human in the loop where the stakes are high, with genuine authority to override and a record of when they do. If overrides never happen, you don’t have a human in the loop, you have a rubber stamp.
- Own the vendor’s model. If you deploy it, you are answerable for it. Demand the fairness testing before you sign, not after the story runs.
- Retest periodically. Populations shift. A model that was balanced at launch may not be in two years, and nothing will alert you.
Final Thoughts
The instinct in most companies is to treat this as a compliance question, hand it to legal, and get a memo.
That won’t work, because the decisions that matter are made during model development, by people choosing objectives and features and thresholds, months before anything reaches a lawyer. By the time it’s a compliance question the choices are baked in.
GDPR is adopted and comes into force in two years, and it contains language about automated decision-making that I suspect will be tested against exactly this kind of system. The companies that will handle that well are the ones that started measuring outcomes by group now, voluntarily, while it was cheap.
The others will start measuring when someone makes them, which is a considerably more expensive way to find out what your model has been doing.