AINate Silver Didn't Get Lucky
Two weeks ago a statistician called every state correctly in a presidential election that most television commentators had spent months describing as too close to call.
The reaction has been to treat this as a story about polling. I think it’s a story about how organisations handle uncertainty, and on that reading it applies directly to how your company makes decisions.
The Actual Disagreement
It’s worth being precise about what the argument was, because it wasn’t about who would win.
The pundits weren’t making a different forecast. They mostly weren’t making a forecast at all. They were describing a state of affairs, that the race was close, and treating “close” as equivalent to “unknowable.”
Silver’s model said something structurally different: given the polls, the outcome distribution puts roughly a 90% chance on one result. Close and predictable are not opposites. A race can be close in vote share and lopsided in probability, because a small consistent lead across many independent-ish states compounds.
That distinction is the whole thing, and almost nobody on television could hold it.
“They kept saying the race was too close to call. It was close. It was not too close to call. Those are separate measurements and conflating them is the mistake.” — Sameer Gupta
The Business Version
Now think about how forecasts arrive on your desk.
Someone says the project will be done in March. Someone says the deal will close this quarter. Someone says the new product will do eight million in its first year.
Single numbers. No distributions. No confidence. And critically, no way to be wrong in a useful sense, because if the deal closes in the following quarter, the forecaster will say it was nearly right, and there’s no record against which to check that claim.
Here’s what a probabilistic version looks like:
- “Sixty per cent chance we ship by March, eighty-five per cent by May.”
- “This deal is at forty per cent. Similar deals at this stage closed about that often last year.”
- “Central case eight million, but there’s a fat left tail if the supplier issue isn’t resolved.”
The second set is more useful, more honest, and considerably more uncomfortable to say out loud. That last part is why it’s rare.
Why Organisations Resist This
I want to be fair about the resistance, because it isn’t stupidity.
Confidence is rewarded and hedging is punished. The person who says “we’ll hit the number” gets the budget. The person who says “seventy per cent chance we hit the number” sounds like they’re pre-building an excuse. That incentive is set by senior people and it is entirely within their power to change.
A probability can’t be graded on one instance. If you say seventy per cent and it doesn’t happen, were you wrong? Not necessarily. Thirty per cent things happen three times in ten. You can only evaluate a forecaster over many predictions, which requires writing them down and keeping score, which almost nobody does.
It exposes how little we know. A point estimate hides the width of the uncertainty. Ask for a range and you’ll sometimes get a range so wide it embarrasses everyone. That’s not the forecast getting worse. That’s the forecast getting honest.
What I’d Actually Change
Three things, none of them requiring a statistician.
Attach a number to the confidence, every time. Any forecast that goes to a decision-maker carries a percentage. It will feel artificial for about a month and then it will feel obviously correct.
Keep a scorecard. Write down the predictions and the probabilities, and revisit them quarterly. You are looking for calibration: of the things you called seventy per cent, did about seventy per cent happen? If ninety per cent of them happened, you’re systematically under-confident and leaving opportunities on the table. If forty per cent did, you have a problem worth finding.
Reward the calibrated, not the bold. This is the hard one because it’s cultural rather than procedural. The person whose seventy per cents come in at seventy per cent is more valuable than the person who is loudly certain and right slightly more than half the time. Most organisations promote the second one.
“A forecaster who is right eighty per cent of the time and says so is worth more than one who is right eighty per cent of the time and claims a hundred. The first one you can plan around.” — Sameer Gupta
The Modelling Lesson Underneath
There’s a technical point in the Silver model worth borrowing too.
He wasn’t running one poll. He was aggregating many, weighting them by historical accuracy, adjusting for known house effects, and simulating the outcome thousands of times to see how often each result came up.
That’s the same ensemble idea that showed up in Watson last year, and it keeps recurring: many imperfect signals, combined with an honest account of each one’s reliability, beat one signal you’ve decided to trust.
Most business forecasting does the opposite. It picks the one source that seems most authoritative and treats it as truth.
Final Thoughts
None of this requires you to hire a statistician or build a model. It requires you to stop asking “when will it be done” and start asking “what’s the chance it’s done by then,” and then to write the answer down where it can be checked later.
That’s a change in language and record-keeping. It’s nearly free, and it’s the closest thing to a reliable improvement in decision quality I’ve come across.
The pundits weren’t beaten by better mathematics. They were beaten by someone who was willing to state a number and be held to it.