A football game only happens once. That single fact makes a probability estimate about it genuinely strange to verify — there is no way to rerun the fourth quarter a hundred times and count how often the trailing team came back. The gap between probability and outcome is where most misreadings of win probability start, and it is worth working through carefully.
What “80% Win Probability” Actually Claims
A model that says a team has an 80% win probability is making a statement about a class of situations, not a guarantee about this specific one. The precise claim is closer to: “among game states that share these characteristics — this score, this time remaining, this field position, and so on — teams in the leading position have historically won about 80% of the time.” It is a statement about frequency across many comparable situations, applied to one instance of that situation, not a prediction that this particular game is certain to go one way.
That distinction matters because it means a single outcome — win or lose — cannot, by itself, confirm or refute the estimate. If the leading team wins, that is consistent with an 80% estimate. If the leading team loses, that is also consistent with an 80% estimate, since one loss in five is exactly what “80%” implies over time. Neither single result moves the needle much on whether 80% was the right number to have said in the first place.
The Test That Actually Works
The piece on what makes a win probability model good works through the calibration test in detail: gather every situation a model rated near a given probability, pooled across many games, and check whether the observed win rate in that group matches the stated probability. That test requires volume — hundreds of comparable situations, not one — because a single event, however dramatic, is a sample size of one drawn from a distribution the model is making a claim about, not a verdict on the distribution itself.
This is also why “the model called it wrong” is a category error when applied to a single game. A probability estimate isn’t the kind of claim that a single outcome can falsify, the way a specific numeric prediction (say, a projected final score) can be checked directly against what actually happened. It is closer to a weather forecaster saying there’s a 30% chance of rain — if it rains, the forecaster wasn’t necessarily wrong, and if it doesn’t rain, the forecaster wasn’t necessarily right; the forecast is only checkable against many days of 30%-chance forecasts and how often it actually rained on those days (Brier, 1950).
Applying a Long-Run Frequency to a One-Off Event
There is a real philosophical wrinkle underneath all of this: a win probability model’s “80%” is built from a long-run frequency across many historical games, but it gets applied to a single game that will only ever happen once. That is not unique to football — the same wrinkle exists in a weather forecaster’s “30% chance of rain today” or a medical test’s “85% accurate” — but football makes it unusually visible, because the single event in question is watched live, in public, by people who then react to its one outcome as though it settled something about the number that preceded it.
The resolution isn’t complicated, even if it feels unsatisfying: a probability estimate about a one-off event is still meaningful, because it is a statement about the class of situations the event belongs to, not a hidden fact about that specific event waiting to be revealed. Two people can watch the exact same 80%-favored team lose and reasonably disagree about almost nothing regarding the model, because both understand that losing from 80% is an expected, if less common, outcome — not a refutation.
Why Single Outcomes Feel More Convincing Than They Are
A trailing team’s improbable comeback is memorable precisely because it was improbable — it stands out, gets replayed, and becomes the story people tell about that game. A model that correctly estimated a small chance of that comeback, and then watched it happen, produced exactly the kind of rare event its own probability said should occasionally occur. Treating that single vivid outcome as proof the model was “wrong” mistakes the model’s honest acknowledgment of uncertainty for a mistake, when the uncertainty itself was the accurate part of the estimate.
A useful reframe: a win probability estimate immediately before a surprising result is itself the best available description of how surprising that result actually was. A come-from-10%-win-probability victory is a genuinely uncommon event — that much is true, and the model captured it correctly by rating it uncommon in the first place. What doesn’t follow is that “uncommon happened” equals “the estimate of ten percent was wrong.” A model that never allowed a 10%-probability team to win at all — one that only ever assigned 0% or 100% — would not be a better model; it would be an overconfident one, and Part 3 of the building series covers exactly why that kind of false confidence gets penalized more heavily by log loss than an honest, hedged estimate does. The same question — treating a decision-making process as separate from a single prediction’s accuracy — is worth reading alongside thinking about statistical decision-making versus prediction.
What Would Actually Show a Model Is Wrong
A win probability model is shown to be wrong the same way any calibrated forecast is: not by a single outcome, but by a pattern across many. If situations rated 80% actually produced wins closer to 65% of the time, pooled across hundreds of comparable plays, that is evidence of a real problem — a missing feature, a miscalibrated pregame prior, or a model overfit to seasons that no longer represent how the game is played (Hastie, Tibshirani & Friedman, cited above). That kind of pattern is detectable and correctable — and worth distinguishing from a genuinely counterintuitive finding, which can be a real signal rather than a modeling defect. A single dramatic finish, on its own, is neither.
A purely hypothetical illustration draws the line clearly. One team completing a comeback from a 12% win probability is a single, vivid data point — striking to watch, but statistically unremarkable on its own, since a well-calibrated model should see roughly one in eight of its 12%-rated teams go on to win. If, on the other hand, a model’s 12%-rated teams won closer to 30% of the time across several hundred pooled situations, that would be the kind of gap worth investigating — not because any one of those wins was surprising, but because the aggregate rate no longer matches what “12%” is supposed to mean. The difference between the two cases is exactly the difference between an outcome and a pattern, and only the second one says anything about whether the model itself needs fixing.
Frequently asked questions
If a team with an 80% win probability loses, was the model wrong?
Not necessarily. A model claiming 80% is stating that teams in comparable situations win about four times out of five — it is not claiming certainty. A single loss from that position is exactly what should happen roughly one time in five if the model is accurate, so one outcome alone cannot confirm or refute the estimate.
How many games would it take to actually test a win probability model?
Enough plays or games rated at a similar probability level to compare against the observed win rate in that bucket — usually hundreds of comparable situations pooled across many games and seasons, not a handful of individual outcomes. This is the same calibration test described in the piece on what makes a win probability model good.
Does a probability estimate become 'true' or 'false' once the game ends?
No — the estimate was never a claim about that single game's certain outcome, so the final score doesn't retroactively prove or disprove it. What the final score does is contribute one more data point to the much larger sample needed to check whether the model's stated probabilities match reality across many similar situations.
Why do people tend to judge a probability model by single outcomes anyway?
Because a single game is vivid and immediate, while calibration statistics require pooling many games and are abstract by comparison. It's a well-documented pattern in how people reason about probabilistic forecasts generally, not something specific to football — a single dramatic exception is more memorable than hundreds of unremarkable confirmations.
Is there a way to describe how 'surprising' a single result actually was?
Yes — the win probability estimate immediately before the outcome is itself that description. A team that wins from a 10% win probability produced a genuinely uncommon result, but 'uncommon' is not the same as 'the model was wrong'; a well-calibrated model is expected to be on the losing side of its own 10% estimate about one time in ten.