A forecast that says a team has a 70% chance to win and then watches that team lose was not necessarily wrong — grading a probabilistic forecast properly requires tools built for exactly that distinction, and two of them, the Brier score and the calibration plot, do most of the work.
Why “Was the Favorite Right?” Is the Wrong Question
The most common way fans evaluate a preseason or weekly forecast is the simplest one: did the team given better odds actually win? This instinct discards almost everything a probabilistic forecast is actually claiming. A forecast of 70% is not a claim that the favored team will definitely win — it is a claim that, across many similar situations, that outcome should happen roughly 70% of the time. A single game’s result, by itself, cannot confirm or refute a probability statement; only a large collection of forecasts, checked against what actually happened across all of them, can.
This is the same reason a weather forecaster who says “70% chance of rain” was not wrong if it does not rain that day. The honest way to evaluate that forecaster is to look back at every day they said “70% chance of rain” and check how often it actually rained on those days — which is exactly the logic behind grading a season of football forecasts, not a single game.
The Brier Score: A Single Number for Overall Accuracy
The standard tool for scoring a probabilistic forecast against a binary outcome is the Brier score, introduced by meteorologist Glenn W. Brier in a three-page 1950 paper in Monthly Weather Review — one of several scoring methods to come out of mid-20th-century weather forecast verification research, a field that had to solve exactly this problem decades before it became relevant to sports.
For a single forecast, the Brier score is the squared difference between the stated probability and the actual outcome (coded as 1 if the event happened, 0 if it did not):
A 70% forecast for a team that goes on to win scores . The same 70% forecast for a team that loses scores — a much larger penalty, since the forecast was confidently wrong, not moderately wrong. Averaged across many games, these per-game values produce an overall Brier score for a full set of forecasts, running from 0 (perfect) to 1 in its modern form (Brier’s original 1950 formulation used a 0-to-2 scale). A forecaster who always says exactly 50% for every game — expressing no opinion at all — scores a fixed 0.25 regardless of outcomes, which serves as a useful baseline: a real forecasting system needs to beat that number to demonstrate it is adding information, not just hedging.
Calibration: Checking Whether the Numbers Were Honest
A single Brier score answers “how close were the numbers, overall” but not “were the stated probabilities honest about their own confidence.” That second question is what a calibration plot — also called a reliability diagram — is built to answer. It works by grouping every forecast into bins by stated probability (all forecasts between 60% and 70%, for instance), then plotting each bin’s actual observed frequency of the event against the bin’s average forecast probability.
A perfectly calibrated forecaster’s points fall exactly on the diagonal line where forecast probability equals observed frequency — the 60-to-70% bin’s favored teams really do win roughly 60 to 70% of the time, no more, no less. Points sitting consistently above that diagonal indicate underconfidence (the stated probabilities are lower than what actually happened), and points sitting below it indicate overconfidence (favored teams win less often than the stated probability implied). This distinction matters because two forecasters can post identical overall Brier scores while having very different calibration shapes — one might be honestly calibrated across the board, while the other is overconfident on heavy favorites and underconfident on close games in a way that happens to average out.
Breaking the Brier Score Into Parts
Statistician Allan Murphy formalized, in 1973, a way to decompose the Brier score into three components: reliability (how far the calibration plot sits from the diagonal — the smaller, the better), resolution (how much the forecaster’s predictions actually vary across situations, rewarding a forecaster for confidently distinguishing likely from unlikely outcomes, not clustering everything near 50%), and uncertainty (a property of the outcomes themselves, not the forecaster, reflecting how inherently unpredictable the underlying events are). This decomposition is useful specifically because it separates two different ways a forecast can fail: a badly calibrated forecast has a reliability problem, while an overly cautious forecast that never strays far from 50% has a resolution problem, even if both could post a similar-looking aggregate Brier score.
A Hypothetical Worked Example
Suppose a hypothetical forecaster makes four preseason predictions, each stating a probability that a given team makes the playoffs, and each is graded once the season ends: Team A at 80% (made the playoffs), Team B at 55% (missed), Team C at 30% (missed), and Team D at 65% (made the playoffs). The per-forecast Brier scores are , , , and , averaging to a season Brier score of roughly 0.14 across the four forecasts. That average is what gets compared against other forecasters’ scores on the same four teams, or against the 0.25 baseline a forecaster expressing no opinion (a flat 50% on everything) would produce.
The same four forecasts can be checked for calibration too, though a meaningful calibration plot needs far more than four data points to say anything reliable — bucketing forecasts into probability ranges and checking observed frequency only becomes informative with dozens or hundreds of graded forecasts per bucket, which is one reason a full season of weekly game forecasts is a much better calibration dataset than a small handful of preseason predictions graded once a year.
Applying This to a Season of Football Forecasts
Grading a season of NFL or college football forecasts after the fact follows the same recipe: collect every forecast made (weekly game probabilities, or preseason win-total projections converted into probabilities), compute the Brier score across the full set, and build a calibration plot binning forecasts by their stated probability. A forecast set with a low Brier score and a calibration plot that hugs the diagonal has both accurately distinguished likely from unlikely outcomes and been honest about how confident it should have been — which is a considerably higher bar than simply counting how often the favored team won. Comparative studies of preseason projection systems exist for exactly this reason: forecast quality varies enough across methods, once graded properly instead of by anecdote, that checking the numbers after the season ends is worth doing, not assuming any single system’s stated confidence can be taken at face value.
The same grading logic applies to any probabilistic football forecast, not just preseason projections — a weekly win probability model can and should be graded the same way at the end of a season, and a forecast built from a small underlying sample carries the same uncertainty caution covered elsewhere: a good Brier score over only a handful of graded forecasts is far less conclusive than the same score over a full season’s worth.
Frequently asked questions
Why can't a forecast just be judged by whether the favored team won?
Because a probabilistic forecast is not a prediction of a single outcome — it is a claim about likelihood. A forecast giving a team a 60% chance to win is not wrong if that team loses; over many such forecasts, a well-calibrated model should see its 60% picks win close to 60% of the time, not 100%. Grading a single game by whether the favorite won discards almost all of the information the forecast actually contained.
What is the Brier score?
The Brier score is a way to measure the accuracy of a probabilistic forecast against what actually happened, introduced by meteorologist Glenn W. Brier in a 1950 paper in Monthly Weather Review. In its common binary-outcome form it is the mean squared difference between the forecast probability and the actual outcome (1 if the event happened, 0 if it did not), so a value of 0 is a perfect forecast and higher values indicate worse forecasts.
How is the Brier score calculated for one game?
For a single game, the score is (forecast probability − actual outcome)². A forecast of 70% for a team that wins scores (0.70 − 1)² = 0.09; the same 70% forecast for a team that loses scores (0.70 − 0)² = 0.49. Averaged across many games, these per-game scores produce the overall Brier score for a full set of forecasts.
What is a calibration plot (reliability diagram)?
A calibration plot — also called a reliability diagram — groups forecasts into bins by their stated probability (for example, all forecasts between 60% and 70%) and plots the actual observed frequency of the event within each bin against the bin's average forecast probability. A perfectly calibrated forecaster's points fall on the diagonal line where forecast probability equals observed frequency; points consistently above or below that line reveal systematic over- or under-confidence.
Can a forecast have a good Brier score but still be poorly calibrated?
Yes, though it is uncommon for the two to diverge sharply. The Brier score can be mathematically decomposed (a result formalized by Allan Murphy in 1973) into a reliability component, a resolution component, and an uncertainty component — a forecaster could have decent resolution (meaningfully distinguishing likely from unlikely outcomes) while still being systematically miscalibrated in one direction, which the aggregate score alone would not immediately reveal without breaking it down further.
Does a well-calibrated preseason forecast need to beat expert judgment to be useful?
Not necessarily — a properly calibrated forecast can be genuinely useful even without outperforming every individual expert, because calibration itself has value: knowing that a stated 70% actually means about 70%, not some unknown mix of overconfidence or hedging, is information a reader can act on. That is also why grading forecasts matters even for a system that turns out mediocre — the grading process is what reveals whether its stated confidence can be trusted at all, independent of how sharp any single prediction looked in hindsight.