A Good Win Probability Model: Brier Score & Log Loss (Part 3)

A win probability model that got the outcome “right” is not automatically a good model, and one that got a single game “wrong” is not automatically a bad one — because the model was never claiming certainty in the first place. Part 3 of this series sets aside the algorithm question covered in Part 2 and asks a more basic one: once a model exists, how is it actually judged?

Why “It Called the Winner” Is the Wrong Test

A model that says a team has an 80% win probability is not predicting that team wins — it is predicting that teams in comparable spots win about 80% of the time. If that team loses, the single outcome says almost nothing about whether the 80% figure was accurate, the same way a single coin flip landing tails says nothing about whether the coin is fair. Judging a probability model on individual outcomes throws away the information the model is actually claiming to convey.

The correct test operates on a pool of predictions, not one prediction at a time: gather every situation the model rated near 80%, across many games, and check whether close to 80% of them actually ended in a win. That property is called calibration, and it is the foundation both Brier score and log loss are built to measure.

Calibration: The Property That Actually Matters

A model is well calibrated when its stated probabilities match observed outcome frequencies across a large enough sample. Of every situation rated 60% win probability, roughly 60% should have gone on to be wins; of every situation rated 10%, roughly 10%. A calibration check groups predictions into probability buckets (0-10%, 10-20%, and so on) and compares the predicted probability in each bucket against the observed win rate within it — the visual version of that check is a reliability diagram, plotting predicted probability against observed frequency, where a perfectly calibrated model traces the diagonal.

Calibration is a different property from plain accuracy. A model can correctly identify the more likely winner in 80% of games — a respectable accuracy figure, the same kind of tally a confusion matrix summarizes for a binary classifier — while still being poorly calibrated, if its stated confidence levels are systematically too extreme or too conservative relative to what actually happens. Both properties matter, which is why calibration checks are used alongside win/loss accuracy rather than as a substitute for it (Hastie, Tibshirani & Friedman, The Elements of Statistical Learning).

A purely hypothetical illustration makes the gap concrete. Suppose a model correctly identifies the eventual winner in 80 out of 100 games — a strong-looking 80% accuracy figure by itself. If that same model also turns out to have called every one of those games at 95% or higher, but its actual win rate at that confidence level was closer to 82%, the model is simultaneously “accurate” in the everyday sense and overconfident in the calibration sense. Accuracy asks only whether the favorite was correctly identified; calibration asks whether the stated confidence level matched reality, and the two questions can point in different directions on the same set of games.

Brier Score: Squared Error for Probabilities

The Brier score, introduced by Glenn Brier in a 1950 paper on verifying probabilistic weather forecasts, is the mean squared difference between each predicted probability and the actual binary outcome — 1 if the team won, 0 if it lost (Brier, 1950). A model that says 90% for a team that wins contributes (1 − 0.90)² = 0.01 to the score; the same model saying 90% for a team that loses contributes (0 − 0.90)² = 0.81. Averaged across every play or every game, lower is better: 0 is a perfect score, 1 is the worst possible.

Brier score rewards two things at once — being well calibrated, and being sharp (confident) when the situation actually warrants confidence. A model that hedges everything toward 50% avoids large penalties for being badly wrong but also gives up the reward for being correctly confident, so it will not generally beat a sharper, well-calibrated model on Brier score.

Log Loss: Punishing Confident Mistakes Harder

Log loss (also called logarithmic loss or cross-entropy) takes the negative logarithm of the probability the model assigned to the outcome that actually happened. If a team wins and the model said 90%, log loss contributes −ln(0.90) ≈ 0.105; if the same team wins and the model had said 10%, log loss contributes −ln(0.10) ≈ 2.303 — more than twenty times worse for a prediction that was only nine percentage points more wrong in absolute terms than a middling guess would have been. Because the logarithm of a probability approaching 0 grows without bound, log loss punishes confident, wrong predictions far more severely than Brier score does, which makes it a stricter test of a model’s honesty about its own uncertainty.

nflfastR’s published model comparison used exactly this metric to evaluate whether adding the pregame spread as a feature was worth the added complexity: folding it in “substantially improved… cross-validation classification accuracy (reduced error rate from 27% to 23%) and log loss (reduced from .52 to .44)” relative to an otherwise identical model built without it (Open Source Football, cited above) — a concrete example of log loss doing exactly the job described here, deciding between two real candidate models rather than a hypothetical one.

Why Both Scores Need Enough Games to Mean Anything

Both Brier score and log loss are averages, and an average computed over too few games is unstable in the same way any small-sample statistic is. A model’s calibration in a single 60%-rated bucket containing only 8 plays could easily show an observed win rate of 50% or 75% purely from the ordinary variation of a small sample, without that gap implying anything wrong with the model. The nflfastR model comparison cited above avoids this trap by evaluating across full seasons of play-by-play data — tens of thousands of plays — rather than a handful of games, which is also why the leave-one-season-out validation approach discussed in Part 4 of this series repeats the evaluation across many seasons rather than settling for one.

A Worked Comparison of Two Models

A purely hypothetical comparison illustrates how the two scores can diverge. Suppose Model A is cautious and rarely strays from 50–60% in either direction, while Model B is sharper and regularly rates games at 85% or 15%. Across ten hypothetical games where the favored team wins eight times, Model A’s Brier score might average around 0.22 — middling, since it never commits to a confident answer — while Model B’s might average closer to 0.09, rewarded for being both confident and correct most of the time. But on the two games where Model B’s confident favorite loses, its log loss contribution spikes sharply (a 15% call on the actual winner contributes roughly 1.9 to log loss, versus roughly 0.7 for Model A’s more cautious 50% call on the same game) — numbers invented purely to illustrate the shape of the trade-off, not measured from any real model. The general pattern they illustrate is real: a sharper model wins more often on both scores, but it also has more to lose on the games it gets confidently wrong, and log loss makes that cost more visible than Brier score does.

Putting the Two Together

Brier score and log loss usually agree on which of two models is better, since both reward calibration and sharpness, but they weight extreme mistakes differently — log loss punishes them harder, Brier score more gently. In practice, both get reported side by side rather than one replacing the other: log loss as the stricter test of overconfidence, Brier score as the more forgiving summary that is easier to sanity-check by hand. Part 4 of this series returns to model evaluation directly, alongside a closer look at feature engineering once the basic scoring tools from this part are in place, and the practice of cross-validation that season-based evaluation depends on.