Part 1 of this series listed six features for a football win probability model; Part 3 established how to score one once it exists. What sits between the two is feature engineering — turning raw game-state columns into the combined terms a model actually learns from — and model evaluation done in a way that respects how football data is structured. Part 4 closes the series with both.
From Raw Columns to Engineered Features
A play-by-play row arrives with score, clock, yard line, down, distance and timeouts as separate columns, but a model that treats them as independent inputs misses how they interact. The clearest example is score and time: nflfastR’s model combines them into a single term, diff_time_ratio = point_differential * e^(4*(3600-game_seconds_remaining)/3600), which amplifies the effect of a point-differential as game_seconds_remaining shrinks toward zero — the same lead is worth more, in the model’s eyes, with two minutes left than with two quarters left, expressed as one engineered number, not left for the model to infer from two separate columns (Open Source Football, cited below).
The pregame spread gets the same treatment in reverse. It does not get fed in as a constant that applies equally at every point in the game; nflfastR’s spread-aware model computes spread_time = posteam_spread * e^(-4*(3600-game_seconds_remaining)/3600) — full weight near kickoff, decaying smoothly as the clock runs down and the in-game state itself becomes the more informative signal. Both terms share the same exponential shape, just running in opposite directions: one grows in influence as time passes, the other fades.
Down and distance can be engineered similarly, and they do not have to be passed in as two separate numbers: a model can be given a combined “conversion difficulty” feature, or the two can be left separate and allowed to interact freely inside a tree-based model, which — as Part 2 of this series covered — can discover useful combinations of raw features on its own, without requiring every interaction to be specified by hand.
Encoding Categorical Game-State Variables
Not every feature in a football win probability model is a plain number. Whether the possession team is playing at home is a yes/no value, typically encoded as 0 or 1 before it reaches the model, and it needs to be included even though home-field advantage is already partly reflected in the pregame spread — the in-game boost from a home crowd on a specific close fourth-quarter series is a separate, situational effect from the season-long advantage a spread already prices in. Whether the team about to get the ball has already received the second-half kickoff is another binary flag: a team trailing that still has the second-half kickoff ahead of it has a meaningfully different set of remaining possessions than one that has already used it, even at an identical score and clock.
These binary and categorical features add relatively little complexity to a logistic regression, where they simply become additional coefficients, but they matter more inside a tree-based model, where a single well-placed binary split (home vs. away, kickoff received vs. not) can meaningfully separate plays that otherwise look similar on every numeric feature.
The Trap in How Data Gets Split
A model is only as trustworthy as the test it was evaluated against, and the most common mistake in evaluating a football win probability model is splitting play-by-play rows at random. Individual plays are not independent observations: every play in a given game shares the same two teams, the same eventual outcome, and a running score and clock that are highly correlated from one play to the next within that game. A random shuffle can put some plays from a single game into the training set and other plays from that same game into the test set, letting information about how the game turned out leak across the split — which produces an evaluation score that looks better than the model actually deserves.
The fix is to split on whole games, or better, on whole seasons, so that no game’s plays appear on both sides of the split. nflfastR’s own model documentation describes exactly this approach: leave-one-season-out (LOSO) validation, in which each season across the training window takes a turn as the holdout set, the model is refit on the remaining seasons, and its predictions are checked for calibration against the season it never saw (Open Source Football, cited below). Repeating the process across every available season — not holding out just one — gives a far more stable read on whether the model generalizes than a single train/test split would.
A hypothetical illustration shows why the random-split trap is so easy to fall into unnoticed. Suppose a single close game contributes 70 plays to a dataset, and a random 80/20 split happens to place 56 of those plays in training and the remaining 14 in the test set. A tree-based model can partially “recognize” that specific game — its particular sequence of scores and drives — from the 56 plays it trained on, and then get an unrealistically easy time predicting the other 14 plays from the same game, since they share the same final outcome and much of the same context. None of that advantage would exist against a genuinely new season the model has never touched, which is exactly the gap a random split hides and a season-based split exposes.
What a Season-Level Evaluation Should Check
The same calibration check introduced in Part 3 — grouping predictions by probability bucket and comparing the predicted rate against the observed win rate — is what a LOSO evaluation is actually looking for, applied to a held-out season instead of to the training data the model has already seen. A model that stays well calibrated across every held-out season is behaving consistently across different rosters, rule eras and offensive trends; one that is well calibrated in training but drifts on held-out seasons is a warning sign that it has learned something specific to the seasons it was trained on, not a generalizable relationship between game state and outcome — the subject the next piece in this series, on uncertainty and overfitting, takes up directly.
Brier score and log loss, from Part 3, both work the same way in a season-level evaluation: computed on the held-out season’s plays only, never on any play the model was trained on, and tracked bucket by bucket (by score margin, by quarter, by field position) to spot exactly where a model’s feature set is still incomplete, not to settle for one aggregate number.
Taken together, the four parts of this series describe a full cycle, not a straight line: features get engineered from raw game-state columns (Part 1, revisited here with the exact decay formulas behind them), an algorithm gets chosen to fit them (Part 2), the result gets scored honestly (Part 3), and the evaluation from this part often points straight back to Part 1 — a systematic gap in a specific score-margin bucket is frequently the sign of an interaction term that still needs to be engineered, closing the loop between evaluation and feature design instead of treating them as separate, one-time steps. The same season-based evaluation discipline applies just as directly to a play-level model like expected points, including how much uncertainty an expected-points estimate actually carries.
Frequently asked questions
What is feature engineering in a win probability model?
Feature engineering is the step of turning raw game-state columns (score, clock, yard line, down, distance) into the combined terms a model actually learns from — like multiplying score differential by a function of time remaining, so the model captures that the same point gap matters more as the clock runs down, instead of treating each column as independent.
What is an interaction term and why does a win probability model need one?
An interaction term combines two features into one, capturing how the effect of one depends on the value of the other. Score differential and time remaining are the clearest example: a 10-point lead means something different with a full game left than with two minutes left, and an interaction term lets a model represent that jointly, not as two separate, additive effects.
How does nflfastR's win probability model weight the pregame spread over time?
It applies a decaying exponential term, spread_time = posteam_spread * e^(-4*(3600-game_seconds_remaining)/3600), which gives the spread its full weight near the opening kickoff and lets that weight shrink smoothly as game_seconds_remaining falls toward zero and in-game evidence accumulates. The score-differential term is built on the same exponential shape, just amplifying, not decaying, as the clock runs down.
How should a win probability model be validated?
By splitting on whole games or whole seasons, never by shuffling individual plays into training and test sets at random, because plays from the same game are correlated with each other and a random shuffle would let information about a game's outcome leak into both sets. Leave-one-season-out (LOSO) validation — holding out an entire season, training on the rest, and checking calibration against the holdout — is the method nflfastR's own model documentation describes.
Why can't play-by-play rows be split randomly for training and testing?
Because plays within the same game are not independent observations — they share a score, a set of teams, and an eventual outcome. A random split would put some plays from a single game in training and others from the same game in the test set, letting the model effectively see the answer during training and produce an inflated, unrealistic evaluation score.
What does a well-calibrated win probability model look like under LOSO validation?
Pooled across the held-out season, the set of plays the model rated near a given probability — say, 50% — should show a win rate close to that same number. Systematic gaps between predicted and observed rates in a specific bucket, like late-game two-score situations, point to a feature or interaction term still missing from the model.