Why Uncertainty in Expected Points Estimates Matters

An expected points model outputs a single number for a given down, distance, and yard line — but that number is an estimate, not a fact, and treating it as exact hides how much it can move depending on which plays happened to end up in the training sample.

A Point Estimate Is Not the Whole Story

Carter and Machol’s original 1971 model was built from 8,373 hand-coded plays across 56 games from half of the 1969 season — a dataset small enough that a handful of unusual drives could meaningfully shift the estimated point value at a given yard line. A modern public model trained on a decade or more of play-by-play data is estimating from a far larger sample, but the underlying problem has not gone away: every expected points figure is still computed from a finite set of past outcomes, and a different sample of games would have produced a slightly different number.

This matters most where evaluation depends on comparing two closely spaced numbers — one team’s EPA per play against another’s, or one season’s figure against the next. If the gap between two teams’ EPA per play is smaller than the uncertainty around either estimate, calling one team “better” on that basis is stating more confidence than the underlying data supports.

Why Some Situations Are Estimated With Less Confidence Than Others

Not every down-distance-yard-line combination is equally common in an NFL play-by-play dataset. First-and-10 near midfield happens thousands of times a season; fourth-and-goal from inside the 1-yard line happens rarely, and first-and-10 pinned at a team’s own 1-yard line is rarer still. A 2024 paper by Ryan Brill, Ryan Yee, Sameer Deshpande, and Abraham Wyner, published on arXiv, put a number on how severe this gets: combinations of just yard line, down, and seconds remaining in the half produce more than 7 million distinct possible game states, against a training dataset the paper described as roughly half a million plays across a multi-season window. Most of those 7 million states are essentially never observed directly, and the model is left to extrapolate from nearby, more common situations — which is exactly where a machine-learning model’s estimate can behave unpredictably, since flexible models like the gradient-boosted trees used in nflfastR’s public model can produce counterintuitive, overfit-looking values specifically in these sparse regions.

The same paper reported that off-the-shelf machine learning models — evaluated by how often a stated 95% prediction interval actually contained the true outcome — achieved roughly 85% coverage instead of the intended 95%, meaning the models were more confident than their track record justified. That gap between stated and actual confidence is the practical cost of skipping uncertainty quantification: a model can look precise while quietly being wrong more often than its own numbers imply.

Bootstrap Resampling: Measuring Uncertainty Without Assuming a Distribution

One standard way to estimate how much an EP or EPA figure could plausibly vary is bootstrap resampling: draw a new dataset by sampling the original data with replacement, refit or recompute the estimate on that resampled dataset, repeat the process many times, and look at how much the resulting estimates spread out. Because it does not require assuming the data follow any particular mathematical distribution, the bootstrap works as a general-purpose tool for quantifying uncertainty in a model whose outputs are otherwise hard to derive a formula for — which describes a gradient-boosted expected points model well, since there is no simple closed-form expression for its prediction uncertainty the way there is for, say, a basic linear regression.

The Brill et al. paper tested two versions of this idea specifically for football play-by-play data. A standard bootstrap resamples individual plays as if each were an independent observation. A cluster bootstrap instead resamples entire drives as a single unit, preserving the sequence of plays within a drive. The distinction matters because plays within one drive are not independent: the second play’s situation is a direct consequence of the first play’s outcome, and the whole drive shares one ultimate scoring result that every play in it is implicitly trying to help predict. Treating plays as independent when they are not tends to understate how much real uncertainty exists. In the paper’s results, the cluster bootstrap came close to the intended 95% coverage rate, while the naive independent-play approach fell noticeably short — evidence that respecting the drive-level dependence structure, not just resampling more plays, is what closes the gap between stated and actual confidence.

A Hypothetical Illustration of a Bootstrap Confidence Interval

The mechanics are easier to see with a made-up example. Suppose an analyst wants a confidence interval around a hypothetical quarterback’s EPA per dropback over a 200-dropback sample, and the observed value is +0.12. A cluster bootstrap would resample the underlying drives (not individual dropbacks) with replacement many times — say, 1,000 resampled datasets — recompute EPA per dropback on each resampled version, and then read off, for instance, the 2.5th and 97.5th percentile of those 1,000 recomputed values to form a 95% interval. If that interval turns out to be roughly +0.02 to +0.22, the headline +0.12 figure is real but imprecise: a quarterback with a true underlying EPA per dropback anywhere in that range could plausibly have produced the observed +0.12 across 200 dropbacks by chance alone. A second, otherwise identical quarterback sitting at +0.08 over a similar sample size would not be distinguishable from the first in any statistically meaningful sense, even though +0.12 looks clearly better than +0.08 as bare point estimates.

This is a hypothetical illustration, not a reported result from any specific study — the exact width of a real confidence interval depends on the sample size, the variance of the underlying plays, and how correlated the drives in the sample are with each other. The purpose of the example is only to show what the interval represents: a range of values consistent with the observed data, not a single precise truth.

What This Means for Reading an EPA Number

None of this means expected points or EPA figures are unreliable in general — league-wide averages built from tens of thousands of plays a season are estimated with considerable precision. The caution applies specifically to smaller samples: a single quarterback’s EPA per dropback over four games, a team’s third-down EPA in one month, or any expected points figure for a rare game state near the edges of the field. In those cases, the number reported is a point estimate sitting inside a wider range of plausible values, and a bootstrap-derived confidence interval — or simply insisting on a larger sample before drawing a conclusion — is what separates a real signal from noise that happened to look like one. The companion piece on how EPA football and the expected points model work covers the mechanics this uncertainty attaches to; the same small-sample caution applies to any per-play or per-game statistic built the same way, including the win probability model covered separately.