Cross-Validation for Sports Prediction Models

Cross-validation is the standard way to estimate how a model will perform on games it hasn’t seen, but the standard method — random k-fold splitting — assumes the rows being split don’t have a time order that matters. Sports data almost always does. A model trained on games from every part of a season and tested on a random subset of the same season has, in effect, been allowed to see the future before predicting it, and the resulting score describes that leak more than it describes the model.

Why Ordinary K-Fold Cross-Validation Leaks Information in Sports Data

Standard k-fold cross-validation divides a dataset into k roughly equal, randomly assigned parts, trains on k−1 of them, and tests on the one left out, rotating which part is held out until every row has served as a test case once. The randomness is the point for most machine learning problems — it breaks up any accidental ordering in the data and gives every fold a representative mix.

That same randomness is the problem for a sports model trained on a season of games. A random assignment will scatter early-season and late-season games across every fold, meaning a model can be trained on November and December games and then tested on a September game from the same season. If any input feature reflects information that accumulated over the season — a team’s rest-of-season point differential, a stat computed with the full season’s sample, anything that implicitly “knows” how the year turned out — the test score for that September game is no longer measuring genuine predictive skill. It’s measuring how well the model can recall information it was never supposed to have at prediction time.

Time-Series Cross-Validation: Training on the Past, Testing on the Future

The fix is to make the split respect chronological order. A time-series split — TimeSeriesSplit in scikit-learn — always trains on an earlier block of games and tests on a later one, and repeats this across multiple folds by moving the cutoff point forward each time instead of reshuffling the data. The first fold might train on weeks 1–4 and test on week 5; the next trains on weeks 1–5 and tests on week 6; and so on. Every fold mirrors the actual situation a model faces in production: some history is known, the next game is not.

This is sometimes called an expanding-window split, because the training set grows with each fold, not staying a fixed size. A fixed-size rolling window — always training on the most recent N weeks, not everything available — is a variant worth testing separately when there’s reason to think older data has gone stale, such as after a major rule change shifted how the sport is played.

What Leakage Across Seasons Actually Looks Like

Leakage isn’t always as obvious as testing on the past. A subtler and more common version happens across season boundaries: suppose a model uses a team’s prior-season point differential as an input feature for predicting this season’s games. That’s legitimate — the prior season is genuinely known information at the start of the new one. The leak creeps in when that same “prior-season” feature is accidentally computed using data through the end of the season being predicted, not frozen at the actual point of use. The split can be perfectly time-ordered by game date and still fail, because the feature itself was built with information from later than its row claims to represent.

A clean way to catch this class of error is to reconstruct, for a handful of spot-checked rows, exactly what a feature’s value would have been on the real-world date of that row — not what the finished dataset says today. If a “current season win total” column for a Week 3 game already reflects the team’s Week 17 record, the leak is in the feature engineering, not the split, and no amount of switching from k-fold to time-series cross-validation will fix it.

A scikit-learn Example: Time-Ordered Splits on Weekly Game Data

The following uses nflverse’s freely available games.csv release (schedules and results, CC-BY-4.0) to build a time-ordered cross-validation loop for a single season, sorted by week before splitting so the folds respect chronological order.

import io
import requests
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

url = "https://github.com/nflverse/nflverse-data/releases/download/schedules/games.csv"
r = requests.get(url, timeout=30)
r.raise_for_status()
games = pd.read_csv(io.BytesIO(r.content), low_memory=False)

season = games[(games["season"] == 2025) & games["result"].notna()].copy()
season = season.sort_values("week").reset_index(drop=True)

# Suppose these columns hold pre-game features known before kickoff
X = season[["home_rest", "away_rest", "div_game"]].fillna(0)
y = (season["result"] > 0).astype(int)  # 1 if the home team won

tscv = TimeSeriesSplit(n_splits=5)
scores = []
for train_idx, test_idx in tscv.split(X):
    model = LogisticRegression(max_iter=1000)
    model.fit(X.iloc[train_idx], y.iloc[train_idx])
    preds = model.predict(X.iloc[test_idx])
    scores.append(accuracy_score(y.iloc[test_idx], preds))

print(scores)  # one accuracy score per fold, each tested strictly after its training data

Each fold in that loop only ever tests on weeks that come after everything it trained on — the structural guarantee TimeSeriesSplit provides, and the property a random k-fold split cannot.

Hyperparameter Tuning Adds a Second Leakage Risk

Cross-validation is also the standard tool for choosing a model’s settings — how many trees in a forest, how much regularization in a linear model — by testing several candidate settings and keeping whichever one scores best across the folds. That process introduces a leakage risk of its own, separate from the train/test split itself: if the same folds used to pick the best setting are also used to report the model’s final performance, the reported score is optimistic, because the setting was chosen specifically because it happened to do well on those particular folds.

The standard fix is nested cross-validation: an outer loop of time-ordered folds provides the final, honestly held-out performance estimate, while an inner loop — itself time-ordered, running only inside each outer training set — is used purely to select hyperparameters. The extra layer costs more computation, since a full inner search runs once per outer fold, but it keeps the number reported at the end uncontaminated by the tuning process that produced it. Skipping this step is a quiet way for a tuned sports model to look better in testing than it will perform on a genuinely new season, even when every individual fold was correctly time-ordered.

A Concrete Leakage Example: Tuning on a Full Season, Testing on Its Own Weeks

Suppose a model’s regularization strength is chosen by running a five-fold time-series split across an entire season and picking whichever setting produces the highest average fold score, and that same average score is then reported as “the model’s cross-validated accuracy.” Every fold individually obeyed chronological order — none of them trained on a week that came after its test week. But the regularization strength itself was selected using knowledge of how well each candidate performed across the whole season, including the later folds’ outcomes, before any of those folds were treated as unseen data. The final number quietly reflects a search that had access to the full season’s results, even though no single fold did. Reporting a separate, untouched final holdout — a block of games excluded from the entire tuning process, not just from an individual fold — is the most reliable way to catch this before it reaches a published number.

Practical Checklist Before Trusting a Cross-Validation Score

A cross-validation score on a sports model is worth trusting only after a short set of checks: the split is chronological, not random; every input feature has been spot-checked to confirm it reflects only information available at the row’s actual date, not information computed with the benefit of hindsight; and the first fold has enough training history to be meaningful, not fitted on a handful of early-season games. None of these checks are visible in the score itself — a leaked model and a clean one can both report a plausible-looking accuracy number, and only tracing the data back to its source distinguishes them.