“Better data” is a phrase that covers two entirely different problems in practice. One is which input variables a model uses — the columns. The other is which games, plays, or seasons it’s trained on — the rows. Feature selection and sample selection both count as improving the data, and a model with a problem in one of them can look identical, in terms of symptoms, to a model with a problem in the other. Telling them apart matters, because the fix for one does nothing for the other.
Feature Selection: Choosing the Right Columns
Feature selection is the process of deciding which input variables actually belong in a model. It sounds like it should always favor “more” — surely an extra stat can only help, or at worst do nothing — but that intuition breaks down once the number of candidate features grows large relative to the amount of data available to estimate their relationships. This is the practical core of what’s known as the curse of dimensionality: as the number of features increases, the volume of data needed to reliably distinguish real relationships from coincidental ones grows even faster, and past a certain point, additional features mostly add opportunities for a model to fit noise rather than signal.
A football-specific version of this shows up constantly: a model predicting game outcomes has access to dozens of candidate stats — yards per play, third-down rate, turnover margin, red-zone efficiency, and many derived combinations of all of them — and it’s tempting to include everything available. With only a season or two of games to train on, a model given thirty correlated features has far more room to fit patterns that happened to exist in that specific sample by chance than a model given the five or six features with the clearest, most defensible connection to winning football games.
Feature selection, done well, isn’t about finding the largest possible set of inputs — it’s about finding the smallest set that still captures what actually matters, which tends to produce a model that generalizes better to games it hasn’t seen, not just one that’s simpler to explain.
Sample Selection: Choosing the Right Rows
Sample selection is a completely separate question: given a fixed set of features, which observations does the model actually learn from? A biased sample can sabotage a model with perfectly reasonable features, because the model is only ever able to learn patterns that are present in the rows it’s given.
The clearest illustration of how badly a biased sample can mislead an analysis is survivorship bias, most famously demonstrated by statistician Abraham Wald’s work analyzing WWII aircraft damage. Military planners wanted to add armor to the areas of returning bombers that showed the most bullet damage. Wald pointed out the flaw: those were the planes that survived being hit in those locations. The planes that took damage to the areas with no visible bullet holes on returning aircraft were the ones that never made it back — the sample of “planes available to study” was filtered by the very outcome being investigated, and the most important information was in the data that was missing, not the data that was present.
Football sample selection has its own version of this risk. A model built only on games from playoff-bound teams, or only on players who accumulated enough snaps to clear a stat-reporting minimum, is working from a sample that’s already been filtered by the outcome it may be trying to explain or predict. A player who got injured or benched before reaching that snap threshold doesn’t disappear from football — they disappear from the dataset, and a model trained on the survivors alone will systematically underestimate how often that kind of early exit happens.
Why Neither One Fixes the Other
The reason it’s worth treating feature selection and sample selection as genuinely separate problems, rather than two flavors of “better data,” is that a mistake in one is invisible to fixes applied to the other. No amount of clever feature engineering recovers information from games that were never included in the sample in the first place — if a filtered dataset systematically excludes canceled, forfeited, or otherwise unusual games, that gap in coverage exists regardless of how many or how well-chosen the input columns are. And no amount of expanding the sample — more games, more seasons, more players — fixes a model whose feature set simply doesn’t capture the thing that actually drives the outcome; a bigger sample of the wrong inputs just estimates the wrong relationship more precisely.
A Worked Hypothetical: Same Symptom, Two Different Causes
Suppose two versions of a model that predicts whether a team covers the point spread both post a disappointing 51% hit rate against a holdout set of games, barely better than a coin flip. In the first hypothetical version, the feature set includes a stat that’s actually a late-arriving proxy for the final score margin itself — effectively letting the model see a shadow of the answer during training, which makes it look fine in-sample and then collapse once that shortcut isn’t available in a genuinely blind test. That’s a feature problem: the input set doesn’t hold up once the model can’t lean on a feature that was secretly correlated with the outcome it was never supposed to have access to.
In the second hypothetical version, the feature set is reasonable and stable, but the training sample was built entirely from a single team’s games over a stretch where that team was unusually inconsistent — an injury-hit season with several backup-quarterback starts mixed in with games the starter played. The model learned patterns specific to that one volatile stretch and never saw the range of situations a general spread-coverage model needs to generalize across. That’s a sample problem: nothing wrong with the columns, something wrong with which rows were available to learn from.
Both versions produce the identical headline number — 51% — and from that number alone, it’s impossible to tell which of the two is actually happening. Only tracing the failure back to its source (checking whether a feature is secretly leaking outcome information, versus checking whether the training games represent a reasonably normal range of situations) separates the two, and only that diagnosis points to the fix that will actually work.
Diagnosing Which One Is Actually Broken
The practical value of separating these two concepts is in diagnosis. A model that performs inconsistently across different feature subsets — swapping one stat for a similar one changes the outcome meaningfully — is showing a feature selection symptom: the inputs aren’t stably capturing the underlying signal. A model that performs well under cross-validation on its training data but falls apart against a genuinely new season, a different level of competition, or a differently constructed dataset is showing a sample selection symptom: something about how the original sample was assembled doesn’t represent the situations the model is now being asked to handle.
Reaching for the wrong fix — adding more features to a sample problem, or expanding the sample to fix a feature problem — can burn significant effort without addressing the actual cause, and can even make a biased model look more confident without making it more correct. Identifying which of the two is actually broken, before trying to fix either, is the step that’s easy to skip and expensive to skip wrong.
Frequently asked questions
What's the difference between feature selection and sample selection?
Feature selection is choosing which input variables (columns) go into a model — which stats, which situational flags, which derived metrics. Sample selection is choosing which observations (rows) go into it — which games, which seasons, which players. Both shape what a model can learn, but they fail in different ways and neither one fixes a mistake made in the other.
Why does feature selection matter for a football model?
Adding more input variables doesn't automatically make a model better — past a point, extra features that don't carry real signal add noise the model can fit to by chance, especially when the number of games or plays available is small relative to the number of features being considered. This overfitting risk is described in machine learning as part of the curse of dimensionality: as the number of features grows, the amount of data needed to reliably estimate their relationships grows even faster.
What is survivorship bias, and how does it relate to sample selection?
Survivorship bias is a specific sample-selection error: drawing conclusions from a dataset that only includes cases which 'survived' some earlier filter, while ignoring cases that didn't make it that far. The classic illustration is Abraham Wald's WWII analysis of returning aircraft — reinforcing the damage visible on planes that made it back would have missed the damage pattern on the planes that didn't return at all, which was the more important data.
Can a good feature selection process fix a biased sample?
No. If the underlying sample is systematically missing certain cases — every game from a canceled or shortened season, every player who got cut before accumulating enough stats to qualify — no choice of input variables can recover information that was never collected. Feature selection can only work with the rows that are actually present; it has no way to correct for which rows are absent.
Can a well-chosen sample fix a feature selection problem?
Also no, for the mirror-image reason. A perfectly representative, unbiased sample of games still won't produce a good model if the input variables it's given don't actually capture what drives the outcome being predicted — a model can have plenty of clean, representative data and still underperform because none of its features carry the relevant signal.
How should a football analyst prioritize between the two?
Diagnose which one is actually broken before trying to fix either. If a model performs inconsistently across different slices of the same feature set, that points toward a feature problem — the inputs aren't capturing the right thing. If a model performs well in cross-validation but fails against a new season or a different competition level, that points toward a sample problem — something about how the data was collected or filtered doesn't generalize. Treating a sample problem with more feature engineering, or a feature problem with more games, wastes effort without fixing the actual cause.