A finding that confirms what everyone already assumed rarely gets written up, shared, or remembered. A finding that contradicts conventional wisdom does — regardless of whether it’s actually correct. That asymmetry is the engine behind publication bias, and sports analytics, with its constant appetite for a new edge or a debunked myth, is an especially fertile environment for it.
Why Surprise Is Not the Same as Truth
A counterintuitive claim carries an implicit narrative that makes it more shareable: everyone else has been wrong, and here is the data that proves it. That narrative is appealing independent of whether the underlying statistics actually support it, which means the population of counterintuitive claims that circulate widely is not a representative sample of counterintuitive claims that are actually true — it’s biased toward whichever surprising claims happened to be interesting enough, or well-timed enough, to spread, filtered only loosely by whether they held up to scrutiny.
This creates a specific trap for anyone building or reading sports models: a surprising result that comes out of a dataset feels like a discovery, and the temptation is to treat “surprising” as itself a kind of evidence — surely a boring, expected result wouldn’t be worth writing up, so the surprising one must be the real finding. That reasoning has it backwards. A genuinely surprising result needs the same evidentiary bar as an unsurprising one — sample size, out-of-sample validation, a method decided in advance — and arguably a higher one, precisely because the incentive to notice and publicize it exists whether or not it’s real.
The Hot Hand: A Documented Reversal
The clearest documented case of this pattern comes from basketball, not football, but the methodological lesson transfers directly. Gilovich, Vallone, and Tversky’s 1985 study is the canonical source of the “hot hand fallacy” — the conclusion that a player’s streak of made shots does not predict a higher chance of making the next one, and that the widespread belief in streak shooting is a cognitive illusion, not a real statistical pattern. The finding was itself counterintuitive at the time (it contradicted what players, coaches, and fans all believed) and it spread widely for exactly that reason.
Decades later, researchers Joshua Miller and Adam Sanjurjo identified a subtle statistical bias in the specific test the original study used: the method for estimating the “hit rate after a streak of hits” systematically underestimates that rate in finite sequences, in a way that pushes the result toward finding no hot-hand effect even when one is present. When Miller and Sanjurjo reanalyzed the same original data with a bias-corrected version of the test, the conclusion reversed — the hot hand effect was there.
The lesson isn’t “the hot hand is definitely real” or “the original researchers were careless” — the original study was a serious piece of work that stood up to scrutiny for decades. The lesson is narrower and more useful: a counterintuitive finding that overturns intuition can itself later be overturned by a more careful look at the method, and the first counterintuitive result being widely cited for thirty years did nothing to protect it from that correction.
Icing the Kicker: A Football Example That Never Settled
Football has its own long-running counterintuitive-claim dispute: whether calling a timeout right before an opposing kicker’s field goal attempt — “icing the kicker” — actually reduces the kicker’s chance of making it. Some studies, including a widely cited analysis of 2002–2003 NFL field goals published in the journal Chance, found a real drop in accuracy on iced pressure kicks in the 40–55 yard range. Other analyses, covering different season ranges and using different samples of “pressure” situations, found close to no difference between iced and non-iced kicks.
What makes this a useful case study isn’t which side is right — the research genuinely hasn’t settled that question. It’s that the disagreement traces back to exactly the kind of design choices that make a counterintuitive claim easy to produce and hard to verify: which seasons are included, how “pressure kick” is defined, and how large the sample is once that definition narrows the pool of qualifying kicks down. A small sample dressed up with a specific, narrow definition of the situation being studied can produce a striking headline number that a broader or differently defined sample doesn’t reproduce.
The File-Drawer Problem
Publication bias has a well-known companion sometimes called the file-drawer problem: for every published finding that a particular pattern exists, there may be several unpublished analyses that looked for the same pattern, found nothing, and were never written up — because a null result rarely feels worth the effort of publishing. Statistician John Ioannidis formalized a version of this concern in a widely discussed 2005 essay arguing that, once selective reporting and small sample sizes are accounted for, a large share of published research findings across fields turn out not to replicate; while his specific estimates have been debated, the underlying mechanism he described is not.
Sports analytics has an especially strong version of this dynamic, because the barrier to publishing an analysis has essentially disappeared — a single interesting chart and a blog post or social media thread is enough, with none of the peer review, replication expectation, or file-drawer visibility that even loosely regulates academic publishing. Dozens of analysts can independently test the same kind of pattern — some situational split, some late-game tendency — across a season, and simple chance guarantees a handful of those tests will turn up something that looks statistically striking, purely from the number of tests being run. The ones that come up empty are never written about; the one that comes up looking interesting gets shared widely, and nothing in that process distinguishes a real signal from the expected rate of false positives across that many independent tests.
A Practical Checklist for Reading a Surprising Claim
Three questions cut through most of the noise around a counterintuitive sports-analytics finding. First, how large is the sample it’s based on — a striking pattern in a few dozen plays carries far less weight than the same pattern across several seasons. Second, has it been checked against a second, independent dataset, rather than just re-described from the same data it was first noticed in — a pattern that only shows up in the exact sample where it was discovered is a strong sign of overfitting rather than a real effect. Third, was the specific test being reported decided on before the data was examined, or does it look like one particular slice of the data (a specific range of point spreads, a specific subset of seasons) was chosen after the fact because it happened to produce the interesting number.
None of these checks require treating every surprising claim with suspicion by default. They require applying the same standard of evidence to a claim that fits the “everyone was wrong” narrative as to one that doesn’t — which is, in practice, a higher bar than most counterintuitive claims are held to on first publication, precisely because being counterintuitive is what got them published in the first place.
Frequently asked questions
What is publication bias?
Publication bias is the tendency for striking, statistically significant, or counterintuitive results to get published, shared, and cited more readily than unremarkable ones — regardless of whether the underlying finding is actually more likely to be true. A body of published research can end up systematically overrepresenting surprising claims relative to how often surprising things actually turn out to be real.
Why do counterintuitive findings spread faster than ordinary ones?
A finding that confirms what everyone already believed isn't especially interesting to share, publish, or write about; a finding that contradicts conventional wisdom is. That asymmetry exists independently of whether the counterintuitive finding is actually correct, which means the incentive to produce and circulate surprising claims is stronger than the incentive to check whether they hold up.
What's a documented example of a famous counterintuitive sports finding that didn't hold up as first reported?
The 'hot hand' research in basketball is the clearest case. Gilovich, Vallone, and Tversky's widely cited 1985 study concluded that streak shooting was a cognitive illusion with no real statistical basis. Decades later, Miller and Sanjurjo identified a statistical bias in the method the original study used to test for streaks, and a corrected re-analysis of the same data reversed the original conclusion — the hot hand effect was there after all, just masked by the flawed test.
Is 'icing the kicker' in football a proven effect?
It's contested rather than settled. Some studies looking at NFL field goals under late-game pressure situations have found a meaningful drop in accuracy when the kicker is iced; other studies covering different seasons and sample sizes have found next to no difference. The inconsistency across studies is itself informative — it suggests any real effect, if one exists, is small enough that study design and sample choice can flip the conclusion.
How should a reader evaluate a surprising sports-analytics claim?
Check the sample size the claim is based on, whether the result has been tested on a second, independent dataset rather than just the one it was discovered in, and whether the analysis was specified before looking at the data or assembled afterward from whichever pattern happened to appear. A surprising finding that clears all three checks is worth taking seriously; one that only clears none of them is closer to noise dressed up as insight.
Does this mean counterintuitive findings are usually wrong?
No — some of the most important results in sports analytics are genuinely counterintuitive, and dismissing every surprising claim out of hand would be its own error. The point is that surprise is not evidence of correctness. A counterintuitive finding earns belief the same way an ordinary one does: through sample size, replication, and a method that was specified before the result was known, not through how interesting it sounds.