How Consistent Are Football Rating Systems With Each Other?

Two football rating systems can look similar or different depending only on which teams a reader happens to glance at first — rank correlation replaces that impression with a single number computed over every team at once.

Turning “these rankings look similar” into a number

Every method covered elsewhere in this series — Massey least squares, the Colley matrix, a continuous-time Markov chain rating, an eigenvector power rating, Elo — is built from a different mathematical mechanism, so it would be surprising if any two of them produced identical rankings for the same season. The more useful question is not whether two rankings match exactly, but how closely they track each other, and by how much they diverge where they do disagree. Rank correlation statistics answer exactly that question.

Spearman’s rho: squared rank differences

Spearman’s rank correlation coefficient, when there are no tied ranks, is

$$\rho =1-\frac{6\sum d_i^2}{n(n^2-1)}$$

where d_i is the difference between the two systems’ rank for team i (rank 1 minus rank 1, rank 2 minus rank 2, and so on, team by team) and n is the number of teams being compared. The coefficient ranges from -1 to 1: a value of 1 means the two rankings are identical for every team, -1 means one ranking is the exact mirror image of the other, and 0 means no consistent relationship between them at all. Because d_i is squared, Spearman’s rho is particularly sensitive to a team that lands in wildly different positions in the two rankings — one big rank swap can move the statistic noticeably even if every other team agrees closely.

Kendall’s tau: counting agreements and disagreements directly

Kendall’s tau takes a different approach, working pair by pair, not rank by rank. For every possible pair of teams, the pair is concordant if both rankings agree on which team is higher, and discordant if the rankings disagree. Tau is then

$$\tau =\frac{(\text{concordant pairs})-(\text{discordant pairs})}{n(n-1)/2}$$

with the denominator being the total number of possible team pairs. Like Spearman’s rho, tau ranges from -1 (perfect disagreement) to 1 (perfect agreement), with 0 representing what would be expected if the two rankings were unrelated. Kendall’s tau is generally considered more robust to a small number of large rank swaps than Spearman’s rho, since every pair contributes equally to tau regardless of how far apart the two teams end up, while Spearman’s rho weights big single-team rank differences more heavily through the squared term.

A worked example, by hand

Suppose five hypothetical teams get ranked by two different systems as follows:

Team	System A rank	System B rank	d_i	d_i^2
Alder	1	1	0	0
Birch	2	3	-1	1
Cedar	3	2	1	1
Douglas	4	5	-1	1
Elm	5	4	1	1

Summing d_i^2 gives 4. With n = 5:

$$\rho =1-\frac{6\times 4}{5(25-1)}=1-\frac{24}{120}=0.80$$

A Spearman’s rho of 0.80 indicates strong, though not perfect, agreement — consistent with what the table shows directly: the two systems agree exactly on the top team and disagree only mildly, by one position each, everywhere else. Working the same five teams through Kendall’s tau: out of 10 possible pairs, 8 are concordant and 2 are discordant (Birch-Cedar and Douglas-Elm each swap order between the two systems), giving tau = (8-2)/10 = 0.60 — a lower number than rho for the identical pair of rankings, illustrating directly that the two statistics are not interchangeable even when applied to the same data.

Consistency within one method, not just between two of them

Rank correlation is equally useful for a question that has nothing to do with comparing two different systems: how stable is a single method’s own output. One practical check is a split-sample comparison — compute the same rating method on only the first half of a season’s games, compute it again on the second half, and measure the rank correlation between the two resulting orderings. A method whose early-season and late-season rankings correlate weakly is telling its user something specific: either the method is highly sensitive to a small sample of games, or the teams themselves genuinely changed enough during the season that an early snapshot should not be expected to match a late one.

A second practical check is leave-one-out stability: recompute a rating with a single game removed from the schedule and see how much the resulting ranking shifts. A method where removing any single game barely changes the ranking (high rank correlation between the full-schedule and one-game-removed versions) is behaving the way most analysts want a rating to behave — reflecting the balance of an entire season’s evidence, not being unduly swayed by any one result. A method where removing one particular game meaningfully reshuffles several teams’ positions is more fragile than it might look from a single computed ranking alone, and that fragility is invisible unless it is specifically tested for, not assumed away.

What actually drives disagreement between football rating systems

In practice, disagreement between reasonable football rating systems is not evenly spread across a ranking — it concentrates in specific, identifiable places. Teams with very few losses tend to rank near the top of essentially every method, because a long unbeaten run is one-sided evidence that is hard for any reasonable formula to read differently. The middle of a ranking is where real disagreement shows up: a team with a handful of close losses might rate strongly under a margin-of-victory-sensitive method like Massey least squares (since those losses were narrow) while rating lower under a margin-blind method like the Colley matrix (which only sees losses, not how close they were). A team that padded its record against weak opponents shows the opposite pattern.

This is also where a method’s tunable parameters matter most directly. A continuous-time Markov chain rating’s bias parameter p, or an Elo system’s k factor, mostly affect how much separation exists between similarly-situated teams, not flip the order of teams that are already clearly ahead of or behind one another — which is one reason rank correlation between two parameterized methods tends to stay high even when the underlying rating values differ meaningfully. Comparing rankings with Spearman’s rho or Kendall’s tau, not trusting a side-by-side glance at two top-25 lists, is the more defensible way to state how much two methods actually agree — and, just as usefully, precisely where they don’t.