the spread the (data) science of sports

Win probability plots -- useful tool?

Sun 18 October 2015

It's been an interesting weekend for win probability models. In case you missed it, on Saturday, Michigan State improbably returned a fumble for a touchdown to win a game in which they never held a lead. This morning, ESPN Stats and Info tweeted the following, indicating that MSU had, in a single play, transitioned from a 0.2% win probability to 100%.

Following up on this, we saw wild swings in the Broncos - Browns game, with a number of football analytics accounts tweeting pictures of the wildly oscillating and criss-crossing win probabilities. I jokingly tweeted that '[e]ventually all of football analytics Twitter will just be hundreds of very slightly different win probability graphs.'

A bit snarky, to be sure, but I think it underscores an important point I made more forcefully about the state of football analytics.

What are these plots telling us?

In the MSU example above, the plot is essentially telling us that MSU lodged a very unlikely victory on the last play of the game -- but we already knew that! The plot doesn't tell us much, other than drawing a straight line between 0.2% and 100%. In terms of data-ink ratio, it's probably more informative to just read the previous sentence than to plot it.

What is the lesson from the win probability plot above? Don't fumble a punt? Don't lose a game on the last play?

Win probability plots provide an intuitive and important way to summarize the events of the game. It's easy to see if one team dominated, if the game had a bunch of lead changes, and so on. But it feels as if in order to be "in football analytics" these days, you need to produce win probability plots in real-time.

A lack of innovation

To me, this feels like stagnation. Win probability underscores a large amount of work in our field, but how we use win probability to drive decision-making and understanding of the game is what's interesting and innovative. Win probability in and of itself is a mostly descriptive tool that many will (fairly) argue just gives us a single-number summary of a complicated game. Win probability plots are mostly backward-looking, telling us what has happened and if it was 'important' or not.

One of the reasons I've enjoyed working on the NYT 4th Down Bot so much is that it's proactive and makes live calls about what the optimal decision is. Does it do it perfectly every time? Of course not, but we're making an effort to shift from "that was a dumb call" to "here's the call you should make."

Do I want win probability plots that incorporate some measure of uncertainty about our estimates? Of course. That's one of the first things I investigated when I started this blog. But I don't think that this will be a game-changer for many readers. And, to be clear, I'm not arguing we should do away with win probability plots -- just that they constitute well-trodden territory.

How we got here

There's a significant amount of isomorphism in sports analytics. This makes sense, and I'm certainly guilty of it. Important problems have been identified by people we respect. One of the ways we learn is to try and replicate and improve upon their work. However, we shouldn't stop there. We should be asking what kinds of interesting questions we can answer with win probability models, how we can advance our knowledge about the game, and how we can communicate that information to people that are interested and invested.

For those of us who are interested in changing the product that we watch every Sunday (and Monday and Thursday and Saturday), I have a proposal.

Let's do original and open research that builds upon existing win probability models rather than treating win probability itself as the goal. Let's work on research that's both interesting to us and helps move the needle towards a more analytical and rational NFL. Let's figure out how to model the interactive and complex nature of individual player contribution. Let's tackle player fit and figure out how to model and predict how players will perform when they change teams.

Let's stop ignoring statistics and machine learning and start doing careful and rigorous work that stands up to both football and analytical challenges.

I'll do my best to practice what I preach.

blog comments powered by Disqus