Wed 01 January 2014
How do we continue to improve the model?
So far we've used a fairly simple set of features in the win probability model. We saw that it performed pretty well on the training set and performed slightly less well, but still much better than chance, on the test set. Now it's time to delve deeper into increasing the accuracy of the model and assessing attempts to do so. This involves two processes: feature engineering and model evaluation.
I already covered some of the details of model evaluation in the previous post in this series. We'll use the metrics I introduced in that post to assess how good our feature engineering attempts are. Feature engineering is the process of selecting the features that you think will predict your target, transforming them to reflect their relationship with the target, and potentially creating new features out of the information you have to get the information you want.
We want to find the features that best classify plays as belonging to winning or losing teams, so we want to try a variety of them. However, every time you add new features to the model, you increase the complexity of the model which can, in turn, lead to overfitting. This means that you'll need to test how well your model does out of sample each time you add new features to the model to see if it improves or decreases the model's accuracy.
In doing so, though, you run the risk of overfitting to the testing set. By repeatedly checking to see how well the model predicts out-of-sample on the same sample, you're essentially using the testing set as an extension of the training set. One way to work around this is to have a small dataset called a validation set that you use for all of this model tuning before you finally roll to evaluating its accuracy on the test set. For simplicity, I'm not going to do that right now, but we'll return to it in a later post. For a good explanation of the differences between the training, validation, and test sets, see this Stack Overflow question.
We've already engaged in some feature engineering -- by transforming the minutes and seconds to a 'seconds adjusted' feature. We're going to add three more features to the model: the Vegas line, the Vegas total line, and a new transformed time feature. The first two are fairly obvious -- including the spread will allow us to incorporate (perceived) team strength into our estimates and the total line will allow for situations when various amounts of offense are expected.
The third feature is a way to account for the fact that time has an unevenly distributed effect on win probability. Being down by 9 points in the first quarter isn't great, but it's not the kiss of death, but being down by 9 points with less than a minute remaining in the game is a very different situation. So we need a number that stays relatively constant but starts changing more as the number of seconds remaining increases. For this feature, we'll use 1/sqrt(seconds remaining + .01).
The training data, from Armchair Analysis, already conveniently has the spread and over/under in the data. The test data does not, so some data munging is required. I'll post the code shortly. I obtained all of the Vegas data here.
Let's look at two models. The random forest model that we've been using up to now, which we'll call the 'limited model', and the new random forest model that includes our new features, which we'll call the 'full model.' You can click on any image to see a larger version.
It looks like the full model, with the blue line, does do slightly better than the limited model, the purple line. To verify, let's look at some evaluation statistics.
[table id=4 /]
WOW! That's not much of an improvement. Surprising, given how much new information we've included. Only a percent improvement in accuracy here and there. While there were some non-trivial gains in the training set, the out-of-sample performance didn't change much (although it didn't worsen, which is always a good thing). This is definitely something we'll want to revisit over time. Feature engineering and model selection is an ongoing process.
Since we're trying to make predictions about future events, we want to know the model's weak spots. We've gotten a birds-eye view with the above metrics, but is this accuracy evenly distributed throughout game situations? Let's find out. One way to look at this is to look at the average accuracy of the model at various points in the game. To do this, I've tested the model for each minute from 0 to 60 and plotted the accuracy score.
Unsurprisingly, the model gets more accurate as the end of a game approaches. Notice that the accuracy increase in the test set isn't as smooth as in the training set. This is to be expected. Interestingly enough, there are actually points where the limited model is more accurate out-of-sample than the full model, though not consistently. This just underscores the idea that there usually isn't the answer to a modeling problem often isn't 'a model', but rather 'several models.'
Now for something really interesting. Since we're making predictions about wins, one metric we might be interested in is how calibrated our estimated probabilities are. Plays that a well-calibrated model estimates as having a win probability of 0.5 should be wins about 50% of the time; a win probability of 0.75 should belong to winning teams about 75% of the time, and so on.
This is super interesting. Essentially what we see here is that the model is wrong at the extremes, but in different ways. Teams with very, very low win probabilities still end up winning the game about 20% of the time. Teams with win probabilities as high as 80% may only win the game 60% of the time. I've also included the 95% confidence interval here because this is based on a relatively small test set.
It's important to think about this, especially when you see games with wild win probability graphs. Thinking about this is one of the reasons I started this project in the first place. This is a really important plot, and not one you'll see on a lot of sites. I tweeted about this, and Tempo Free Gridiron, who produces win probabilities for college football, helpfully produced one as well.
Brenton Kenkel asked me on Twitter why the errors in the above plot aren't symmetric -- i.e., why isn't the model wrong the same amount of time around 0 and around 1? I don't have an immediate answer for this, but it's a good question! I'd love to hear your thoughts.
Let's revisit the Baltimore-Denver game that opened up the 2013 season and that I used as an example in a previous post. Here's the win probability for that game using the full model. I've altered the uncertainty estimates, though, and am instead using a bootstrapped 95% confidence interval based on bootstrapped samples of the estimated probabilities from each tree of the random forest.
The bootstrapped confidence interval got rid of some of the wild swings in uncertainty in the final portions of the game, but the interval also appears to be wider in general than the previous one. That being said, I think I trust these estimates more than the previous hack-y ones.
Also of note is that Denver, which was a 7.5 point favorite in this game, starts the game with approximately a 75% win probability and, despite trailing at the half, never dropped below a 50% win probability (although the confidence interval at halftime stretches from 20% to 90%. That tells a much richer story than the \~58% mean win probability expressed by the single blue line.
Progress! Next up, I'm implementing a tool that will allow you to interactively compare and plot game situations with all of the uncertainty and evaluation metrics included. I'm still working on live, in-game win probability estimates for the playoffs. I don't know if that will happen or not.
The code for this post will be posted ASAP!