Wed 23 September 2015
What should an analytically driven NFL franchise look like?
I've been having versions of this conversation a lot lately, with folks both inside and outside of the industry. Then, this morning I was catching up on Three Cone Drill, where Rivers and Danny were having a very similar discussion. It's almost certainly confirmation bias (more on that below), but it feels like we're having another one of those moments where some fans are pining for a more analytical NFL. Here are my very nascent thoughts on this topic. Ben Alamar, the newly appointed Director of Production Analytics at ESPN, has doubtlessly covered much of this material in his popular book. However, I must sheepishly admit to not having read it yet and therefore any duplication of concepts is coincidental.
Analytics is not just a matter of hiring someone who's good with spreadsheets (in fact, being good with spreadsheets may be a weak, but not deterministic, signal towards an inability to manage and analyze larger data sources). In fact, analytics is not just about number-crunching. It's about decision-making. Being an analytical organization necessitates hiring people who are knowledgeable in the statistical analysis of data but also in decision-making, game theory, behavioral economics, and (last but certainly not least) football. This may seem like a broad range of topics, but fundamentally so much of what an organization does is about making decisions with data under conditions of imperfect information and very limited time. Thankfully, this is an extremely well-studied area with lots of robust empirical findings. Chances are you're not going to find a single person to fill this role, but rather a team of people with complementary skillsets.
The analytics division should hold coordinator-level authority and be on the same horizontal as the other coordinators in the org chart. This means not treating the position like an internship or an entry-level position. It means hiring both a leader and an individual contributor. Being a coordinator is as much about management as it is about expertise. The team must be prepared to look outside of the current football hiring pool for this person/persons and be prepared to spend the required money to secure talented individuals who are highly sought after for non-football positions. Organizational diversity is good and leads to better decision-making. Hiring exclusively people with close personal connections to the game leads to groupthink and conventional wisdom.
The analytics coordinator should be comfortable not only conducting original research but relating and defending the findings of his or her subordinates. He or she should be familiar with all or most of the topics that I've covered on this blog and not fall prey to the common traps of overfitting, star-gazing/p-hacking, and overinterpreting spurious relationships. This person needs to be able to stand on equal footing in a senior coaches' meeting. Data-driven decision-making means not deferring to "football people" when the stakes are high. It also means being honest about how certain or uncertain one is about one's findings. Data-driven decision-making means not bending the results of the analysis or selectively interpreting findings. It means overruling traditional football people when the data are competently and thoroughly analyzed and differ with the conventional football wisdom. The analytics coordinator needs to be humble but firm and be adept at communicating complicated analyses to a skeptical non-expert audience.
EDIT: As the always insightful Seth Partnow pointed out, I want to stress that the analytics coordinator should absolutely be able to "talk football" with the coaches. The coordinator should have broad and deep football knowledge, not be an overzealous quant who thinks, with the right models, he or she can show up the football people. As a slight counterpoint to Seth, I also firmly believe that the coaches should be expected to speak analytics with the coordinator. The burden to be an expert in everything shouldn't be placed solely on the analytics coordinator.
This position will be unsuccessful without significant institutional support. It is incumbent, starting with ownership and working down the org chart, that an analytical and data-drive culture are seen as vital to the organization's survival and success. This means that the analytics coordinator provides an equal voice in team discussions, that the team adopts a process-driven rather than outcome-driven style of decision-making, and that employee evaluation is backed by this. Coaches who maximize their chances of winning by going for it on fourth down in the appropriate situations but that fail to convert are not punished. Conversely, coaches who overrule the analytics coordinator for a suboptimal decision but are still successful are subject to criticism. The latter is probably the most difficult hurdle for an organization, but not as much as one might think. Players that miss their assignments on plays are routinely criticized by coaches even if the play turned out successfully. Execution is about process and making decisions that put you in a winning position -- not rolling dice that are weighted against you and getting lucky.
A crucial part of institutional support is extending the time horizons of the people involved. So much of the decision-making in football is suboptimal due to the rotating door of management. Tying employee evaluation to cooperation with an analytical strategy helps with this. But the opposite needs to be true -- the analytics coordinator and his or her subordinates need to have assurances that they are working in a stable, future-oriented organization rather than one that will sweep away their jobs with the next general manager.
This further means that the analytics coordinator needs to have equal footing in how their input is implemented, including in-game decision-making, player evaluation, draft research, and game planning. This does not mean that the analytics coordinator trumps the football people, only that their voice is an equal one.
In order to do so, the analytics coordinator will need a budget that allows him or her to recruit and retain talented employees and purchase and train on the necessary technology. It's quite remarkable how often a few hundred thousand dollars are seen as essentially rounding error on a replacement level player's contract, but could employ an analyst with every available database for several years.
As referenced above, hiring is a tricky practice, especially for an organization that is currently not analytically driven. I've talked about this on Twitter before, but I'll reiterate my thoughts here. There is a serious cold start problem here -- how does an organization with no analytics talent successfully evaluate candidates for an analytics position? In the absence of the ability to competently evaluate talent, most organizations will tend to favor easily observable signals for candidate quality -- an Ivy League degree, an MBA from a prominent business school, a career in finance. These may be useful heuristics, but they're also highly noisy.
Teams hire consultants for a variety of tasks all the time. I'd recommend doing the same for the initial analytics hires. There are plenty of highly visible people in the analytics world who don't/can't work for teams who are still involved with teams and have their respect. Hire these people as consultants to help in the search. Consider looking to the tech world and talking to people who have hired data scientists or advanced analysts before. Also consider talking to academics in statistics and decision sciences departments. Have a review system in place where the resumes and interviews are reviewed independently (i.e., without interaction between the reviewers) to get a semi-unbiased view of the candidates. Hire a consultant to help you draft up some homework / audition problems for people. Don't just go for a flashy degree or connections in the sport.
An analytically driven NFL franchise invests in learning about the core cognitive biases and implements organizational and institutional safeguards for counteracting them. Every member of the analytics organization should, at the very least, read Thinking Fast and Slow and understand how decision-making is systematically affected by the shortcuts our brains make to make our lives easier. Further, the non-analytics organization needs to be coached on these things in a way that connects with their daily duties. This is extremely important material and communicating it in a way that makes it compelling for other coaches is vital. Coaches need to understand loss aversion. General managers need to understand recency bias in player evaluation. For God's sake, coaches should understand confirmation bias.
Most teams spend a significant amount of time drilling on highly specific situations -- the ball has been fumbled, when is it appropriate to knock the ball out of bounds? Players and coaches spend a great deal of time making these decisions become automatic. The same rigor should be demonstrated on the coaching end. When is it worth it to take a five-yard delay of game penalty instead of burning a timeout? How can we avoid burning a timeout and then still punting the ball? The analytics coordinator should be in the booth with the other coaches providing information about this to the head coach in real time. When this information is provided in-game, the analytics coordinator must have buy-in and authority in order for this system to be effective.
This is a multiple year project. The organization cannot expect magic from the analytics team ever, but certainly not in the first year. A team embracing this approach will have to evolve in the way that it hires coordinators, evaluates employees, and conducts daily business. This might mean pulling from less well-known talent pools, recruiting more heavily from college, and taking some risks. The first year should be able establishing the personnel and technological infrastructure needed to move forward. The analytics coordinator and his or her team should take on an advisory and educational role during this time period. Subsequent seasons should see an increase in involvement until the ideal of an equal voice is met.
Along the way, the analytics organization must adopt a different stance than I often seen in the analytics world. They must not be condescending, they must make themselves part of the team. Senior leadership and ownership must reinforce this. Doing so will mean being a little less secretive than many teams are currently comfortable with -- acknowledging the importance of their analytics organization, allowing the coordinator to speak with the press and be seen as an integral part of leadership. The culture should be one of collaboration between analytics and football operations.
Various teams in the leagues are doing some of these things to various degrees already, but to my knowledge, no one is "all in" on being an analytically driven franchise. There's a tremendous amount of low-hanging fruit for an enterprising team to grab. Teams are making demonstrably bad decisions every week -- this isn't even debatable or "lying with statistics."
These are not just pie-in-the-sky ideas. Some team is going to really latch onto these ideas and exploit the hell out of them for a while. It's simply a matter of which team and when. Playing catchup is going to be a lot more difficult at that point. All I can say is: