Football and NFL sabermetrics

My friends and I are huge football fans, and recently we got into a ridiculously long email discussion about the study of football, how to get smarter about analyzing it, and the natural obstacles to doing so.

It was fun to think about, and thought it would be worth saving onto the blog (bonus: it counts as a blog post, much of which was contributed by other people!)

These emails were slightly edited for length and content, as well as superfluous profanity and racial slurs.

To: Everyone

From: Matt

Subject: the state of my football fanhood

so yesterday it became clear to me upon watching the eagles that michael vick is actually a very good quarterback nowadays. i had previously thought that vick was overrated. i now feel like i was wrong. my issue is that i dont trust the media or anyone to give me information about football.

here's the issue. for two decades, i was a big phillies, buckeyes, and eagles fan. i listened to what the announcers/beat writers/espn told me, and they all painted this same view of sports with this movie-like depiction of sports as the team who tries harder always triumphs, and how 99% of success is mental, how you can always infer who is the best team from who won the game, especially a big game. and now i know that is all bullshit. i know better now. having researched baseball, i now know that announcers and beat writers are always doing a mixture of trying to fill space, projecting things onto players, and trying to make it seem like they know something that no fan could know on their own.

this has made baseball viewing immeasurably more enjoyable for now. because of my own research, i can watch a pitcher and know that he is more likely to continue his success if he's blowing the ball past hitters instead of inducing shallow flyouts, and i like watching as an expert. carry this over to football. i listen to announcers say the same kind of things, ascribe mental toughness as the reason that everything happens, insist that we learned who the best team was based on who won, and all that stuff. and i know they're wrong. i no longer feel like i can be a couch expert like i felt six years ago when i watched the eagles go to the super bowl. i don't think i know better, something almost every fan thinks sometimes, because i know i don't. i've learned that the people making decisions-- GMs, coaches, etc.-- these are experts, and i know that they are surely wrong sometimes but i don't trust announcers, beat writers, couch experts, anyone, to tell me.

so i find myself caring less about football. last night's game would have left me with a great feeling all day today, and while i enjoyed it-- particularly since i am surrounded by redskins fans-- i don't feel like i know much other than i should defer to andy reid on personnel issues, and that he is almost always right. i'd read football outsiders, but i would need a primer on what i was reading. so i'm in football fanhood limbo. i know i love watching football, but i don't like it as much because i feel like i don't understand what i'm watching with the understanding i want to, and i don't feel like i have enough information to know what i'm watching.

To: Everyone

From: Jared

Subject: RE: the state of my football fanhood

You're saying you don't like it as much because you don't have as much information. I can understand that. However, you're operating under the assumption that the information you're looking for EXISTS, when I would argue that in most cases it doesn't. I love watching football, and while I like to think that I understand the game pretty well, and I understand the players pretty well, I'm also comfortable with a few key facts.

1 - So much of the game is randomly determined, I would argue more so than any other major sport

2 - The information we would need to understand the game at a more sophisticated level is not available, so we take what we can get and use what we have

Unless you become comfortable with that level of ignorance, and the lack of objective measures, you may continue to have a tough time enjoying the game.

There are a few problems with trying to understand the game of football on a level akin to what we can in baseball. I think there are three major roadblocks...

1 - The 'Why Didn't He Block that Guy!' Problem

2 - The 'He Won't Show Up on the Box Score But He's Got Tons of Guts' Problem

3 - The 'Thanks for Falling for the Play Action Camera Guy' Problem

Problem 1 - This is the most fundamental obstacle to understanding what's going on in the game. To me, it's a problem of determining intent. If we know what a player's intent on a particular play is, we can in many cases objectively measure whether or not they have succeeded. In a game like basketball, it's often very easy because each player has generally the same responsibilities and those responsibilities are almost always connected to points. A player shoots to score, passes to assist. We can see what % of shots they make. No one shoots with the intent of missing. In a game like baseball, it's also fairly straightforward. A batter looks to get a hit or a walk. A runner tries to steal. A pitcher tries to get an out. Again, all the roles in hitting and pitching are consistent. They are the same for everyone (and I'm simplifying obviously. Basketball defense is tough to measure, baseball pitchers might look to induce double plays, pitch around guys, hitters might sacrifice, etc.)

However, in the game of football, we have two major elements that blow determining intent out of the water. The first is a matter of complexity. There are 22 starters on offense and defense, most of which are in very different roles, QB, WR, OT, etc. Each of these specialists is responsible for completely different tasks, and those tasks can be completely different from play to play. So even if we knew what everyone was trying to do, all the time, you'd still be left to think about all those permutations yourself and try to analyze it as you watched. That is a recipe for a trip to Crazytown.

But it doesn't stop there, it gets worse! In other sports we have an advantage in determining intent. We can guess with good accuracy what the player is trying to do! A hockey player wants to shoot the puck in the goal every time he takes a shot. In the simplest example, what does a 100M runner want to do? He wants to run as fast as possible (unless he's in a slowest man wins kind of race in some sort of opposite day scenario)

I would dare anyone to watch one NFL game, even one drive, maybe even one play, and tell me exactly what each player was SUPPOSED to be doing, what they were TRYING to do. It's just about impossible. Each play is designed for the eleven men to be coordinated in completing a variety of tasks. What kind of blocking assignments are the offensive linemen picking up? Is the RB supposed to chip the defensive end? And then what route is he assigned to run? Is the slot WR running a square out or a square in? Unless we know what these guys are supposed to do, we have no freaking clue of whether they're actually doing it!

If this were a Madden world, we'd be all set. Because we'd know each play called on offense and defense, we'd have a record of what everyone was supposed to do. But we don't live in a Madden world (to the dismay of John Madden, who would probably be some sort of Emperor and travel in an even bigger bus)

Some times you'll see an offensive lineman standing around, not blocking anybody, and the QB gets sacked. And you'll scream, 'why didn't you block that guy!?!' Maybe the lineman blew his assignment, but it's also possible that someone else blew theirs.

Problem 2 - This is a large part of what you describe in your email. We know what writers and announcers tell us, and it seems logical. So and so is a great leader. That guy is focused on his contract. Etc. You also describe why they say such things, and that's because they have to! The NFL is the most popular sport in the country, and demands round the clock coverage. In a world with a lack of objective measurement (in part because of problem #1), what are we left with? We have to say something. So we get the cliches. (As an aside, I almost never trust anything any TV or national football guy says. They simply have too much to focus on, following all the teams at once. You can't possibly know a lot of valuable insight regarding each team. An exception to that would be someone who watches a sh*t-ton of game film and illustrates it for you. Ron Jaworski does that a lot and has a show that is on ESPN at some god-awful early hour where they only do that. I love that show, most of the rest is just noise to me)

If we had objective statistics, we might make some headway. But we're not there yet, and I'm not sure how close we can even get due to problem #1. The stats that are thrown around today are, to put it diplomatically, less than ideal. By now most people are comfortable with the idea that Rushing Yards isn't the best measure of a running back, because if I were playing for the Eagles, and I received 10,000 handoffs, I could potentially gain 1000 yards. Of course, I'd also be dead, but you get the point. Yards per carry is a marginally better indicator, but even that can't distill it perfectly, because a running back is hugely dependent on his teammates and their ability to execute. It's also very dependent on the situation the running back finds himself in. Example: running back A gains 12 yards on a draw play, and running back B gains 3 yards. Who's better? In identical situations, running back A. But if running back A gained his 12 yards on a 3rd down and 26 yards to go, while running back B gained 3 yards on a 3rd and 2, it's a very different answer. Oh yeah, and that's for a running back, who at least gets his outcome measured. Good luck on an offensive lineman or a safety!

This is in large part what Football Outsiders is trying to address with their DVOA stat. It tries to evaluate each player's actions given the situation faced, with adjustments for things like opposing defense. I know it's not perfect, and they'd tell you the same thing, but it's an attempt to isolate the player by himself, because we just don't have good stats to measure them.

There are some others out there trying other things. There's KC Joyner for ESPN, but I'm even less convinced by his work because I looked through a sample of his book some years ago and saw lots of tables on things like, WR Success Catching Out Patterns, where they were ordered by percentages but it was often a sample size of 4. There's another one who's name escapes me, but they have someone watch and grade each player on each play. Don't even get me started on the red flags there. But the work is advancing.

However, until it makes a ton of progress, we're still going to hear about guts because we don't know enough about anything else, and we have to say something.

Problem #3 - The last major problem on my list, is simply one of data availability. In football, well, there really isn't any. These guys are data hoarders, laughing maniacally in their houses amidst piles and piles of game film that none of us will ever see. Or maybe they're just not as open minded, I prefer the former.

If any of us really wanted to understand the game, and understand the strengths and weaknesses of players, we would watch game film and we'd watch a lot of it. We'd look at the coaches cameras, and we'd play it back over and over and over. Then you'd be able to see specific plays where individuals messed up, where someone may have made the wrong read, where someone didn't get a good block.

But we don't have that, not by a long shot. We only get to watch the game on TV, from the network perspective. Hell, our view for that isn't even good. We never have any idea what the safeties are up to when the ball is snapped. Brian Dawkins could have been starting each play all these years from the crane kick position. We'd never know without the game film, and no one has the game film outside of a) coaching staffs, and b) NFL Films I think. I have no clue why they keep it so tight. Coaches/GMs might not want it publicly available I suppose.

The NFL also doesn't put out a ton of great data on the plays of each game. They have a game book for each one, but it leaves a lot to be desired. As someone who helped Football Outsiders with gathering their play-by-data, I'm more than familiar with that mess. It all adds up to a world in which getting more information is very very difficult.

So we live in a world where we're not sure exactly what the players are supposed to be doing, we don't have a great way to measure what they do do, and we can't get access to the information we'd need to confirm what we even thought they did.

It means we're really kept from performing deep analysis on the game. But that doesn't mean you can't understand it better. I'd start by not paying attention to what the announcers say, especially if it's Joe Buck (not because he's particularly wrong, just because he's a jerk). That's job #1. Other things that I try and do is look at the plays objectively, don't just watch the skill players, and mentally discount situations where luck is pretty freaking obvious.

It's still a great sport to watch, and don't feel concerned that you don't understand it like you want to. You need to understand that no one else does either, and you're view is probably as good as anyone not on the sidelines.

To: Everyone

From: Matt

Subject: RE: the state of my football fanhood

Alright, Jared, I hear what you’re saying and I agree. There probably is not very good data out there and football, so conclusions are hard to come by. I think that maybe I gave the impression that my issue is that I want to have a better relative understanding of what went on. That’s not the issue. I want to have an objective understanding of what happened enough that I can infer something about what happened on the play I just watched other than “Michael Vick through the ball far and DeSean Jackson caught the ball.”

Like, I want a crib notes of Moneyball for football. Like, I’m pretty sure that going for it on 4th down is probably a good idea all the time at midfield and punting from the opponents’ 38 is bad. I like that. I can work with that. You told me once that kickers are not consistently good at kicking short field goals or maybe long field goals, which one? I can work with that fact whichever way it goes.
I want basic things to know about football, whatever can be inferred. Enough that I know something about what types of positions are overrated and underrated. Who was the most underrated player in the victory? Can we say something about why yesterday’s game went the way it went?

On the luck issue, while the 16 game season makes games inherently dependent on luck to win enough games, the spread of talent in football in terms of how well teams can beat other teams is very big in football. If you figure out the standard deviation of football win percentages net of the natural standard deviation that you’d get from luck, and then do the same for baseball, there’s still like twice as much variance in team skill level in football than baseball on a per game basis. Which makes sense because you’d never see a baseball team given a 90% chance to win a game on pinnaclesports.

So really I just want some understanding of what I’m seeing. If I can’t trust the announcers, fine, I expected that. But I’d like some sense of what types of things I can look for about which I can have some knowledge. Is QB TD% a stupid stat? I’ve always thought INT% was more important even if QB rating treats them equally. Is Completion% important once you adjust for Yards/Attempt instead of Yards/Completion? Things like this could help immensely. Even knowing why certain stats are stupid would help.

To: Everyone

From: Jared

Subject: RE: the state of my football fanhood


Ah, well in that case, I would say that reading some of the general Football Outsiders stuff can be helpful. However, I'm not convinced of all their conclusions. Some things, coaches are way too conservative on fourth down, a kicker's accuracy beyond 45 yards is not consistent year to year, make a lot of sense to me. Others, like their QB theory that college completion percentage and number of collegiate games started predict success, or that 3rd down conversion % is not consistently repeatable, I'm not fully on board yet.

But with that said, a lot of what they say is definitely interesting and it's trying to make sense of the game in a logical and straightforward way.

It's hard to just give a whole bunch of things to know. But I'd bet a lot of them are things you probably already think.

Example: Establishing the run is a made up thing that announcers say. Announcers typically say something like, 'when this team runs the ball more than 30 times, they are 12-0.' Well of course they are. Any team with a lead in the 4th quarter is going to disproportionately run the ball to keep the clock running. The announcers look and see that lots of runs and a winning record are correlated. But they assume the runs lead to the victory.

Now, if the announcers talked specifically about how using those running plays are going towards setting up play action, that makes a little more sense and is something I might be inclined to believe.

In terms of specific stats, there are some that I like, and some that I don't really care about. But remember, this is all my opinion, and not necessarily right

RB - I like yards per carry, which seems pretty obvious. I think that's the best conventional stat. Football Outsiders has it's DVOA stat, which I think is better because it takes into account the result relative to the situation and other factors.

QB - I'm not a huge fan of most of the stats you mentioned. Things like completion percentage can be messed up by running a west coast offense vs. a run and gun offense. Yards per attempt is a pretty good one.

WR - Yards per catch maybe? I like FO's catch rate, which they release every year and while it's based on someone manually charting games, it seems pretty accurate to me. As an example, Jason Avant had the best hands in the league last year. If you watched all the Eagles games closely, you'd be like, 'well that's obvious, even if he does suck because he's from Michigan'

Defense is much much harder because there aren't a lot of good stats. Tackles are good? I suppose their better then not tackling the guy, but if your safeties have tons of tackles, that doesn't mean they're good, it might mean your defense sucks. Another example, interceptions. Yes, they're very very good. However, if you're Revis, maybe you're so good at covering your man that they never throw your way. Or maybe you get a lot of INTs because you gamble a lot, and you also give up a ton of big plays. I'd love to see more on defensive linemen, in particular, things like hurries and pressures/knockdowns. When you force a QB to throw, that is usually a good thing and can lead to other good things (not converting 3rd downs, interceptions). But that's not a very well kept or well publicized statistic.

In terms of underrated or overrated players. A simple rule would be that the importance of the offensive line is systematically underrated, and the defensive line is also underrated (but not nearly as much as the OL). The offensive line is the only thing that allows the offense to succeed, but it's very hard to see exactly what they're doing or understand what they're supposed to do. They also don't catch the ball, run the ball, or score touchdowns, so the announcers very rarely talk about them. If a running back is tackled behind the line of scrimmage, do we really think it's the running back's fault? On Vick's last touchdown throw last night, he did a great job of escaping the one guy who got to him, but there's a reason the other guys didn't and Vick could keep scanning the field to find Avant in the end zone. Its similar for the defensive line, particularly the tackles who don't get many sacks. People see Patterson and Bunkley without sacks and they say, 'these guys suck!,' but they don't suck, they just aren't pass rushers. But we don't have stats to measure them, and the announcers rarely care, so no one pays attention, but the line play is one of the most important things in the game and something a lot of people don't look at.

So yeah, watch the lines.

To: Everyone

From: Matt

Subject: RE: the state of my football fanhood
a lot of that stuff is definitely logical and stuff i could try to incorporate into my watching. i like the 4th down thing, the long range accuracy thing is interesting though i tend to think that there is selection bias where kickers are asked to try harder kicks in the wind and stuff based on how good they are. the college games started issue is really just a proxy for how good people thought he was when he was younger. it's not useful. it's like saying that guys with bigger signing bonuses in baseball do better. yes, but not because of that. i guess that's useful for proxies but it doesn't get at what i want to know about watching games.

the "establishing the run" thing is something i've always thought was mostly bull. i think that there is an issue with whether you keep 4-8 guys in the box, and so i'm sure that you need to be a game theorist to make the defense have to prepare for both run and pass on each place, but the number of times run is bullshit obviously. also, i've always noticed that most qb's who throw for 300 yards lost.

does DVOA value individual players? do they describe their methodology anywhere? i can never find anything on that site and i'm always left to throw up my hands and give up. it's actually designed worse than baseball prospectus' website which i thought was impossible.

the catch rate thing for WRs is interesting but...i dunno...like, how do they do it? do i just trust their numbers? i don't want to just trust numbers blindly. i'd like some sense of where they come from. i feel like there isn't much of that.

defensive stats all seem bad. that doesn't help me get into things either.

is yards/carry really the best stat for RBs? i'm always inclined to think it's based on how many really long runs you get. that's certainly useful but isn't the point of runs usually to either get a first down or to get closer to the first down marker? like, if you have a north/south back who just goes up the middle all the time, and he gets 3 yards per rush but never fumbles and rarely loses yards, i'm inclined to think that might be better than a back who averages 4 yards per rush, but that's because his rushes are divided between 18 yard rushes, no gain rushes, and 6 yard losses. like, 2/3 of the time he's unsuccessful.

is there a way to measure line play? some kind of metric of some kind?

what do football quantitative researchers know that football scouts don't know? that's my question.

i really would like to learn enough information such that i could watch a game and have some knowledge about how smart a coach's decision was or who was to credit/blame for big plays. like, who was better on the vick/jackson td to start the game? vick or jackson? or the o-line?


To: Everyone

From: Jared

Subject: RE: the state of my football fanhood

Lots of questions, some with answers, some without

- Yes Matt, they do DVOA's by player for the skill positions, they have some other metrics as well, and while I don't know how precise they are, I generally agree with their direction.

- You bring up a fantastic point about rushers, in that there's a value in consistency vs. the value of a boom or bust running back who can gain 20 yards but also lose 3. FO has talked a lot about this, and they obviously have more advanced metrics than yards per carry, I was just citing that one as the one that's on TV that I actually look at (as to say, the best of what's readily available). They (FO) have something called success rate, which gets at the point you're focused on but I don't know too much about the methodology off the top of my head

The basic question you have is what do researchers know that football scouts don't, and I don't know that there's a great answer for you. To figure out where the advantages/knowledge gaps would exist, we need to think about where data is available and where it's not. I think it's fairly safe to say that quant analysts looking at things that can be objectively measured can provide insights that go against conventional wisdom (i.e. what the 'football scouts' believe). Things like the kicking accuracy not being correlated from year to year, and that distance on kickoffs is far more consistent and a better measure of leg strength. There are stats on that, and through basic analysis you can draw a conclusion or at least a hypothesis. I think the same thing is doable on general tendencies (i.e., when to go for it on fourth), because they have complied historical data that they can look at and draw conclusions which people on coaching staffs might not have (or maybe they do, we don't really know for sure).

In terms of player analysis, I don't believe they are fundamentally better than the regular football guys. I think their stats work to show which skill players are actually delivering value, and I think those are good, but I'm not convinced they're better than the average football guy (although they may be more informed than the below-average football guy, i.e., the raiders or the redskins). But we should keep in mind the advantages football guys have over the stat guys, those that I mentioned earlier. The football guys have all the film and know much more about what plays were called and the players' intent. They also spend days and weeks and months analyzing it, which all of us can't do. Now granted, they don't build objective pieces of information into data sets for real thorough analysis (again, maybe some do, but we don't know), so my guess is a lot of it becomes the basis for 'gut' decisions, and that might make it less accurate then if you had a supercomputer doing it, but it's still an area where they possess an informational advantage.

I think a great example of where both groups collectively have no idea is in college player evaluation. NFL draft picks become busts regularly, and it's clear that teams haven't figured it out (and let's remove coaching and opportunity from the equation to simplify. Those things are obviously huge deals in player development, but let's just say that even if those were constant, there would still be lots of busts). Teams try to figure out who won't be good, and the quant analysts are trying too, but no one has convinced me that they have a good model yet. I don't know what the pro teams are up to obviously, but I'm sure they're trying to be data-driven (the 49ers have a former management consultant as one of their top guys, as an example). FO has done work there too, trying to find measures of college players that correlate with success in the pros. But that stuff is in its infancy and I don't know that I trust what's out there yet. The same problems that exist for analyzing professional football exist for college, only with 100+ teams instead of 32.

As an example of some stuff that I like reading (and one specifically related to your question about vick/jackson), I'd recommend this article from FO today. It doesn't have any stats, it's just taking certain plays and really focusing on them to see exactly what happened. You definitely won't hear this type of stuff on the broadcast, and I haven't seen it in any kind of reporting.

http://footballoutsiders.com/cover-3/2010/cover-2-career-days

It actually diagrams some plays and gets into detail. Now it's a sampling, sure, but when you read it you start to get a feel for the types of things that can be important and/or should be looked at

To: Everyone

From: Renato

Subject: RE: the state of my football fanhood


I'm really enjoying this thread. It should be published.

Local Sports Content

News came out the other day that reinforces earlier stuff I've been saying about sports media content.

Comcast Joins Sporting News

Comcast Sports Group has entered into an electronic publishing deal with the Sporting News to create regional editions of Sporting News Today in the markets where Comcast has regional sports networks.

The first edition launched this week in Philadelphia.

I'm still of the mind that local sports broadcasting rights will be one of the only genres of television programming that will maintain its value to traditional advertising. The continued acceptance of, in order, DVRs, on-demand, and web-based alternatives like Hulu pretty much ensure that timeboxed viewing of shows will generally go away.

However, people still want to watch sports, and they still want to do it live as it happens.

So that content will still bring eyeballs, and for years TV players have been fighting for position to establish themselves as the dominant player in major regions.

Fox has regional networks, Comcast does as well, and it would be 100% non-shocking to see ESPN rollout regional networks as well (lord knows they have enough other ones). ESPN's already creating online portals for regional sports news, and while I think Comcast has also done it, I don't know anyone who goes there for their sports news. But this article indicates a more concerted effort to build an online complement to their TV presence.

People will keep watching sports on TV, and they'll keep watching the ads because it's an event that people want to see live as it happens (event viewing as opposed to program viewing). So TV players will keep trying to build their way into those positions to control the viewing.

But I wonder if it's all for nothing at the end of the day. There's no question that for smaller sports leagues, selling your TV rights to an established entity so that they can run production makes a ton of sense. You probably won't see an MLS network in the near future.

But for a business like the NFL, which is obsessively watched more than any other sport in the U.S., eventually they're probably going to keep it all to themselves. Some local teams already do this, like the Yankees and the Maple Leafs in Toronto (because yes, some people care about hockey!). The NFL is gradually working its way there, in establishing its own network, fighting for broad distribution, and slowly moving its games there (Thursday night football only on NFL Network!)

Pretty soon, they'll run the whole thing. But for now, the media companies will still fight to bring it to us in the interim.

Chrysler Commercial - - > A minivan for nerdy kids

I don't get to see much TV, and when I do, I tend to skip the commercials. However, if I have it on at the gym while I'm on a treadmill, there's really no way to escape them regardless of how fast I run.

On a number of occasions I've seen this commercial for a Chrysler minivan. And it's a commercial that's been really really bothering me. Not because it's deceiving, and not because it's way too loud or features some other kind of crazy concepts conjured up in a Don Draper vision.

This commercial has been edited and re-dubbed in an illogical way. It makes absolutely no sense, presents us with images that cannot connect with the story being told, and I feel it implies that everyone watching it is too dumb to notice.

Here's the offending commercial, or at least part of it. I couldn't find a full clip on YouTube, maybe because Chrysler feels so guilty about thinking we're stupid that they're trying to erase it from existence:



Anyway, the beginning of the commercial has a voice over from a kid saying, 'Hey Parker, I'll race you home' Or something to that effect.

However, it's fairly obvious from the video, that kid isn't racing anybody, he's trying to get the hell away from those punks who are going to beat him up!

A more complete version of the ad shows the kid hiding behind a tree, and when he does actually throw himself into the trunk of the conveniently placed van, it's clear he's just happy to be alive.

At no point, was it a race among friends. That kid is a nerd. And those kids wanted to beat him up (which implies they're jocks, the nerds' natural enemy)

Anyway. I'm guessing someone at Chrysler with a beat up kid saw the ad and was like, 'Um, guys, this is a terrible message. Unless we're selling to the parents of kids who get their ass kicked every day'

So they changed it, but they changed it for about five cents by re-dubbing the first line and keeping the rest of the commercial as is.