I like football a lot. I like analytics a lot. Through some kind of not quite transitive property, I like football analytics a lot too.
But I'm so sick of seeing debate/criticism on Twitter of Pro Football Focus, who seem to constantly occupy a spot at the nexus of football analytics arguments.
What's weird is it's not the typical analytics arguments between guys yelling about 'guts' and guys yelling about data and which one is better. It seems a bit different, just pairing data-literate people against each other to argue over which stats mean something and which don't.
Pro Football Focus, from my vantage point, seems to provide a pretty valuable service.
They have a team of employees who watch the games and track all kinds of metrics. They focus on adding more detail and specificity on things that conventional stats on ESPN don't get it. That's a valuable service - and can be tremendously insightful.
They also publish player grades - which is the result of a detailed film review to score what each player did. The grades are the result of multiple reviewers, so it's not just one person, but the team collectively evaluates how all the players did, publishing results afterwards.
That's a less valuable service, doesn't mean it isn't insightful also, but it's less valuable than all the raw data. But since it takes a very complicated evaluation and boils it down to a simple positive or negative number, that's what people focus on.
These grades, as you can imagine, are often the source of Twitter feuds between PFF defenders and detractors. Those who say the grades are bullshit vs. those who say the numbers are valid.
I'm so sick of seeing these go back and forth on my timeline. Both sides are wrong.
The folks that hate on PFF argue that the grades are bullshit, and there are certainly cases where that's true. The fundamental flaw of their grading system is that, to be truly accurate, you need to know what a player was supposed to do, not just what they did. That means their grades are inherently subjective because they're guessing. In most cases, they're probably right, but not always. But with that said, are they probably pretty close to the right answer? Yeah. It's what we'd call 'directionally correct' in the consulting world. I'm sure any individual play could be off because you don't know a player's true assignment, but over the course of a game, a season, those mistakes should wash out. So if you're arguing over PFF's credibility on an individual grade, I'd say settle down, because on balance the ratings should be mostly right.
At the same time - PFF seems to get pretty damn smug about their system when someone criticizes it, and that's pretty stupid too.
This article notes the CEO of PFF remarking that criticism of player grading because of lack of visibility into assigned responsibility is lazy. I think that's wrong. It's not lazy to point out a methodological weakness. If you want to say the results are still valid on the whole - that's one thing - but let's not pretend your system is infallible.
I also hate that PFF constantly promotes the fact that 19 NFL teams subscribe to their services, because I think it leaves out a massive part of that equation. PFF uses the fact that NFL teams are subscribers to demonstrate the validity of their model and the value of their data. But the true indicator of your product's value isn't how many teams 'subscribe,' it's how much those teams pay for the subscription.
Anyone who has done business with a data service in any role knows that the service is always going to try and get you hooked on their platform with access at a low initial price. If I were running PFF, I would've tried to break into the league with an extremely low price to become the established platform for detailed game metrics - to become ubiquitous - and then focus on increasing the price/expanding to NCAA. It's not a complicated strategy.
So PFF brags about all the teams using their data. But unless we know how much they're paying, that evidence doesn't persuade me. However, I'm sure that it works on a lot of folks - and ultimately that's what you need to build a market standard platform.
I still think their data is probably pretty good and valuable for teams - even if there are some concerns around their grades. I wonder why a company like STATS, who pretty much owns the NBA's advanced analytics data source, hasn't grabbed them or launched a competitor. The barrier to entry seems pretty low (just find some football nuts who want to grade film - which STATS definitely already has).
What's more interesting to me isn't which platform becomes the standard for pro team offices, it's beyond that. Ultimately, sports wagering will be legal in the US. Whether it be restricted to Daily Fantasy or not, the walls will eventually come down. And at that point, the ability to offer proprietary data for sports analysis becomes much more valuable - because there will be much more real money riding on the outcomes. Bloomberg, Capital IQ, and other services like that are the analogs - and I'd have to think that the ultimate market for sports betting users (talking power users here, the ones who claim to win tons and tons in DFS), would be potentially significant.
I think that end game is where companies like PFF should be focused - not defending their grades on Twitter. After all, in that world, bets made based on their grades would ultimately speak for themselves and finally settle the question.
