Monday, December 1, 2014

Theoretical Decision Model for NFL Coaches Challenges

One of the great things about the NFL is that every game features tons of small coach and player decisions -- but to be more precise -- one of the great things about the NFL is the ability for armchair QBs like me to second guess any and all of those decisions!

Which is why I've had such a hard time getting the Eagles week 8 loss to the Arizona Cardinals out of my head. I know the Eagles just finished a great Thanksgiving pasting of their division rival - but that Cardinals game keeps coming back to me. Specifically, the Eagles decision not to challenge a call late in the fourth quarter.

If you'll recall, the score was tied late in the fourth quarter (17-17), and the Eagles had the ball deep in Cardinals territory. On 2nd down with 4 yards to go from the Cardinals five yard line, Eagles running back Chris Polk ran the ball for a three yard gain - but one that looked like it may have been spotted incorrectly by the officials.

In my mind - a challenge was automatic. But the Eagles disagreed - they kicked a field goal, gave up a touchdown on the next Cardinals possession, and lost the game.

Afterwards, what bothered me, besides the fact that the Eagles lost - was the attitude towards challenging the play. Here was Chip Kelly's response when he was asked about it:

"“I don’t think the league rate at overturning spots, in terms of where his knee was down—most of the time the err, I think percentage wise in the league, if you study the percentage of it there you err on the previous call on the field because it’s very difficult to tell on the spot unless it’s kind of a clear-cut open field kind of thing,” Kelly said."

This was what I took issue with. The Eagles didn't challenge because most spot challenges aren't overturned.

Now, that may be true (another article I saw reported that only about one-third of spot challenges have been overturned this year) - but that reasoning has one giant flaw - and that's situational leverage.

What I mean by that is - what is the relative upside to winning the challenge, given the game situation. Winning a spot challenge to get an extra two yards when you're up by 35 points would be meaningless, but winning a challenge to put you in FG range with 2:30 left in the fourth quarter in a tie game could be huge.

Treating those potential challenges, because they're both spots of the football, as the same leaves out a massive part of the equation.

So, I started to think, is there some way we can actually calculate that piece of the puzzle? And could that potentially shed light on when teams should challenge calls???

I think the answer is unequivocally yes, and yes.

Logic on Challenge Win Probability

The reason I started thinking along these lines at all is that I was comparing the Eagles situation, or any NFL team for that matter, to a poker player making the decision to call a final bet. If you're playing a game of poker and deciding whether to call - the decision isn't just about whether you think you have the best hand, it's also about how much you'd stand to win. Calling a $50 bet with a pot of $500 is a LOT more attractive than calling a $500 bet with a pot of $50.

That's what I mean when I talk about leverage. Now in poker, players will use the concept of pot-odds to help them make their decision on whether or not to call. Let's take that first situation I just described. A $50 bet to win a pot of $500. In that situation, winning the hand will pay out at 10:1, so if you thought you had a 50/50 shot at having the best hand, would you call the bet?

Of course you would. If you thought you'd win 50% of the the time a 10:1 payout will, over time, be very profitable.

Now in the other situation - a $50 bet to win $500 - well now you're getting a 1:10 payout. Now if you have a 50/50 shot of having the best hand - would you call this bet? Of course not - because the upside potential is horrible!

Poker players will use these pot odds to help determine the win probability required for a bet to make money. In my first scenario, if a poker player guesses he has a 25% chance of having the best hand - he should still call the bet. That calculation makes the decision to call much clearer.

To me, challenging a call in the NFL is the exact same thing. NFL coaches should not only be considering whether or not they'll be winning the challenge in absolute terms, but they must also consider the upside of winning the challenge itself. In fact, if you work backwards, you can calculate the probability of winning a challenge you would need for the challenge to be worthwhile - just like poker players impute the win probability they need to call a bet. Only then should you be referring to historical overturn rates or anything else to see if it's a smart bet.

So...if you're on board with the logic, let's put it into practice:

Estimating Challenge Win Probability Required

OK - so for any of this to make sense, we need a basic metric to get at odds of winning the football game. As a coach, that's the metric we're trying to maximize - and we need to simplify all the various game situation components we have (time left, down and distance, score differential, etc.)

Fortunately, that already exists in the form of win probability estimates. Brian Burke of Advanced Football Analytics has his model, Pro Football Reference has one as well, there may be others - but I'm going to use Burke's for these examples because he has a calculator that let's you play with different situational variables like the examples I'm going to use.

So if we wanted a simple equation to help us evaluate whether or not to challenge the call on the field, what would that look like?

(Odds of Winning a Challenge)*(Benefit from Winning a Challenge) - (Odds of Losing a Challenge)*(Cost of a Challenge)

In this equation - you would challenge whenever the result is greater than zero, and you would not challenge when the result is less than zero.

To determine what odds of winning we would need to want to throw the challenge flag, we just have to set our equation equal to zero and solve for the missing variable. So let's dig into each component in a bit more detail:

1. The Odds of Winning the Challenge: This is what we want to solve for, the same metric the poker players use. It's going to be in percentage terms from 0% to 100%.

2. The Odds of Losing the Challenge: This is just the companion to our odds of winning the challenge. Together they have to add up to 100%. If you have a 75% chance of winning the challenge, well, you have a 25% chance of losing. Pretty straightforward. If the odds of winning the challenge = X, then this variable is just (1-X)

3. Benefit from Winning a Challenge: OK, this is the key component to the equation. I mentioned earlier we would use win probability estimates, and that's what we're going to use to estimate our 'benefit' from winning a challenge. If you're deciding to challenge the call on the field, you're effectively looking to change the current game situation. In this context, game situation is a combination of field position, down and distance, score differential, possession, and time remaining. As I mentioned earlier, win probability models can determine the probability of winning for any individual game situation - that is, any unique combination of those factors.

For example: if your team is starting the second half with the ball in a first and ten on your own 20 and a 3 touchdown lead - your win probability is estimated at 94%

If your team has the same field position, but now is down by 6 points and with only five minutes left in the 4th quarter - your win probability is 21%

Make sense? OK.

Now in the case of a challenge, a coach is looking to overturn a ruling that will result in a new improved game situation. And in just about every challenge I've ever seen, we know exactly what the old and the new situations would be. This variable in our equation then, is just WP^Overturn - WP^Upheld. That's our positive benefit from winning the challenge.

Here's an extreme example to illustrate a high leverage situation:

Let's say you're the Eagles, only its 1991 and you have the greatest video game QB of all time, QB Eagles - because this is Tecmo Super Bowl.

Now let's say it's 3rd down and ten from your own 20 - because you've thrown two long bombs to Fred Barnett that haven't worked yet. And let's say the score is tied.

So you call the long bomb again, and true to Tecmo form, immediately take a 30 step drop for QB Eagles into the back of the endzone, uncorking a beautiful rainbow that travels 108 yards in the air towards Barnett. He does that thing where the received jumps up and they show the cut-scene of him catching it, and he lands at the two yard line.

But the ref rules him out-of-bounds...says his foot touched down out.

Now let's say Tecmo has added the functionality to challenge the ruling on the field - what would the benefit be??? (I only now realize this example could probably be done with the latest version of Madden - and would make a lot more sense to anyone under 30 - ugh)

So if we use Brian Burke's calculator, here are is the first scenario - the one where if you win the challenge:

Score Differential: 0
Time Left: 2:00 Second Quarter (made up)
Field Position: Opposition 2 yard line
Down: First
Distance: 2 YTG

Challenge Victory Win Probability - 70% (In part because you're expected to get 5.8 points off this drive - even though the computer has NO IDEA how awesome QB Eagles is)

Now - what's the scenario if you LOSE the challenge

Score Differential: 0
Time Left: 2:00 Second Quarter (made up)
Field Position: Own 20 yard line
Down: Fourth (an incompletion on third makes it fourth down - you should totally go for it, FWIW)
Distance: 10 YTG

Challenge Loss Win Probability - 45%

So - if you won that challenge, not only would you get some good yardage to pad QB Eagles stats, but you'd also go from a 45% win probability to a 70% probability.

The benefit then - would be 70-45, a 25% improvement.

This is the element, the payoff, that we need.

4. The Cost of a Challenge: Slightly more tricky, but necessary to understand the cost of our challenge. If a challenge didn't cost anything - there'd be only upside to throwing the flag and we'd have red flags galore. The only people who would be happy would be Mike Carey and the other officiating experts - the rest of us would be super pissed. So there's a cost to being wrong on a challenge in the form of a timeout. Note - there's no cost if you win a challenge - so we don't need to account for that in the first half of our equation.

But the loss of a timeout is a bit harder to quantify. It's obviously valuable - but we'd need some estimate of it in terms of win probability for this math to really hold up. Fortunately, Brian Burke and the Sports Collective have done independent work on exactly that question (I don't have the links anymore but if you're really interested in it, just google it.

Now - their research shows a timeout to be worth between 3% and 5% win probability. So I split the difference and assumed 4%. Now - there are a whole bunch of non-trivial caveats as the value of timeouts would technically be very dynamic. It would vary depending on the game situation as well as how many timeouts remain in general (the final timeout is more valuable than the first). But that's a question for another time. We're going to use 4% - because I don't have the data/time to churn through that question on my own.

Putting it all together

Now we have all the pieces that we need to put our equation together. So let's take our Tecmo example...I used that as an example of a very high payoff....that means you would be willing to challenge even if you thought the odds of it getting overturned were smaller than normal. Instinctively that makes sense, but what does the math say?

(Odds of Winning a Challenge)*(Benefit from Winning a Challenge) - (Odds of Losing a Challenge)*(Cost of a Challenge)

Odds of Winning a Challenge = X% (remember we're solving for this)

Benefit from Winning a Challenge = (.70-.45) = .25 WP

Odds of Losing a Challenge = (1-X)%

Cost of a Challenge = .04 WP

(X)(.25) - (1-X)(.04) = 0

X, or the probability of winning the challenge required for it to be worthwhile = 14%

And that makes a lot of sense! You've got tremendous upside to winning the challenge, so even if you think there's only a 20% chance the ref will overturn it, it's still worth it to try, because the payoff is so great.

Now - what if the payoff wouldn't be great? What does our equation say in that case...let's take another example.

Let's say the score is tied 0-0 in the third quarter with 12 minutes to go, the Eagles have the ball on their own 20, second down and one yard to go. They run a WR screen where the pass appears to skim the ground as Jeremy Maclin grabs it before getting tackled with a one yard gain. It's ruled incomplete, but maybe we could challenge it and get it overturned (maybe the coach has Maclin in a fantasy PPR league)

So - what does the WP calculator say about these two scenarios, the first where the Eagles win the challenge (1st and 10 from their 21), the second where they don't win (3rd and 1 from their 20)

Well - the benefit in this case is 9% win probability (Eagles go from 50% win probability to 59% with the improved field position).

So if we put that in the equation, what does that tell us?

(X)(.09) - (1-X)(.04) = 0

X = 80%

So in this case, the payoff is so much lower, that you need to believe there's at least an 80% likelihood that the ref will overturn the ruling on the field.


The purpose of this thought exercise wasn't to create a bunch of weird examples - but what I wanted to do is illustrate a mental approach to challenging calls in NFL games. Most coaches probably pursue an approach that's intuitively aligned with this thinking if not calculated out to the same degree. Understanding the likelihood of winning the challenge is good - but it needs to be taken within the broader context of how that change in game situation will impact likelihood to win.

If it's a big play, the less confident you need to be in the replay to take a shot with a challenge. This is even more important in cases where there might not be enough time for coaches to see a replay.

Ideally - it would be great to create a simple widget where anyone could input the before and after circumstances of the challenge and determine the probability you would need to be willing to challenge. That would be fun to play with - and you could even create some simple diagram or framework that outlines when a challenge requires a very high confidence and when a challenge doesn't require much at all.

That's something I'd do if I had a win probability model of my own - but I don't, so you'll have to live with the theory. But in my mind, this is the right way to think about potentially challenging the ruling on the field.

Oh - and by the way, that Eagles-Cardinals challenge that got me so steamed in the first place. The model suggests I was wrong (although there's a big caveat).

Because the play happened so late, in a tie game, with the Eagles only a couple yards from the end zone, the win probability for a 1st down vs. a third down doesn't change things all that much (.82 vs. .81). And because the win probability difference is so small, there's not nearly enough upside.

However - I will say that this points out at least one of the shortcomings in this approach. There's no probability model out there that will account for time outs remaining on either team - and I think that would've played a role in the 'true' win probability. The Eagles ability to run the clock with a 1st down would've forced Arizona to use their timeouts or left them with little time left to score. So I think the win probability impact from winning that challenge would actually be a bit more.

But why should we let reality get in the way of some pretty cool theory!

No comments: