How much do the playoffs matter? Part II
Everything interesting is selection bias - Adam Harstad
The all-time postseason leader in Adjusted Net Yards per Attempt (ANY/A) is Kurt Warner, who finished his career with 7.95 ANY/A in 13 playoff starts. His regular season ANY/A of 6.71 was notably lower which gives the impression that he stepped his game up when it mattered most. Well, not so fast.
Warner played in the postseason only five times in his career (the 2003 Rams also made the playoffs with Marc Bulger after Warner was benched early in the season). Not coincidentally, those five playoff trips followed the five best regular seasons of Warner’s career. His unweighted regular season ANY/A in those years was a very strong 7.43. Meanwhile, his best ANY/A is non-playoff years was only 6.48.
What we have here is a classic case of selection bias. Comparing his career regular season and postseason ANY/A, Warner improved in the tournament by +1.24 ANY/A. However, if we compare his regular and postseason efficiency in the five seasons he participated in the playoffs, his ANY/A only jumps +0.47 in the biggest games. Given the small sample of just 13 playoff games, we’re well within the range of his apparent improvement being nothing more than random chance.
This instance of selection bias makes intuitive sense. If Warner (plus his supporting cast and coaches) was performing at a certain level during his best regular seasons, it logically follows that he’d continue to perform at a similarly high level in the playoffs of those same seasons. As odd as it may sound, Warner’s playoff legacy was aided by him playing poorly enough to miss the postseason in his worst seasons. Had he somehow made the playoffs during his horrible 2002-04 stretch, it’s very likely that his shoddy play would’ve continued into those hypothetical playoff contests, and his career postseason ANY/A and win-loss record would look significantly worse than they do now.
But the case of Warner (and many others!) isn’t the only type of selection bias we see in the NFL postseason. It’s also the playoff format itself that can warp statistics and records. In a single elimination tournament, losing teams and players are kicked out and prevented from adding to their sample of work. And with that larger sample artificially removed from some players’ statistical resume, variance has no time to even out and we end up with a playoff sample that doesn’t accurately represent a player’s range of performance outcomes.
This type of selection bias is bad enough in a sport like tennis where the only variables on the court are the player and his opponent. But in a team sport like football where the QB is given all the credit/blame despite being at the mercy of dozens of other players, the ugly monster of bias can get really out of control. Imagine a QB is on a hot streak entering the postseason and plays great in the Wild Card game, only to lose because his defense couldn’t stop a nosebleed. Not only is this QB branded with a loss on his record that he didn’t deserve, but he’s denied the opportunity to stack more great performances in subsequent games. This happened to Peyton Manning several times, notably in 2007, 2008, and 2010.
The opposite scenario is also in play. Quarterbacks who play poorly in their first playoff game usually lose. But some fortune souls get bailed out by their teammates and are afforded the opportunity to play again the following week. Joe Burrow in 2022 is a perfect example - he was bad in the Wild Card game but Cincinnati still won due to a 98 yard fumble return TD. Then the following week Burrow played a legitimately great game in a Divisional Round win over Buffalo - a great game that most QB’s wouldn’t have had the chance to even play had they performed as poorly as Burrow did the week prior.
Of course I can’t write about playoff selection bias without mentioning the cartoonish career of Eli Manning. The younger Manning played 16 seasons in the NFL and only won playoff games in two of those seasons, yet he somehow retired with an 8-4 postseason record. That’s because Eli benefited from both types of selection bias. He and his teams were bad enough to miss the playoffs 10 different times; this almost certainly saved Eli from a number of bad playoff performances and losses being added to his record. Then he had two separate runs of being good and lucky enough to play the maximum four playoff games in a season, and these two miracle runs mostly drown out the ugly stuff on his postseason resume. Using playoff W-L record to judge quarterbacks is foolish and shortsighted, but if we must, then adding a loss for every missed playoff season would at least make the ledger a little more accurate (Eli would go from 8-4 to 8-14, big difference!).
Expect detailed postseason breakdowns of many star quarterbacks in the future, as there are countless misconceptions that are begging to be shone the light of truth.