A Ranking of NFL GMs, and Why Howie Roseman isn’t Bad at His Job

Howie Roseman, across the NFL, is considered to be one of the league’s top executives. Whether it be managing the salary cap or being a wizard with trades, Roseman is considered to be amongst the best in the league at his job. But a curious narrative has arisen around Howie’s tenure in Philadelphia: he is bad at drafting. Especially after this weekend’s past draft and the selection of WR Jalen Reagor and Backup QB Jalen Hurts. Is that true, or is this narrative simply a result of an incredibly passionate and critical Philadelphia fanbase? Based on a quantitative analysis, it appears that while Howie is not the best drafter in the league by any stretch, it would be disingenuous to say that he is bad at drafting in comparison to the rest of the league. While we’re at it, let’s put the GMs into tiers.

For the purposes of this article, I compared Howie to his direct peers: other current GMs who made at least 20 picks between 2010 and 2017. I stopped at 2017 because of the rule of thumb that draft classes in full cannot be properly evaluated until they have played 3 full years in the NFL. Using this framework, I analyzed Dave Gettleman, John Elway, David Caldwell, John Schneider, Les Snead, Jason Licht, Kevin Colbert, Marty Hurney, Mickey Loomis, Rick Spielman, Ryan Pace, Steve Keim, Thomas Dimitroff, Tom Telesco, Bill Belichick, and Duke Tobin.

Since the main contention about Howie Roseman’s ability to draft that he is bad at drafting players, and not that he misvalues different positions in respect to the modern NFL, we’ll go about trying to answer the simple question: does Howie usually pick good players in comparison to other GMs? But answering this question offers even more issues. How are we going to define a good player or pick? Certainly, it wouldn’t make much sense to apply the same standards of QBs drafted in the top 10 of an NFL draft to a player drafted at the end of the 7^th round.

To get around these issues, I used Pro Football Reference’s  Approximate Value (AV). AV is Pro Football Reference’s attempt to put a single number on the seasonal value at a given position (each position has a different formula), however it is still a limited metric. Directly from PFR’s explanation of the metric they state that a player with an AV of 14 didn’t necessarily have a better season than a player with an AV of 16, but to think of it as a metric similar to number of season as a starter or number of pro bowls. That is, it is more likely that a player with a higher AV is better than a player with a lower AV, and the bigger that difference is the more likely the higher AV player is better. It is a good indicator as to whether or not a player had a generally good season or not.

What about the expectation issue? To get around this issue I averaged the AV/year of all players drafted by round between 2010 and 2017. I then observed the difference between every player selected by the above GMs between 2010 and 2017 and got the average difference for their picks over this time span and ranked the GMs accordingly. An extra category was made for QBs drafted inside the top 10 picks as they have higher expectations than usual.

To give some landmarks, an AV of 8 roughly corresponds to a mid-level starter (Lamar Jackson’s rookie year), 5 roughly corresponds to a productive role player (Ziggy Ansah in 2016), 3-2 roughly corresponds to a backup, and 1 roughly corresponds to a player at the bottom of the roster that rarely sees the field.

Here are how the GMs performed in respect to these averages:

To understand the interpretation of this, take a 3^rd round selection from Tom Telesco as an example. On average, the AV/year of that player will be about 3.15, which is 0.216 above the average value for a player taken at that round. So, what does this say about the GMs observed and about Howie in particular?

This splits up the observed GMs into 4 clean tiers

Tier 1 – Ryan Pace, John Schneider, Dave Gettleman, Mickey Loomis, David Caldwell

These are the GMs who were best at just picking good players over the observed time span. Ryan Pace is only here because of drafting Eddie Jackson and Tarik Cohen in 2017 who greatly outperformed their draft positions. Considering his more recent drafts, he will likely move into tier 2. Mickey Loomis is here after having 2 historically strong drafts in 2016 and 2017, even though the drafts earlier in his career were not quite as great. David Caldwell is surprisingly in this tier, but this is because he only has one egregious pick which was Dante Fowler Jr in 2015. Even though Luke Joeckel is widely considered a bust, according to AV/year he was not a large bust as he still had some production for the Jaguars.

It is not surprising that John Schneider and Dave Gettleman would perform well in this kind of analysis as they simply do not draft busts very often. Their main criticisms are in how they value different positions, exemplified by the Seahawks recent pick of Jordyn Brooks and Gettleman drafting a running back #2 overall with an aging, declining QB. Brooks and Saquon Barkley could very well end up being great, and Saquon arguably already is, but that is not the core of the complaints towards Gettleman and Schneider. This just goes to prove that simply drafting good players is not the only requirement to be a good GM.

Tier 2 – Tom Telesco, Les Snead, Thomas Dimitroff, Bill Belichick

These are the GMs who are good, but not great, at picking good players over the observed time span. Tom Telesco and Thomas Dimitroff have 2 very good drafts (2011 and 2016 for Dimitroff and 2015 and 2017 for Telesco) and one very bad draft (2014 for Telesco and 2010 for Dimitroff). The remainder of their drafts are about equally good or bad. Les Snead and Bill Belichick are similar in this vein, where they only has one very good draft class, 2017 for Snead and 2016 for Belichick, and the remaining drafts were average, save Belichick’s 2017 which is addressed later. They all have a few huge hits, Aaron Donald for Snead, Julio Jones for Dimitroff, Keenan Allen for Telesco, and Joe Thuney for Belichick and they all have a few large busts, but they are fewer in number and less in degree. That means their misses are less egregious than their hits are great.

This in some part works against the narrative that Belichick is, much like Howie, bad at drafting. This one is likely fueled by his undeniable infatuation with Rutgers and Naval Academy players, or maybe some desire to bring down the evil empire, but it simply not played out in the numbers. The only draft in this time span in which was overwhelmingly negative was the aforementioned 2017 draft, in which Belichick drafted 4 players, 2 in the top 100, and 3 of them busted. There is room to consider this poor asset management to only have 4 picks, and there are some arguments to be made about not properly valuing positions by spending high capital on interior defensive linemen and running backs. But it is frankly wrong to say Belichick is bad at scouting and picking players. Belichick is a good, not great, drafter.

Tier 3 – Jason Licht, Rick Spielman, Howie Roseman, Kevin Colbert, Marty Hurney

This tier covers the widest range of average difference from average EV/year, but I feel comfortable placing them all together. They all go together because the players they draft, on average, tend to perform about how they’re expected to. Licht’s drafts follow amusingly follow a similar pattern to his former QB Jameis Winston’s performance. His 2015 draft was incredible, with every draft pick (in terms of AV) being a huge hit until their 5^th, 6^th, and 7^th round picks which is very forgivable. His 2016 draft, however, was essentially the opposite with picking a kicker in the 2^nd round who ended up with an AV of 0, and only a single pick being considered a hit: Backup OT Caleb Brantley in the 5^th round. The 2017 and 2017 drafts were generally positive, with the hits (Chris Godwin, Mike Evans, and Kendall Beckwith) being balanced out by the misses (Jeremy McNichols and Austin Sefarian-Jenkins). Rick Spielman follows a similar model where his 2015 draft was incredible, which was immediately balanced out by this with his awful 2016 draft, and his remaining drafts resulted in a net 0 essentially.

Hurney is undeniably the worst of the group with only one of his three drafts in this time range was a net positive in 2012 when he picked future hall of famer Luke Kuechley and all pro Josh Norman. In the prior 2 drafts combined he had exactly 3 players which were considered good by AV, Cam Newton, Greg Hardy, and Brandon LaFell. Kevin Colbert also according to this model is similarly maligned where he only had 3 positive AV drafts in 2010, 2013, and 2017. Of the negative drafts, only one was particularly egregious in 2015 where he picked only a single positive AV player in 7^th round DE Anthony Chickillo. It is worth noting that essentially if a 7^th rounder makes the team he is considered a positive AV pick.

Now we get to Howie. Howie is unique from the rest where he only has one egregiously bad draft, and that is in 2014 where he made arguably the worst pick in NFL history by selecting Marcus Smith, who failed to record a single statistic in his rookie year, in the first round. Beau Allen and Jordan Matthews save this class from being amongst the worst observed. His 2010 draft was also negative, but most of that resides in the fact that many 4^th rounders and less didn’t make the team and using an average puts Brandon Graham, who had a late surge in his career, at a disadvantage. 2017 was also negative, with Derek Barnett, who has only had a single season as the primary starter and an AV of 7, and Sidney Jones who has undeniably been a disappointment. The remainder of the negative AV picks were 4^th round and later picks who did not make the team or had little impact. The remainder of his drafts have been net positive, or essentially 0. Does this mean Howie is bad at drafting? Can Howie not pick good players? No, absolutely not. The narrative that Howie is “bad” at picking good players is simply not played out in the data. He is certainly not among the league’s best, especially with his mixed history picking on day 3, and there are arguments to be made about his failure to properly value defensive backs. But the idea that he is bad at picking good players is not true. He an average GM in terms of drafting good players, and excels in every other aspect of his job.

Tier 4 – John Elway, Duke Tobin, Steve Keim

These GMs are the worst of the ones observed. In a combined 20 drafts, only 6 of them had a net positive AV. Half of those came from John Elway in 2011, 2012, and 2016 when he drafted Von Miller, Julius Thomas, Derek Wolfe, Malik Jackson, Danny Trevathan, Justin Simmons, Connor McGovern, and Will Parks. His two most egregious drafts were in 2013 and 2017 where only 3 players in total were drafted with an AV were Montee Ball (who had one good season and then fizzled out), Garrett Bolles, and Isaiah McKenzie. Duke Tobin lucked his way into two positive AV drafts, in 2011 when he drafted AJ Green, Andy Dalton, and Clint Boling and in 2016 when he drafted Tyler Boyd, Nick Vigil, and Andrew Billings. Most egregious was his 2015 draft when his sole positive AV pick was CJ Uzomah. To say the very minimum, that is not a great look.

Steve Keim is far and away the worst GM of those observed. Of the 36 picks he made over this time span, only 11 of them have been positive AV picks: Tyrann Matthieu, Alex Okafor, Andre Ellington, John Brown, Markus Golden, David Johnson, Rodney Gunter, JJ Nelson, Brandon Williams, Evan Boehm, Budda Baker, and Rudy Ford. Given his relative incompetence in the observed drafts, and his inefficiency in his signings, I do not believe it is a stretch to say much of his job security comes from the fact that Bruce Arians is a terrific coach leading the team to many wins, and Keim’s history of getting cheap trades for DeAndre Hopkins, Carson Palmer, and Chandler Jones. That, or Steve Keim is a lizard person. Considering I have never seen Keim near, or in, a body of water I am going to believe that he is a lizard person and that firing him would lead to intergalactic war ending in the annihilation of the human race.

Draft Twitter is one of my favorite parts of the NFL. It is a welcoming community which allows for many people to indulge their passions and learn more about the greatest game ever made. It allows for fans to learn about and appreciate future NFL players before March and hearing what McShay and Kiper have to say. But like all communities, and like all things with the draft, they get things wrong sometimes. In particular, they are wrong about Howie Roseman. Howie Roseman is not bad at drafting players, but he is not among the league’s elite. He is just okay at drafting, and with how strong he is at the rest of his responsibilities, that is more than enough.