Dave Berri writes:
Saw you had a post on the research I did with Rob Simmons on the NFL draft. I have attached the article. This article has not officially been published, so please don’t post this on-line.
The post you linked to states the following: “On his blog, Berri says he restricts the analysis to QBs who have played more than 500 downs, or for 5 years. He also looks at per-play statistics, like touchdowns per game, to counter what he considers an opportunity bias.”
Two points: First of all, we did not look at touchdowns per game (that is not a per play stat). More importantly — as this post indicates — we did far more than just look at data after five years.
We did mention the five year result, but directly below that discussion (and I mean, directly below), the following sentences appear.
Our data set runs from 1970 to 2007 (adjustments were made for how performance changed over time). We also looked at career performance after 2, 3, 4, 6, 7, and 8 years. In addition, we also looked at what a player did in each year from 1 to 10. And with each data set our story looks essentially the same. The above stats are not really correlated with draft position.
This analysis was also updated and discussed in this post (posted on-line last May). Hopefully that post will also help you see the point Rob and I are making.
I’m out of my depth on this football stuff so I’ll leave it to you, the commenters.
The problem lies in what to do with the players who have no playtime statistics. Do they fail to start because they're not good enough, or because there just aren't enough QB spots and teams go with the players they heavily invested in? Assume they're as good as the other 6th rounders who do go on to have long careers and you see no correlation, assume they're all bad players and you see strong correlations.
Barring more data, the best we can probably do is look at other sports, specifically baseball. Baseball is actually known for its difficulty in predicting performance, with lots of first rounders never making it to the Bigs. The nice thing is that its a lot easier to measure baseball performance statistically, and a large minor league system means there's plenty of actual game situations to evaluate players who don't make it. It's possible that Baseball prospects can be scouted and QB's can't, but barring other evidence it seems to make sense that the two tasks are roughly equal in difficulty.
I'm not going to do the full statistical analysis, but this suggests by year 5 of playing, draft position has stopped mattering (the bad players having been dropped in all/most draft groups). But earlier there is a difference, and it does appear that far fewer lower round picks make it. Not having done the analysis I can't say for certain there isn't still confirmation bias on behalf of top picks, but given all the stats available for player performance and all the top rounders who don't make it, it appears that this is a legitimate difference in ability.
Anyways, this has been an enlightening statistical topic to think about. So many different ways the results can be viewed given an important, hard-to-measure variable.
When I saw this discussion, I thought Gladwell was credulously believing something that just can't be right. Having looked at it a bit, though, I'm not so sure. It's hard to see a really good way to evaluate quarterback quality versus draft position, mostly because there is no good measure of QB quality. Unlike baseball, in which many aspects of the game can be analyzed one or two players at a time (hitter versus pitcher, fielder versus ball in play), there really are no single-player stats in football. Even a great quarterback looks average (or even bad) if he has an offensive line that can't give him time, or a receiving corps that can't get open. Even an average quarterback can look good if they have great receivers and a great offensive line. And most quarterbacks look bad for their first several NFL games, even if they go on to become great players.
I went to pro-footballreference.com and got the data on every QB drafted in the first four rounds of the draft from 1985 through 2000. I calculated the "passer rating" (according to the NCAA formula, which is conceptually similar to the NFL formula but is even simpler to calculate). (I also invented my own rating to correct for what I see as a deficiencies in the NCAA formula, but the results are so similar to the NCAA formula that it doesn't much matter which one I use). And… well, it's just a fact that when I look at quarterbacks who have a substantial number of plays — for instance, the ones whose passes + rushes is greater than 50 (many plays would not be classified as either, such as a handoff) — the early draft picks are not better than later ones. Recall that I'm not looking at really late picks, just the first few rounds. It's my impression that if a quarterback is thought to have star potential, some team will take them in the first 10 or 20 picks, even trading to be able to draft earlier if necessary; the ones below position 50 or so, nobody really thinks is a likely star.
QBs taken with very early picks include some who are very highly rated (Daunte Culpepper, Peyton Manning, Donovan McNabb, Steve McNair, Troy Aikman. Quarterbacks have interesting names). But early picks also include low-rated players such as Akili Smith, Ryan Leaf, Kelly Stouffer, and Heath Shuler. I'm not much of a football fan, but I thought I remembered that Jeff George was a big disappointment, and maybe he was, but he had a pretty good passer rating. Anyway, the early QBs are all over the map.
Well, the later picks are also all over the map. There are low-rated quarterbacks (most of whom I've never heard of) like Will Furrer, Jeff Carlson, Jeff Lewis, and Browning Nagle, but also highly rated ones like Steve Beuerlein, Brian Griese, Rich Gannon, Rob Johnson, Chris Chandler, and Kordell Stewart.
(Hmmm….here's my theory: QBs with unusual first names tend to be chosen earlier in the draft.)
If you want to show that early picks are better than later ones, you can probably find a way to justify that belief, if you look at the data the right way. In fact, in my data the QBs taken after about pick number 90 are noticeably worse (on average) than the ones taken earlier. (Mean rating, from the NCAA formula, of those taken with the first 10 picks: 117. Taken with picks 11-79: 117. Picks 80-140: 110).
Certainly if you assume there is any correlation between number of plays and QB quality, or number of passes and QB quality, you will find that early picks look better: everyone agrees that highly-paid early draft picks are given more chances to play.
As I said earlier, football is very much a team sport; the quarterback rating is a very, very noisy measure of quarterback quality. You have teams that invest in a good offensive line and good receivers, but can't afford an early-pick quarterback so they end up drafting lower down; this QB can still end up with healthy stats even if they're not that great. On the other hand, you have mediocre teams that pin their hopes on a premier quarterback, and these QBs may end up with only average stats even if they're actually quite skilled.
I think it's possible that coaches/scouts/owners are doing substantially better than chance at evaluating QBs, but that this doesn't show up in the very crude "quarterback rating" measure. But now that I've seen the data I also think it's possible that the coaches/scouts/owners don't have a clue.
If you think of QB success as a binary variable based on playing time, then yes there is a correlation: a higher pct of 1st round picks become "successful" than 6th round picks.
The causation for that correlation is debatable(actual QB quality vs opportunity), but I think even more important is that the correlation seems utterly irrelevant to me.
Every NFL team has a starting QB, so merely measuring if a player was a starting QB reveals nothing about his ability to actually help his team win. Since the whole point of the NFL is to win, that presumably is the point of the draft as well. In other words, drafting a bottom tier starting QB in the 1st round is NOT a success in my estimation, and I doubt it would be considered a success by any NFL team as well.
What matters to NFL team success is to have good players, not just players. So I don't see how QB success can be predicated on a measure which doesn't distinguish between good and bad QB's. The appropriate question to ask is how much better, if at all, are early QB picks vs late QB picks.
The answer, at least looking at average rate stats, is that high round picks are little better than low round picks. Furthermore, considering that the cost of a QB draft choice to an NFL team is not a linear function of draft round, that 1st round picks are paid many multiples of what low rounds picks make, it suggests that a good strategy for an NFL team might be to go with quantity over quality in terms of QB's.
That's actually an impractical strategy as an NFL team simply can't have 10 QB's in training camp due to roster constraints, etc., but it is interesting to note that well run NFL teams do seem to engage in QB prospecting. They draft a low round guy, try to develop him for a year or two and then make him the starter or flip him for a higher pick. I'm thinking of Tony Romo with Dallas, Tom Brady with NE as guys who became starters, and guys like AJ Feely and Matt Cassel who were developed, got to play due to an injury to a starter and then were flipped for a higher draft pick when the starter returned to health. Poorly run teams like the Lions and Redskins seem to repeatedly draft QB's in the 1st round at great expense, while smarter teams try to find them in cheaper ways if possible. Of course this is all anecdotal, I'd have to think about if there is anyway to quantify it and test it.
The MLB draft is a very different animal. For one thing, players can be as young as 17 and have never played at a higher level of competition than high school. On top of that there huge regional differences in the quality of high school and even college baseball. So, the NFL draft is from a much more homogeneous group of players, that have all played at a level of competition closer to the NFL than the average MLB draftee.
Additionally, there are structural differences. A high school baseball player typically has a lot of leverage with the MLB team that selects him. If he doesn't receive a contract offer that he likes he almost certainly has a college scholarship waiting for him, and if he has already graduated college, there are independent minor leagues that he could play in for a year and then re-enter the draft. The impact of this is that many high round draft picks are selected not solely on talent alone, but also on "signability". Teams that are strapped for cash may be fear not being able to sign a top prospect and so purposely select a lesser player who they know they can sign within their budget. Furthermore, teams that are flush with cash, often use late round picks to "roll the dice" on players generally perceived to be unsignable due to a college scholarship. So, a player may be considered a 1st round talent, but also be considered very unlikely to sign because he really wants to go to State U, but then the Yankees draft him in the 10th round and offer to pay him as though he were a top 5 pick and suddenly he decides college can wait.
Bottom line there are lots of structural differences between MLB and the NFL which make their drafts very different, and the added leverage that baseball draftees have can dramatically distort the draft order. Because of that and the age/experience differences I would expect MLB draft order to be much less correlated with success than the NFL draft order.
Having said that, MLB data is much easier to work with, because the whole playing time question goes away. MLB draftees do not simply sit on the bench and never get to play. They go to the minors and have to work their way to the majors. So, a great deal of the missing data problem regarding low round NFL picks, is solved when looking at MLB data.
This is a good issue for statistical analysts to think about because it's a problem that comes up in a lot of fields.
Most of the top draft picks get significant playing time in the NFL. Some of them turn out to be even better than expected (e.g., Peyton Manning), some about as good as expected (e.g., Carson Palmer), and some worse than expected (e.g., Ryan Leaf).
With low draft picks, some turn out in training camps and practices to be better than expected and eventually get a lot of game time. Others turn out to be about as good as expected and get just a little or no playing time at all. Others turn out to be even worse than expected and never get any chance in games to put up any statistics.
Counting statistics, such as career yardage, are biased in favor of high draft choices because they get more playing time even if they aren't as good as expected.
Per play statistics, which Berri uses, are biased in favor low draft choices because they only get a lot of playing time if they are better than expected. Berri largely ignores all the lower draft choices who turned out to be no better than expected and thus didn't get much playing time. Not surprisingly, the lower round picks who turned out to be surprisingly good have unsurprisingly high per play statistics.
Moreover, lower draft pick quarterbacks are seldom thrown into the starting role before they are mature, whereas top draft picks often rack up a lot of bad per play numbers before they hit their physical and mental primes, and are often worn down by injuries when they reach what should be their primes in their late 20s.
Also, very early draft choice quarterbacks (e.g., top ten picks in the first round) generally go to bad teams, which tend to drag down the QB's stats.
For example, Compare Peyton Manning's per play statistics to Matt Cassel's after they first had 500 pass attempts.
Peyton Manning was drafted #1 by a bad team and was immediately made the starter at age 22 and was told to throw a lot of passes. He threw 28 interceptions his rookie season, so his per play statistics that year were bad.
In contrast, Matt Cassel mostly sat on the bench for four years at USC behind two Heisman winning quarterbacks, got drafted by New England in the 7th round, sat on the bench behind Tom Brady, and finally became a starter at age 26, when he was plugged into the New England offensive juggernaut for the injured Brady, and had a very good season.
So, after the seasons in which Manning and Cassel each passed the 500 attempts threshold, Cassel's per play statistics were much better than Manning's. Cassel signed a big contract with a lousier team, and then had a mediocre season.
One way around the bias in favor of top draft picks getting a lot of playing time is to set the bar for success very high, such as selection to the Pro Bowl all star game. For all quarterbacks drafted from 1980-1999, the correlation between draft order and career Pro Bowl picks is -0.33. So, there is a lot randomness, but no, it's far from completely random.
The commenters on Prof. Berri's post on his blog do a good job of explaining why the claim he's now making (which is much more limited than Gladwell's version of it that Pinker called out) is based on a very dubious assumption.
Huh. Very interesting. No matter how much I learn about baseball, there's another layer behind it. I knew that teams had to negotiate with their picks, but I never really considered how much that could impact draft order.
But the noise in MLB drafting actually seems to make it better as a proxy for the NFL, because, as you say, it seems likely that drafting is harder in baseball (and that is the reputation it has). If we condition on the hypothesis that baseball drafting is harder, then any correlation we discover there would be a lower limit on the minimal correlation we'd expect to find in football.
While I agree that there are some advantages that might inflate the rate stats of lower round picks, there is a major disadvantage as well. Ultimately teams have to decide if a player is good or not. While low round picks may not be forced into playing too soon thus conferring an advantage, the disadvantage they face is they are more likely to be judged too quickly. In other words, a 1st round pick while he may be rushed and forced to play before he is ready, is more likely to get additional opportunities or time to prove himself. A low round pick likely will not.
For example, it didn't matter that Peyton Manning wasn't "Peyton Manning" in his rookie season, he continued to get a chance to start. In other words, because of their status as top picks, teams are more patient with 1st round QB's and usually wait before making a final judgement on them. The larger number of observations affords highly selected QB's a greater probability of displaying their true level of ability on the field. You would expect fewer False Negatives(rejecting a QB who actually can play) and thus more True Positives among high draft choices.
But among lower picks, teams have less invested, presumably have less faith in those players(who often get their chance due to injuries in front of them like in Brady and Cassel's cases), and are more likely to make a decision on a player without having a large enough sample size to support it. For example, even though Tom Brady is a Hall of Fame caliber QB, I'm sure it is possible to find a sequence of games in his career where his performance was average at best. Had his career started with that sequence of games, Drew Bledsoe might well have regained his starting job when he was healthy and the world would have fewer bastard children of supermodels. It seems likely that the lack of opportunity or shorter leashes associated with lower round picks would create a bias towards more false negatives and fewer true positives.
This is important because essentially it seems the percentage of high round picks who could have made the Pro Bowl but didn't get the chance to prove it would be near zero, while among low picks it could be much higher. So, low picks are biased against by the fact that decisions made about them are more likely to be based on incomplete information, and therefore more based on statistical noise than true ability.
Sure, but you can make up hypotheticals the other way, too. For example, what if Ryan Leaf had been drafted by a good team at 28th instead of 2nd, given a small contract so he couldn't afford drugs, and made the third stringer and told to watch and learn how to play in the NFL? He might have eventually lived up to his physical ability.
Look at Steve Young. He had one awful year as a young starter in the NFL for a bad team, then sat on the bench behind Joe Montana for years. When he finally became the starter for SF, with Jerry Rice to throw to, he racked up a career of astounding per play numbers. But his career per play averages wouldn't have been that high if he'd played a lot of years for Tampa Bay as an immature starter.