In the 1970s and 1980s, Bill James put a lot of effort into predicting how well young players would do. In 1988, he summed up 15 things he'd learned, and three of them related to forecasting young players' development:
James essentially found, unsurprisingly, that the closer players got to the majors, the easier it is to predict their major league performance. Minor league hitters can be predicted reasonably well from statistics alone. Drafting college players was usually safer than drafting high school players. High draft pick high school pitchers, I believe, were especially likely to flame out.
Partly this effect was maturity and injuries, but, I imagine, it also had to do with the usefulness of college statistics v. high school statistics. If a prospect hits .350 for the Rice Owls, you can conveniently look up how that compares to what Lance Berkman hit at Rice (.385) and what Berkman is hitting in the majors (.299). But if the prospect hits .500 for Horace Mann High School, which hasn't sent anybody past Rooke League ball in decades, how do you know how good that number is?
One problem with predicting quarterbacks' performance relative to other kinds of athletes is that, typically, only one gets to play at a time. Baseball teams have five starting pitchers. Baseball hitters can typically get squeezed in for a look at multiple positions. Football running backs generally substitute in and out so that they get a breather. But second and third string quarterbacks can get stuck for years with very little opportunity in games to show what they can do. If Joe Montana had been as durable as Peyton Manning, Steve Young might have been a career backup.
Kurt Warner didn't get to start until his senior year in college, then went undrafted. He was invited to the Green Bay Packers camp, where he competed for a job with Brett Favre, Mark Brunell, and Heisman-winner Ty Detmer. Not surprisingly, he wasn't as ready for the NFL as those guys. It took him four years stocking shelves, playing Arena football, and for the Amsterdam Admirals to make it to the NFL. By his second year in the NFL, he was the MVP.
Because the quarterback is so central to the offense, quarterback changes are a big deal, hashed out endlessly on sports talk radio. Moreover, because coaches don't like to change quarterbacks, starters play banged up a lot. If a big league pitcher is at 85% physically, so that his fastball drops from 93 to 79 mph, he's out of the rotation until he gets better. But if a quarterback is at 85% physically, he probably continues to start to maintain team continuity. This means his per play performance (which Berri measured) drops.
This also means an overlooked quarterback can be healthier than a heralded starter of the same age, boosting his per play performance when he finally gets in the game. For example, Matt Cassell didn't start a game between his senior year in high school in 2000 (?) and his taking over successfully for the injured Tom Brady in 2008. (He was stuck behind two Heisman trophy winning quarterbacks at USC.) That's a lot of punishment he didn't absorb.
