Skip to content
 

The Commissar for Traffic presents the latest Five-Year Plan

What do Paul Samuelson and the U.S. Department of Transportation have in common?

Phil Price points us to this news article by Clark Williams-Derry:

As the State Smart Transportation Initiative at the University of Wisconsin points out, the US Department of Transportation has been making the virtually identical vehicle travel forecasts for well over a decade. All of those forecasts project rapid and incessant growth in vehicle travel for as far as the eye can see. Meanwhile, actual traffic volumes have flattened out, and may actually be falling.

VMT-C-P-chart-big1-541x550

Each of the rising colored lines represents a forecast from a different year. The black line represents actual traffic trends on US roads—which never rose as quickly as the forecasters had predicted, and actually started a modest decline in 2007.

I’d like to see a label on the y-axis, and I’d recommend labeling the x-axis at 5-year intervals rather than every year, but the point seems pretty clear.

Williams-Derry continues:

In this entirely USDOT’s fault, though. These forecasts are a “roll-up” of forecasts made by state DOTs. The US agency just collects the forecasts and reports them to the public: garbage in, garbage out.

But in a way, that’s even more sobering than if the fault were localized in USDOT, since it provides clear and compelling evidence that the nation’s entire transportation forecasting apparatus is completely broken. In the aggregate, all of those hard working forecasters in all of those state DOTs are just making up numbers.

Indeed. I’m reminded of this notorious graph from the legendary Paul Samuelson textbook (from 1961):

samuelson.png

As Alex Tabarrok points out, the graph is even worse than it looks: “in subsequent editions Samuelson presented the same analysis again and again except the overtaking time was always pushed further into the future so by 1980 the dates were 2002 to 2012. In subsequent editions, Samuelson provided no acknowledgment of his past failure to predict and little commentary beyond remarks about ‘bad weather’ in the Soviet Union.”

The bit about the bad weather is funny. If you’ve had bad weather in the past, maybe the possibility of future bad weather should be incorporated into the forecast, no?

As Tabarrok and his commenters point out, this mistake can’t simply be attributed to socialist sympathies of the center-left Samuelson: For one thing, various other leftist economists did not think that the Soviets were catching up to us; and, for another, political commentators on the right at the time were all telling us that the communists were about to overwhelm us militarily.

As with the Commissar for Traffic, the problem was the linear model, the assumption that what has changed in the past will continue to change in the same way in the future.

What’s unforgivable, both in the case of Samuelson in the 1960s and the Commissars today, is not in making the mistake (surely, a linear extrapolation is quite reasonable in many settings) but in not recognizing the problem, even after the forecast has been completely destroyed by the data.

49 Comments

  1. Rahul says:

    Conflict of interest? How do DoT planners justify increasing budgets if vehicle travel forecasts keep dropping?

  2. Mark says:

    Although these examples both present absurd projections from linear models, I think there’s something of a disconnect or logical error in what you say or seem to imply by “the problem was the linear model, the assumption that what has changed in the past will continue to change in the same way in the future.” All prediction models, whether linear or not, seem to make that assumption to some extent, even if they allow for unprecedented change in some aspects of the problem they must assume constancy in others. Justifiable? Probably less so than we think, or so argue Taleb.

  3. Reminds me of the quarterly reports from our CFO when I worked at Speechworks (a speech recognition startup with great programmers and a lousy business model). Each quarter, the company’s expenses were about $2 for every $1 of revenue, but with strong revenue growth (this was right before the dot-com bubble burst). And each quarter, the CFO projected costs to be roughly flat going forward, leading to a kink in projections of profit that always coincided with the day of the presentation. I watched this every quarter for two years until the company sold itself on the brink of bankruptcy. And each quarter, I said, “Who believes these projections? How can he say this with a straight face?”. Turns out that wasn’t the only funny business going on; the SEC settled a fraud allegation relating to inflating revenue; at least they banned him from ever being a CFO again.

    • zbicyclist says:

      Bob Carpenter’s saga sounds all too real to me (except the fraud part). The upshot was also interesting where I worked. Naturally, in order to meet client commitments the headcount budget had to be exceeded at some level. So, to meet profit commitments, salaries would be frozen or cut, bonuses would be zeroed out, full time employees would be converted to contractors, and the best “rainmaker” salesmen would move on to greener pa$tures.

      Unrealistic forecasts have real consequences.

  4. Hi says:

    Why no links to the articles being discussed?

  5. It would be neat to see the original article, and the data on which the colored-line linear projections are based. It seems odd not just that the projections are wrong in the face of data, but also that they all have (roughly) the same slope! Is there some particular period, years past, that gave a 0.075 whatevers/year growth rate that’s being mindlessly applied to the present and future?

  6. ben hyde says:

    The mile traveled number was, for decades, tightly correlated with GDP, and GDP growth has been extremely smooth. When the business cycle wobbles so does the miles traveled, but both have tended to fall back to the trend line. So these estimates reflect that conventional wisdom, and the flattening out shows that the GDP growth has turned lousy. It is not irrational for planners to assume that recessions will end and the catchup with trend will happen yet again. That said, if we subtract away the correlation with GDP there does appear to be some deviation form the GDP::Miles-traveled correlation.

    Planners have a hell of a time telling weather form climate change :). And, only recently have people started talking about stagnation, i.e. a real phase change in the economy.

    The russian and us economy were a lot more independent systems, and hence not correlated, as compared to the miles traveled and the GDP.

    So I don’t know, maybe this isnt’ as mindless and example of an institution unable to admit things are changing as it seems.

    • Andrew says:

      Ben:

      Again, see the last paragraph of my post above, and see the graph which is flat since 2004. At some point these people should be learning, not just repeating last year’s error. Same goes for Samuelson.

      • ben hyde says:

        sigh. I have no opinion about Samuelson. But here is some data (35 years) on the miles traveled as a function of gdp

        http://www.theoildrum.com/story/2005/10/20/51230/831

        and so I think your cynical critique might not about traffic planning, but more it’s about the last 20 years of economists assuring us that we would return to growth pattern seen in those 35 years.

        • Phil says:

          I’m not sure how that wouldn’t make it a critique of traffic planning. Suppose my method of traffic planning were to ask Jimmy the Shoeshine Boy how much traffic he thought there would be in ten years, and that Jimmy does OK for while but at some point he has a terrible track record and steers me wrong year after year after year, always in the same direction. I think it would be fair to say “you have a lousy method of predicting traffic volumes.” Sure, I could point my finger at Jimmy and say “it’s not me, it’s HIM”, but I think if it’s my name on the prediction then it’s my responsibility to not have those figures be wrong in the same direction for two decades.

          • ben hyde says:

            It’s cute but a poor cartoon to suggest that taking the consensus GDP growth numbers as an input to the traffic planning model is no better than asking Shoeshine Boy.

            Yes each planner owns his predictions, but no it is entirely appropriate for him to say: “Here’s my model, here’s where it went pear shape, and this is the input that caused that.” It’s perfectly fair for Andrew to point out that something seems broken here, all I’m suggesting is (a) the breakage might be in the input not the model. And (b) that you don’t change a model that has worked well for decades impulsively.

            And yes, I agree that 20 years is a long time to be tabling changes to the model; but I’m more sympathetic as to why the motion to table carried in any given time frame over those 20 years.

        • Trevor says:

          Ben: the problem with using the graph to which you link as support for travel modeling assumptions is that it cuts off in 2005, right as changes in VMT start to seriously diverge from changes in GDP. The real critique of transportation modelers is that they’ve by-in-large been really bad at recognizing this divergence and the various mechanisms behind it.

          For discussion of the divergence the mechanisms, David Levinson is a great source: http://transportationist.org/2013/12/26/structural-errors-in-forecasting/

        • For kicks, I quickly looked up data on GDP and vehicle-miles traveled, plotting the ratio over the past few decades (http://eighteenthelephant.wordpress.com/2014/01/22/you-can-use-any-model-you-want-as-long-as-its-linear-and-has-a-positive-slope/). It seems to show a more pronounced and steady change in its trends that isn’t evident in the “oildrum” graph above (which goes to 2002).
          It’s possible the data aren’t correct — “quickly” being a key word above — but in all it seems like the reality of the past ~decade is pretty obviously different than that of 1960-1990, and it seems absurd not to account for this in a prediction.

          • ben hyde says:

            Those charts are great! Everybody who’s interested should go look at them. The sharp corners in the last two are fascinating. The bend in the miles/person looks to my eye as it’s the tepid economy followed by the recession. The longer term decline in miles/$-GDP? I got nothing to explain that. Well I can make up insta-theories(tm) but I don’t like ‘em much. For example it makes me I wonder if there was a change in how GDP or miles traveled was measured in 2000.

  7. jonathan says:

    I loosely follow traffic forecasts. What I see is much better: they don’t know how to model what’s been happening so they report actual and don’t hide their error. That paints Samuelson as much worse.

    I know that in the years since miles dived, there were expectations about rebounds, etc., and a belief that the curve had bent but maybe that was short term – and thus maybe the dive was short-term, etc. I’m not sure if it would be better to trash the model for “better” or to rely more on actuals.

    Another thing is that these numbers become meaningful mostly at local levels where actual counts are more the decision driver – sorry about the pun. I’ve done a bunch of traffic improvement projects and each relied mostly on current conditions plus expectations with much more emphasis on solving current issues. So states and localities with lower counts are shifting resources around – in part, because they’ve been cut so much they don’t have enough anyway and in part because they always have problems that need money. And some states and localities see growth and can’t keep up. I’m less familiar with those areas, but I assume they continue, to the extent they have money, to plan for future capacity because one time building is much less expensive than retrofitting.

  8. […] tip to Andrew Gelman who compares with some other famous […]

  9. Fernando says:

    To paraphrase one former IMF official: “The problem is not the forecast. The problem is road traffic has yet to catch up with our forecast.”

    Maybe USDOT should outsource their traffic forecasts to investors with money riding on the forecast, or simply create a Kaggle competition.

    They ought to have Chinese Walls separating forecast generation from forecast use, and even then…

  10. David says:

    Looks like Indiana nailed the window to do a long term lease of their toll road in 2006! (I-80/I-90). Taxpayers win; caveat emptor.

    Source: I was part of the State of Indiana deal team.

  11. Phil says:

    I think someone should mention how great the title of this post is. Consider it noted.

  12. Filloaks says:

    I started my career in the electric utility business in New England. We had forecasts that looked very much like the DOT after the first OPEC embargo, when households and businesses started conserving because of price.

    Utilities kept right on assuming that electric consumers would return the old trend line, after people got tired of turning off the lights after the left a room.

    We used to call those DOT like graphs, fan graphs.

  13. […] It is inherently challenging to use a model to make predictions beyond the range of data to which the model was fit — making predictions about the future, for example, based on the past and present. Or, as the Danish proverb more elegantly says, “It’s difficult to make predictions, especially about the future.” Still, there’s no excuse for predictions that are quite as bad as these, from the U. S. Department of Transportation, on the predicted amount of total vehicle travel in the U.S., which I learned about via Andrew Gelman’s blog: […]

  14. Øystein says:

    Criticism seems justified in these cases, however at least for some public entities, not revealing market-sensitive information may be an objective in itself. A simple linear trend is a particularly straightforward (sorry) way of achieving that. Maybe then it should not be called a “forecast”, but it could still be useful in 1) prompting thoughts on what underlying factors may be different in the future, and 2) giving perspective on what the historical development was actually like.

  15. Bill Harris says:

    The first graph looks a lot like a graph (the NERC fan) presumably in http://www.jstor.org/discover/10.2307/1391589?uid=2&uid=4&sid=21103313095957 showing NERC forecasted electricity consumption in the USA, except that the forecasts rose exponentially, not linearly. I haven’t read that article; I’ve seen the graph on p. 642 of John Sterman’s /Business Dynamics/. Chapter 16 of that text is a good one to read when thinking about forecasting.

  16. StreetEYE (@StreetEYE) says:

    fair enough, but you’d have a field day with forecasts from National Association of Realtors, S&P, Gartner Group.

    even the crowdsourced forecast known as the financial market is often spectacularly wrong.

    when anyone makes a forecast, you need to understand, the context, incentives, etc.

    human fallibility is not a left/right commissar/capitalist pig issue.

  17. […] graph (from Clark Williams-Derry via Andrew Gelman) shows US national traffic forecasts over time (the straight lines) and the reality (the black […]

  18. skeptonomist says:

    Could we infer from this that the DOT’s (and Samuelson) did not have a crystal ball? Without a crystal ball, most projections are nearly worthless, even those done more carefully than the traffic projections. The exceptions are when things have actually been constant for decades.

    • Andrew says:

      Skeptonomist:

      1. Organizations make forecasts because they need to make decisions. Good forecasts are better than dumb forecasts. If you think most projections are nearly worthless, that’s fine, you can do something else, they’ll hire someone else to do forecasts who is actually interested in the job.

      2. Actually, that “crystal ball” thing is just a story. Crystal balls don’t really tell the future. Don’t believe everything you see in the movies.

      3. See the last paragraph in my above post. To make a mistake is fine. To repeat a mistake over and over after it’s been contradicted by the data, that’s not so fine.

      • Anonymous says:

        Andrew:

        “Organizations make forecasts because they need to make decisions”

        OR

        “Organizations make forecasts because they need funding”

        OR, more generally

        “Organizations make forecasts because they need [X]“

        Depending on X, USDOT’s forecasts my have been very useful indeed.

  19. […] in Another View. shared by @TheStalwart, @kevinroose, Marginal Revolution, @mccarthyryanj  The Commissar for Traffic presents the latest Five-Year Plan – Statistical Modeling, Causal In… andrewgelman.com What do Paul Samuelson and the U.S. Department of Transportation have in common? […]

  20. Steve Sailer says:

    The Department of Transportation was hardly alone in making this forecasting error. During the 2000s, the health of the economy rested upon the assumption that housing in the exurbs was a great investment because people would commute ever further to work. The collapse of some subprime firms in 2007 raised the first questions about this assumption. The gasoline price rise in the first half of 2008 finally killed that fantasy, leading to the collapse of Lehman Brothers in the late summer of 2008 and the ensuing deep recession.

    • Steve Sailer says:

      In other words, up through more or less September 14, 2008, the Big Money Boys were betting overwhelmingly on more miles being driven. Isn’t it asking a little much to demand that the lowly bureaucrats at the Department of Transportation have contradicted the Wisdom of the Markets?

  21. […] Charts Forecasting #fail – Andrew Gelman […]

  22. Dale says:

    We anchor ourselves to certain assumptions or worldviews and it can be hard to change. For another example see the link to how anlysts are almost always too optimistic with projecting earnings at SP500 companies even though they are only projecting 1-2 years in advance.

    http://blog.yardeni.com/2013/01/s-500-earnings-squiggles_8.html

  23. Dave Tufte says:

    In Utah, I’ve worked as a consultant for opponents of building a water pipeline from Lake Powell to St. George.

    I have no personal preference in this case, but it’s relatively easy to support the opposition with arguments based on data.

    State population forecasts are largely linear extrapolations. The local water authority has calculated the maximum population that can be supported by all available water resources in the area (including Lake Powell). The state’s population forecast rises smoothly to that limit, and then plateaus.

    It never seems to occur to anyone in the water authority, the county, or the state, that the plateau is determined by the yes/no answer to whether or not a pipeline will be built. In fact, the position is along the line that the pipeline must be built because the population estimates (based on the assumption that the pipeline will be built) can’t be supported unless the pipeline is built.

  24. SouthTexas says:

    Shades of global cooling/warming/climate change. Watch the computer model, not the thermometer!

Leave a Reply