The “all else equal fallacy”

I like John Tierney’s New York Times column (for example, here), but sometimes he goes over the top in counterintuitiveness.

Here, for example, Tierney writes about someone who says, “in some circumstances it’s better to drive than to walk. . . . If you walk 1.5 miles, Mr. Goodall calculates, and replace those calories by drinking about a cup of milk, the greenhouse emissions connected with that milk (like methane from the dairy farm and carbon dioxide from the delivery truck) are just about equal to the emissions from a typical car making the same trip. . . . Michael Bluejay, who’s done some number-crunching at BicycleUniverse.info, says that walking is actually worse than driving if you replace the calories with food in the standard American diet and if the car gets more than 24 miles per gallon. . . .”

This is interseting to me because these guys are making a classic statistical error, I think, which is to assume that all else is held constant. This is the error that also leads people to misinterpret regression coefficients causally. (See chapters 9 and 10 of our book for discussion of this point.) In this case, the error is to assume that the walker and the driver will be making the same trip. In general, the driver will take longer trips–that’s one of the reasons for having a car, that you can easily take longer trips. Anyway, my point is not to get into a long discussion of transportation pricing, just to point out that this seemingly natural calculation is inappropriate because of its mistaken assumption that you can realistically change one predictor, leaving all the others constant.

As we like to say, it’s a great classroom example.

P.S. More here (also see discussion in the comments below).

14 thoughts on “The “all else equal fallacy”

  1. The big fad in land-use planning called smart growth, and more specifically "new urbanism," maintains that people will walk up to 1,500', but beyond that, they'll get in a car and drive. So for the legions of community activists & planners out there who want to make "walkable" communities, this is relevant in that (as I've been told time and again) people will walk further with better designed communities. In addition to wanting people to walk further, in places where destinations are within that 1,500' radius, driving 2,000' may still have less global-warming effect than walking 1,500'. So, redesigning a community may be bad for the environment…ostensibly, anyway.

    I think the assumption that's really off the mark is the calorie replacement. We can probably do quite a bit of walking before our walk-drive decision comes down to optimum calorie saving or use.

  2. I don't think you can call this a fallacy. The way I read him, the question at hand was simply whether a given distance could be covered more energy-efficiently by car or on foot, and that was it.

  3. Lemmus,

    The article focused on the calorie costs of a given trip, but it drew implications about lifestyle choices. Tierney described the analysis as "an interesting challenge to the notion that taking short trips in a car is bad for the planet." As we say in statistics, the question is always, "compared to what?". The fallacy is supposing, when altering the mode of transportation, that the length of trips will stay the same. (A similar fallacy is made when comparing air to land travel.)

  4. Did the original analysis take into account that on a per km/mile basis, cars doing short journeys
    a. use more fuel
    b. emit more pollutants
    c. wear out more quickly
    Compared to longer journeys

  5. Having just reread the post, I don't see where it "drew implications about lifestyle choices".

    I think that the post addressed a simple question: If you want to travel a given distance, is it more efficient to walk or to drive? (Sidenote: I have not read the research Tierney refers to, but I have a suspicion that it does not take into account that in urban areas, driving to a spot a mile away means driving a mile and then starting to look for a space to park.)

    I fully understand that there are other aspects – once people have decided to buy a car, they'll be more likely to go to the big box store five miles away rather than their local cornershop, for example. Of course you'd want to take these things into account when assessing the full environmental impact of cars and making political decisions about things like how to set gas taxes.

    Maybe we should just agree to disagree about our respective interpretations of the post; I really don't feel like comparing you to Hitler over this ;-)

    (Anti-disclaimer: Personally, I really hate cars, so that's not where I'm coming from.)

  6. "This is interseting to me because these guys are making a classic statistical error, I think, which is to assume that all else is held constant…. In this case, the error is to assume that the walker and the driver will be making the same trip. In general, the driver will take longer trips–that's one of the reasons for having a car, that you can easily take longer trips."

    No, this is not an error, and the challenge is nothing short of bizarre to me. The reason we assume that everything else is constant is that that's the comparison being made. If we change the variables then suddenly we're making some other, completely different comparison. Yes, people who drive tend to go farther distances than people who don't. That couldn't be more irrelevant to the point being made, which is that walking ultimately uses more fossil energy than driving, on a trip of the same distance, if the calories that fuel the walking come from the typical animal-rich American diet.

    Now, one could argue that the values used in the calculation are in error, and therefore the results are too, and such an exploration into the numbers would be a useful contribution, but most of the charges actually leveled against the articles are neither useful nor relevant. I do encourage people to carefully read the article(s) in question, since most of the criticism I see of them leads me to believe that the critic either hasn't seen the article or has just skimmed it briefly — or have perhaps relied on others' incorrect impressions about the articles. For example, blog posts often refer to these articles with a headline something like "Save the planet by driving", which is absolutely *not* the point I am trying to make. :)

    My article is linked from my name.

  7. Michael,

    In statistical terms, the problem is one of extrapolation or generalizability. You're free to make a calculation assuming all else is constant, but I don't think that's so relevant in comparing decisions. People travel more when they have a car than when they have to walk. As noted in my above entry, this arises in a lot of applications of statistical models: in practice, when you change one input to a system, other inputs change also. (As the economists say, variables are "endogenous.")

  8. Michael, I completely agree with you that people should read your article before criticizing your viewpoint.

    On a similar note, did you read the book chapters Andrew referenced before criticizing his viewpoint?

  9. I've read all of the articles from this discussion so far, although not all of the higher-order links (i.e. links found in those other articles). Apologies in advance if the answer is already given somewhere: did anyone document the energy consumption and greenhouse gas production for the various cases?

    Just to give some specifics for discussion, here's what I've got. Let's look at the "average" diet, by taking the total amount of calories consumed and dividing by the total amount of energy used in the chain from food production to consumption.
    My online searches (guaranteed correct 30% of the time!) say that in the U.S. people consume an average of about 2050 Calories per day and that food production/shipping/preparation consumes about 170 Million Tons of Oil Equivalent (that's a unit of energy, a particularly ugly one). Simple division gives me about 1 Cal of food energy per 10^-6 TOE, or 1 Cal per 2*10^-3 pounds of oil. Walking 1 mile takes about 100 calories (for a typical guy), which means 0.2 lbs of "oil equivalent." Call it 0.2 lbs of gasoline. In contrast, driving a mile takes about 0.3 lbs of gas (at 20 mpg).
    So: (1) these numbers are indeed roughly comparable, but (2) my estimate says that driving is 50% worse. On the other hand, the food estimates here don't take into account other greenhouse gas emissions (such as methane from cows). On the other other hand, they also assume that all of the energy for food comes from burning oil, whereas in fact some will be less greenhouse-gas-intensive than that (but, on the other other other hand — I guess we're talking about Shiva here — some may be worse, too). Taking it all together, it looks to me like (contrary to some claims) walking and then eating the "average" diet leads to _lower_ greenhouse gas emissions than driving, but not by a huge margin. But replacing those calories with a high-greenhouse-gas food like meat or dairy does indeed make walking worse than driving by this very narrow measure.

    I'm in between Michael and Andrew on whether this is an appropriate comparison to make. I think it's a useful fact — that driving a given distance and walking it are in the same ballpark in terms of the greenhouse gas emissions associated with the calories. But many people interpret this to mean that if we all walked places instead of driving, we wouldn't be better off in terms of greenhouse gases, and that's not true for several reasons including, as Andrew points out, that if we walked instead of driving, we wouldn't travel nearly as far. (I know people who commute 30 miles EACH WAY every day; try doing that by walking!)

    One more important point: As several people, including Michael B, have pointed out, things look a lot better if you eat something less energy-intensive than the "average American diet" to replace those calories (or, better still, if you don't replace them at all). Meat, dairy products, and foods fried in corn oil are ridiculously greenhouse-gas-intensive. If, instead, you have a piece of fruit or some mashed potatoes or, indeed, almost anything that isn't meat and isn't fried, then walking comes out way better than driving.

    But of course, also as noted by Michael B on his blog, the real message is: biking rules. Ride your bike and get another factor of 2.2 or so!

  10. The argument includes the exogenous costs of walking (food calories) but not the exogenous costs of driving other than fuel– air pollution, danger, the environmental cost of roads and parking lots, the resources used in building the car and making the fuel.

    It assumes the use of a very inefficient way of making food calories, and does not include the health benefits of walking that.

    Those authors are stooping pretty low in order to sell books.

Comments are closed.