Skip to content
 

(What’s So Funny ‘Bout) Evidence, Policy, and Understanding

[link]

Kevin Lewis asked me what I thought of this article by Oren Cass, “Policy-Based Evidence Making.” That title sounds wrong at first—shouldn’t it be “evidence-based policy making”?—but when you read the article you get the point, which is that Cass argues that so-called evidence-based policy isn’t so evidence-based at all, that what is considered “evidence” in social science and economic policy is often so flexible that it can be taken to support whatever position you want. Hence, policy-based evidence making.

I agree with Cass that the whole “evidence-based policy” thing has been oversold. For an extreme example, see this story of some “evidence-based design” in architecture that could well be little more than a billion-dollar pseudoscientific scam.

More generally, I agree that there are problems with a lot of these studies, both in their design and in their interpretation. Here’s an story from a few years ago that I’ve discussed a bit; for a slightly more formal treatment of that example, see section 2.1 of this article.

So I’m sympathetic with the points that Cass is making, and I’m glad this article came out; I think it will generally push the discussion in the right direction.

But there are two places where I disagree with Cass.

1. First, despite all the problems with controlled experiments, they can still tell us something, as long as they’re not overinterpreted. If we forget about statistical significance and all that crap, a controlled experiment is, well, it’s a controlled experiment, you’re learning about something under controlled conditions, which can be useful. This is a point that has been made many times by my colleague Don Green: yes, controlled experiments have problems with realism, but the same difficulties can arise when trying to generalize observational comparisons to new settings.

To put it another way, recall Bill James’s adage that alternative to good statistics is not “no statistics,” it’s “bad statistics.” Consider Cass’s article. He goes through lots of legitimate criticism of overinterpretations of results from that Oregon experiment, but then, what does he do? He gives lots of weight to an observational study from Yale that compares across states.

One point that Cass makes very well is that you can’t rely too much on any single study. Any single study is limited in scope, it occurs at a particular time and place and with a particular set of treatments, outcomes, and time horizon. To make decisions, we have to do our best with the studies we have, which sometimes means discarding them completely if they are too noisy. And I think Cass is right that we should take studies more seriously when they are large and occur under realistic conditions.

2. The political slant is just overwhelming. Cass throws in the kitchen sink. For example, “When Denmark began offering generous maternity leave, so many nurses made use of it that mortality rates in nursing homes skyrocketed.” Whaaaa? Maybe they could hire some more help in those nursing homes? Any policy will have issues when rolling out on a larger scale, but it seems silly to say that therefore it’s not a good idea to evaluate based on what evidence is available. Then he refers to “This evidence ratchet, in which findings can promote but not undermine a policy, is common.” This makes no sense to me, given that anything can be a policy. The $15 minimum wage is a policy. So is the $5 minimum wage, or for that matter the $0 minimum wage. High taxes on the rich is a policy, low taxes on the rich is a policy, etc.

Also this: “Grappling with such questions is frustrating and unsettling, as the policymaking process should be. It encourages humility and demands that the case for government action clear a high bar.” This may be Cass’s personal view, but it has nothing to do with evidence. He’s basically saying that if the evidence isn’t clear, we should make decisions based on his personal preference for less government spending, which I think means lower taxes on the rich. One could just as well say the opposite: “It encourages humility and demands that the riches of our country be shared more equally.” Or, to give it a different spin: “It encourages humility and demands that we live by the Islamic principles that have stood the test of time.” Or whatever. When evidence is weak, you have to respect uncertainty; it should not be treated as a rationale for sneaking in your own policy preferences as a default.

But I hate to end it there. Overall I liked Cass’s article, and we should be able to get value from it, subtracting the political slant which muddles his legitimate points. The key point, which Cass makes well, is that there is no magic to evidence-based decision making: You can do a controlled experiment and still learn nothing useful. The challenge is where to go next. I do think evidence is important, and I think that, looking forward, our empirical studies of policies should be as realistic as possible, close to the ground, as it were. Easier said than done, perhaps, but we need to do our best, and I think that critiques such as Cass’s are helpful.

6 Comments

  1. David Pittelli says:

    Great post, about a great article, IMHO.

    I think Cass’s point would have been better expressed had he used the word “program” instead of “policy” in “This evidence ratchet, in which findings can promote but not undermine a policy, is common.”

    Also, in Denmark there was already full employment of nurses, so the “unintended consequence of this parental-leave program was that the nursing workforce suddenly shrank by 12 percent.” It “reduced the number of nurses by 1,200 in nursing homes persistently and that increased the number of moralities by about 1,700 per year” (Freakonomics). Of course, in the long run they could train more nurses to make up for their increased rate of parental leave, but to paraphrase Keynes, in the long run many people were all dead.

    http://freakonomics.com/podcast/nurses-to-the-rescue/

  2. Keith O'Rourke says:

    A couple things here seemed related to on of my past posts https://andrewgelman.com/2017/09/27/value-set-act-represent-possibly-act-upon-aesthetics-ethics-logic/ – OK someday someone will make a comment there :-(

    These excerpts “Determining when the state should take property from one group and give it to another is a philosophical and cultural question as much as an economic one, so it is impossible to judge scientifically.” and “In the case of a universal basic income, assessment should begin from a philosophical inquiry into the proper role of the state and its relationship to the development of healthy families and communities.”

    Might be informed by Peirce’s thinking that you first need to decide what you value above all, second how one should deliberately act to best obtain what you value and third how you should best represent what you plan to act upon prior to acting in the world. The “how one should best represent” to profitably advance inquiry being logic. So Aesthetics -> Ethics -> Logic.

    In this thinking, the design and analysis of experiments falls into the last step “to profitably advance inquiry”. When I was reading the paper, I kept thinking what to value and how to act were being mixed up in the design and analysis of experiments. That is, if someone values quality and believes it is permissible to act in ways that re-distribute income they _should_ design and analyse experiments to make that work well.

    Alternatively, if they value individualism and believe it is not permissible to act in ways that re-distribute income they _should_ design and analyse experiments to make that work well. That does line up with Cass’ comment that “good evidence can inform tactical choices about program design” [which was preceded by] “[given] policymakers have already decided that government should act and on the form that action should take”.

    And this is hard not to at least be amused by “With their enthusiasm for EBP, policymakers are running headlong into a collapsing structure, pushing past inhabitants [Social scientists] fleeing in the other direction.”

    • Keith O'Rourke says:

      For “quality” read “equality”.

    • “Peirce’s thinking that you first need to decide what you value above all”

      I try to make this point a lot when discussing politics with friends and acquaintances. Too often people *identify with a particular policy* rather than a particular *goal or value*.

      So essentially something like “all right thinking people will agree we should raise the minimum wage to $15/hr” but in fact, they can’t possibly actually believe that, if you showed clear and concise evidence that doing so increased poverty and deaths among vulnerable populations for example, but they might believe something more like “all right thinking people will agree that no-one should live in abject poverty in a the modern day US” and this is a value *not a policy*

      but note, there are many ways to get to “no one lives in abject poverty” and it’s perfectly reasonable to agree with that value and *still think the $15 minimum wage is a terrible idea that actually will ultimately lead to more poverty or help less than other much better alternatives* (which I do). So, in your example of a Universal Basic Income, this policy has in my opinion a MUCH BETTER chance of achieving the goal/value of “no abject poverty” and has other very good properties as well, such as reducing the overhead of providing welfare for people, and letting people allocate their resources using the knowledge they have about their own situation, and reducing the time and effort spent on compliance with means-testing laws, and increasing the number of hours available and the incentive to do productive work (provided the UBI amount is tuned well).

      Yet, if you say “well, I think the $15 minimum wage is likely to be a terrible policy and I don’t support it at all” then… of course you’re an evil “not right thinking person”.

      sigh

      In general if you want to make good policy, it’s *ABOVE ALL ELSE* important that you establish what the value are that it’s trying to promote, and then evaluate the policy possibilities based on *how well it promotes those underlying values*.

      Unfortunately, too often, the value that is being promoted by policy makers is self-serving “get votes for my party, get reelected, block my opponents” etc and so it really doesn’t matter what the policy is, if your side supports it, and it gets through, the *true* goal of the policy maker has been achieved.

  3. D Kane says:

    He’s basically saying that if the evidence isn’t clear, we should make decisions based on his personal preference for less government spending, which I think means lower taxes on the rich.

    No. You are misreading Cass, who is a libertarian. His main point is that he wants the government to have less power, less ability to force people, under thread of imprisonment, to buy heath insurance or pay a minimum wage or offer employees family leave or not engage in prostitution. He thinks that much of the “evidence based” arguments used to justify these laws are bunk. Give him his libertarian minimal state, and he would not complain (much) about big taxes on the wealthy which are used to fund cash transfers.

    • Dean Eckles says:

      I think Andrew is implicitly arguing against the action vs. inaction / doing vs. allowing distinction as applied to government. Such a distinction might be regarded as motivating a mild libertarianism. In its absence, one could instead see this just as a personal preference for a policy (less government intervention).

Leave a Reply