Skip to content
 

I’ll use this line in my talk this Wednesday at the Society for Research on Educational Effectivness

I had a conversation with a policy analyst about the design of studies for program evaluation—the post is scheduled to appear in a few months—and he expressed some frustration:

The idea of evidence based policy has put a gun to our heads as researchers to give binary responses with absolute confidence to a question that is not binary, or we we are not realistically able to be confident in the answer. It puts money and jobs at risk instead of allowing for thoughtful academic discussion.

As my correspondent puts it, “It isn’t just about tenure or some research grants. 100’s of millions are being put at risk over arguments with evidence based policy.”

Yup. We care about getting the statistics right because we care about getting the science right.

7 Comments

  1. Dan Simpson says:

    I mean I don’t disagree with you Andrew but you’ll be fine. As someone who is doing the tenure-track game (even if I’ve been employed waaaaay beyond the tenure bar before), I will always care about still having a job in k years before I care about anything else. Then I care about society. Then I care about science. Eventually that first one will leave but I will not begrudge a person the first one.

    I personally think that tenure is the worst idea ever. But I’m choosing to participate in the system so I am explicitly supporting those who are making bad science decision in the name of tenure. And when I have tenure I will be implicitly supporting it by participating in the system.

    I hope that I’m not doing bad science. And I don’t think I’m doing bad science. But it’s by a combination of luck, skill, and collaborators if I never look back on something and think it’s bad science. I will have beat the system.

  2. Keith O'Rourke says:

    Part of the problem here is the statistician down the hall or the colleague that can fit statistical models agreeing to “authority” away the insurmountable uncertainty.

    It also raises the uncomfortable question about how honest and clear one _should_ be with the uncertainties discussed here https://andrewgelman.com/2018/01/23/better-enable-others-avoid-misled-trying-learn-observations-promise-not-transparent-open-sincere-honest/

    Fully agree its about getting the science right – but also maybe when (later?) and how (staying in the game?).

Leave a Reply