Skip to content

Experimental reasoning in social science

As a statistician, I was trained to think of randomized experimentation as representing the gold standard of knowledge in the social sciences, and, despite having seen occasional arguments to the contrary, I still hold that view, expressed pithily by Box, Hunter, and Hunter (1978) that “To find out what happens when you change something, it is necessary to change it.”

At the same time, in my capacity as a social scientist, I’ve published many applied research papers, almost none of which have used experimental data.

In the present article, I’ll address the following questions:

1. Why do I agree with the consensus characterization of randomized experimentation as a gold standard?

2. Given point 1 above, why does almost all my research use observational data?

In confronting these issues, we must consider some general issues in the strategy of social science research. We also take from the psychology methods literature a more nuanced perspective that considers several different aspects of research design and goes beyond the simple division into randomized experiments, observational studies, and formal theory.

Here’s the full article, which is appearing in a volume, Field Experiments and Their Critics, edited by Dawn Teele.

It was fun to write a whole article on causal inference in social science without duplicating the article that I’d recently written for the American Journal of Sociology. But I think it came out pretty well. Actually, it contains the material for several blog entries had I chosen to present it that way. In any case, I think points 1 and 2 are central to any consideration of causal inference in applied statistics.


  1. I would love to use experimental data, but my answer to (2) is: In astronomy, we get the photons we get, and no more, no matter what! That makes astronomy oddly like many of the social sciences.

  2. Bill Jefferys says:

    A big difference between astronomy and social sciences is that we actually have a robust theory that in principle can accurately model the phenomena we are studying…that is to say, physics. It's true that often the physics has to be approximated in our models, making them semi-empirical, but it's my impression that in the social sciences the models are entirely empirical.