My article “Experimental reasoning in social science” begins as follows:
As a statistician, I was trained to think of randomized experimentation as representing the gold standard of knowledge in the social sciences, and, despite having seen occasional arguments to the contrary, I still hold that view, expressed pithily by Box, Hunter, and Hunter (1978) that “To find out what happens when you change something, it is necessary to change it.”
At the same time, in my capacity as a social scientist, I’ve published many applied research papers, almost none of which have used experimental data.
In the present article, I’ll address the following questions:
1. Why do I agree with the consensus characterization of randomized experimentation as a gold standard?
2. Given point 1 above, why does almost all my research use observational data?
In confronting these issues, we must consider some general issues in the strategy of social science research. We also take from the psychology methods literature a more nuanced perspective that considers several different aspects of research design and goes beyond the simple division into randomized experiments, observational studies, and formal theory.
It’s a chapter in the book, “Field Experiments and Their Critics,” edited by Dawn Teele and based on a symposium at Yale a few years ago featuring Don Green, Alan Gerber, Abhijit Banerjee, Esther Duflo, and several other political scientists and economists.