Skip to content
 

When do stories work, Process tracing, and Connections between qualitative and quantitative research

belles_histoires

Jonathan Stray writes:

I read your “when do stories work” paper (with Thomas Basbøll) with interest—as a journalist stories are of course central to my field. I wondered if you had encountered the “process tracing” literature in political science? It attempts to make sense of stories as “case studies” and there’s a nice logic of selection and falsification that has grown up around this.

This article by David Collier is a good overview of process tracing, with a neat typology of story-based theory tests.

Besides being a good paper generally, section 6 of this paper by James Mahoney and Gary Goertz discusses why you want non-random case/story selection in certain types of qualitative research.

This paper by Jack Levy is another typology of the types and uses of case studies/stories.

I had not heard about process tracing, and I’ll have to take a look at these papers. I’m very interested in the connections between quantitative and qualitative research. Indeed, one of my themes when criticizing recent research boondoggles such as power pose and himmicanes has been the weakness of the connections between the qualitative and quantitative aspects of the work. And recently I got a taste of this criticism myself when I was presenting some of our findings regarding social penumbras: a psychologist in the audience pointed out that one reason our results were so weak was because there was only a very weak link between qualitative theories of changes in political attitudes, and the particular quantitative measures we were using. In short, I was doing what I often criticize in others, which was to gather data using a crude measuring instrument and then just hope for some results. We did find some things—I still think the penumbra work has been a successful research project—but we could’ve done much better, I’m sure, had we better tied qualitative to quantitative ideas.

13 Comments

  1. Ben Goodrich says:

    You were thanked for comments on Macartan Humphreys and Alan Jacobs’ paper on integrating things like process tracing into quantitative research. I can only assume your comment was to use Stan because they did

    http://www.macartan.nyc/projects/integrated-inferences/

  2. NatashaRostova says:

    I always think of ‘qualitative data’ as humans attempt to generate prior distributions based on our neural/linguistic model of the world, which is frequently more accurate in aggregate, but with insane human-to-human variance and sensitivity to initial conditions.

    I think that’s why psych research sucks so badly. Because all the researchers first create a ‘qualitative prior’ by using their brain to simulate another brain experiencing a cognitive bias. Often they (rightly or wrongly) place massive prior power on this, and then do the ‘formal stats’ as a formality.

    • Keith O'Rourke says:

      Natasha:

      Previously, I construed the main value of ‘qualitative data’ as being for abduction (e.g. coming up with better conjectures, representations, priors, likelihoods etc.) rather than being material for which rules could be discerned and applied to revise representations (i.e. deduce a posterior).

      This other role seems to be the main focus of Macartan Humphreys and I think it is interesting – though I still think its primary role should be for abduction.

    • Martha (Smith) says:

      Natasha: Nice description.

  3. Keith O'Rourke says:

    So whats new?

    Represent one’s reality of interest (settle on prior and likelihood), follow the rules (to get the posterior) assess the convincing-ness and settle on or revise and redo. When settled on, make sense of the posterior (how should it reshape your view of reality and how to act in it.)

    OK, now you need explicit rules for defining and calling the presence of a clue (qualitative observation) and also assess the probability that such would be present for various realities one is unsure between. These probabilities will be more challenging as there are no default distributions as with quantitative observations. Here folks might worry about checking these likelihoods (probability that such would be present for various realities) more that priors for clue occurring.

    Nice catch, thanks for posting and Ben for poster the paper I found more direct to my interests.

    Now, one of my old qualitative mentors, Paul Perron http://french.utoronto.ca/profiles/48 argued that few qualitative researchers had the skills and temperament to actually follow rules (rigorously) in their work. So I am guessing this is not that popular among qualitative researchers?

  4. An interesting case in point: Proponents of the Implicit Association Test hypothesize that people who do not consider themselves racist may still *act* racist–and that this tendency can be measured in an association test. The test itself has numerous problems, but because of its hype and proliferation, people (and organizations) have come to believe in its results.

    Jesse Singal wrote a terrific analysis of the issue: http://nymag.com/scienceofus/2017/01/psychologys-racism-measuring-tool-isnt-up-to-the-job.html

  5. PT says:

    Diana – It’s not just the race version of the IAT that has questionable validity as a measure of anything vaguely similar to an attitude (or a preference). Consider the fact that overweight individuals “prefer” healthy food when their attitudes are measured by the IAT and depressed individuals show positive associations with the self (refs below). What amazes me is how the IAT seems unstoppable – if you look at the critical articles they receive far fewer citations that non-critical articles published at approximately the same time.

    De Raedt, R., Schacht, R., Franck, E., & De Houwer, J. (2006). Self-esteem and depression revisited: Implicit positive self-esteem in depressed patients?. Behaviour Research and Therapy, 44(7), 1017-1028.

    Roefs, A., & Jansen, A. (2002). Implicit and explicit attitudes toward high-fat foods in obesity. Journal of abnormal psychology, 111(3), 517.

    • Thank you for these citations, PT. I have made note of them.

      I wonder why these IATs get so much favorable press and hype. It may have to do with a pseudoscience of self; it’s tempting and handy to “learn something new about yourself” with just a few clicks. You can click away and then nod in agreement with the prefab inferences.

  6. HN says:

    Check out the political scientist Jason Seawright for more on process tracing and mixed methods. Mixes register data, surveys, case studies and RCTs…

Leave a Reply