Maybe this paper is a parody, maybe it’s a semibluff

Peter DeScioli writes:

I was wondering if you saw this paper about people reading Harry Potter and then disliking Trump, attached. It seems to fit the shark attack genre.

In this case, the issue seems to be judging causation from multiple regression with observational data, assuming that control variables are enough to narrow down to causality (or that it’s up to a critic to find the confounds). It speaks to a bigger issue about how researchers interpret multiple regression in causal terms.

Any thoughts on this, or obvious/good references critiquing causal interpretations of multiple regression? (like to assign to my PhD students)

My reply: Hi, yes, I saw this paper months ago. I suspected it was a parody but someone told me that it was actually supposed to be serious. I still think it’s a kind of half-parody, it’s what social scientists might call a “fun” result, and it’s published in a non-serious journal, so I doubt the author takes it completely seriously. Kinda like this: you find an interesting pattern in data, it’s probably no big deal, but who knows, so get it out there and people can make of it what they will.

Twenty years ago, social scientists could do this and it would be no problem; nowadays with all this stuff on shark attacks, college football, power pose, contagion of obesity, etc., it seems that people have more difficult putting such speculations into perspective: any damn data pattern they see, they want to insist it’s a big deal, from data analysis to publication to Ted talk and NPR. In some sense this Harry Potter paper is a throwback and it would probably be best to interpret it the way we’d have taken it twenty or thirty years ago.

It’s impossible for me to tell whether the author, Diana Mutz, is writing this paper as a parody. Intonation is notoriously difficult to convey in typed speech. It’s a funny thing: if the paper’s not a parody and I say it is, then I’m kinda being insulting. But if the paper is a parody and I take it seriously, then I’m not getting the joke. So there’s no safe interpretation here! (I could ask Mutz directly but that’s not much of a general solution; I’d rather think of a published article and its implications as standing on their own and not requiring typically unobtainable “meta-data” on authorial intentions.)

DeScioli responded:

It does have some whimsical passages so maybe it is half-parody.

And I continued:

Yeah, there’s this genre of research which is not entirely serious but not entirely a joke, kinda what in poker we’d call a semibluff. Back in the good old days before Gladwell, PPNAS, NPR hype, etc., it was reasonable for researchers to try out some of these ideas, they were long shots but had some appeal as part of the mix of science. For awhile, though, it was seeming like this sort of open-ended-speculation-backed-by-statistically-significant-p-values had become most of social science, and this has reduced all of our patience for this sort of thing. Which is kinda too bad. Another example is that observation that several recent presidents were left-handed. It seems like it should be possible to point to such data patterns, and even run some statistics on them, without making large claims.

DeScioli followed up:

Seems it could still be as fun and interesting to look for these types of correlations without claiming causality. I was just surprised to see the paper double down on the causal interpretation with the argument that the analysis controlled for everything they could think of. (My assumption is that observational data has countless confounding correlations that no one could think of.) I don’t think this paper is worse in over-interpreting than many others I’ve seen. It’s just easier to notice because of the whimsical topic.

What’s the lesson for avoiding this for a more serious-sounding theory? I typically restrict causal judgments to experimental manipulations. But maybe that is too restrictive? The only other thing I can think of is if a researcher knew so much about their subject that they could boil down the possible causes to a handful. Then maybe multiple regression with controls could help sort between them. If so, the issue with Harry Potter is that it’s one of millions of similar cultural influences that are all hopelessly tangled and so can’t be untangled with observational methods.

18 thoughts on “Maybe this paper is a parody, maybe it’s a semibluff

  1. Anyone who has seriously studied causal inference from observational data knows that control variables are not enough to provide identification (in laymen’s terms, establishing cause and effect). The author has found a link between reading Harry Potter and opposing Donald Trump. The link remains after including controls. But this does not establish cause and effect. More likely, people who would support Trump are less likely to read HP (witchcraft and whatnot). I’d guess you have textbook 101 reverse causality here.

    But Prof Gelman is right: this looks like a semi-serious paper, not unlike what some econ journals used to publish as Miscellany: correctly done (in principle) analysis of funny topics, like the optimal stopping problem solution to Elaine’s (Seinfled) “spongeworthy” dilemma.

    • Jack:

      Yes, one unfortunate effect of the flood of NPR/Gladwell-endorsed junk science seems to have been to remove some of the space for intriguing speculations.

      Or maybe we just know better now. For example, those papers on air rage and himmicanes and ovulation-and-clothing and ovulation-and-voting could be taken as intriguing speculations, nothing more, except:

      1. They appeared in top journals: Psychological Science and the Prestigious Publications of the National Academy of Sciences, and as such were not labeled as nothing more than intriguing speculation;

      2. A careful look at those papers revealed such problems in the data and analysis that there wasn’t much left that was intriguing. To put it another way, those papers (and others like them) offered very little value beyond hypothetical versions of those papers that were 100% theory and speculation with no data at all.

      The dentist-named-Dennis paper was a better modern-day example of intriguing data-based speculation, in that (a) the speculation itself was kinda cool (much more so than, say, a claim of an interaction between relationship status and time of the month in determining vote preference), and (b) the flaws in the data analysis in the dentist-named-Dennis paper were subtle and it was not so clear that it could be dismissed out of hand.

      • Maybe what we need is to revive the Miscellany tradition as long as it is clearly labelled as such: intriguing speculations that do not have clear answers and that warrant further study (or not!). I like speculation, as long as it is clear that’s what it is. If someone wants to point out a link between air rage and whatever, that’s cool, but be honest about this finding being speculative.

        It’s as if detectives on a murder case labelled the first Person of Interest they find the suspect and then had the DA prosecute him… not good…

      • In the past few years I’ve noticed a lot of papers like this that look like they were engineered to provide soundbites for political partisans to cite to.

        For instance, this Harry Potter paper clearly signals which way the results are intended to come out. It could be subtitled “Another cite for the proposition that Trump is evil.”

        Another paper in this mold was one that found there was “no evidence” that murders increased after Ferguson. (Pyrooz, Decker, Wolfe, and Shjarback, “Was There a Ferguson Effect on Crime Rates in Large U.S. Cities?” https://www.researchgate.net/publication/292982674_Was_there_a_Ferguson_Effect_on_crime_rates_in_large_US_cities )

        Another paper clearly wanted to deliver a cite for the soundbite that “diversity is good”. It claimed to find that diverse student groups performed better than homogenous groups in calculating stock prices. (Quote from the NYT article: “Our research provides such evidence. Diversity improves the way people think.”) https://www.nytimes.com/2015/12/09/opinion/diversity-makes-you-brighter.html?action=click&pgtype=Homepage&clickSource=story-heading&module=opinion-c-col-left-region&region=opinion-c-col-left-region&WT.nav=opinion-c-col-left-region)

        Andrew has highlighted others: the paper that found Trump supporters reacted more negatively to a picture of a black man, and the paper that found that “fake news” significantly helped Trump in the election.

        From the first paragraph you know what these papers are going to find, and you often get the impression that they were engineered from the ground up to find that result.

        Was it always thus? I don’t know.

  2. “Can Harry Potter defeat Donald Trump? Is his orange wig actually a horcrux that, if captured, could weaken the strength of his electoral base? Just as He-Who-Must-Not-Be-Named gains power from having others refer to him, is Trump’s appeal likewise a function of nonstop media fascination and repetition? These questions remain to be answered.”

    Now I have no idea what half of that means, but I am pretty sure it is the academic equivalent of this.

    https://www.youtube.com/watch?v=aPRB4xnYT6A

    I mean, sure, it is a fake ad. But at the same time, you know Swisher paid for it, and it came out of the advertising budget*. So if a fake ad can be a real ad, a fake paper can be a real paper.

    Plus we don’t have to cite Rawls or Kant anymore now that we have “If half- bloods, werewolves and others should be treated with respect and fairness… so too should all human beings.” I’m tired of those guys sucking up all the citations anyway.

    *I actually have no knowledge that this is true.

  3. This paper is almost certainly tongue-in-cheek — Diana Mutz has been a big proponent of experiments in political science (http://press.princeton.edu/titles/9620.html). The Harry Potter-and-Trump paper is similar to another paper she wrote in 2010 on the role of dogs in the 2008 election (https://www.sas.upenn.edu/polisci/sites/www.sas.upenn.edu.polisci/files/Mutz_dogs.pdf) — it’s just supposed to be fun, not to be taken too seriously.

  4. A first impression after reading this study is that the effect sizes found imply a very large effect from the totality of popular culture. The paper finds a 1-2 point effect on the “feeling thermometer” toward trump from reading a single Harry Potter book. (The thermometer is scaled from 0-100.)

    While this is a small effect from a single book, it implies an enormous effect from the totality of all popular culture. Almost all popular culture has a cultural/political viewpoint similar to the Harry Potter books. So, if a single popular book has a 1-2 point effect, all of popular culture should have at least 10 times that effect and perhaps 50 times. (Source: I pulled these out of my ear, your ear contents may vary.) This implies a 10-100 point negative effect on perceptions of Trump from the totality of all popular culture.

    This can cut two ways. You can conclude from this that the effects this paper finds are too large to be believable. (A recurring theme of Andrew’s.) Or you can conclude that our political landscape is hugely distorted by the effects of the alleged liberal slant in popular culture. (This harks back to a previous post by Andrew about the effects of media bias on voters’ political positions, where Andrew found large effects to not be credible.)

    It’s hard to tell which is true (or if both are true in weakened form). The author of this study clearly wanted very much to find the result she found, so it is highly likely the study is fatally flawed and she published it only because it gave a “good” result. OTOH, I find it easy to believe that popular culture has a profound influence. For instance, I personally was deeply affected by reading To Kill a Mockingbird.

    • The curious thing is that JK Rowling’s books would appear to be slightly to the right of “Tom Brown’s Schooldays” and Kipling’s “Stalky and Co.” ideologically: traditional boarding school is worshipped, wizardry is genetically determined (following same dominant-recessive pattern as blue eyes), etc.

      So why has Harry Potter become a touchstone of liberalism?

      • Steve:

        I’ve only read a couple of pages of Harry Potter so I’m just guessing here, but I don’t have the impression that Harry Potter is a touchstone of liberalism or that it has any strong political message. Here’s what I see:
        – The Harry Potter books are popular and have been talked about a lot, and some of this talk happens to be political.
        – Some politically conservative religious figures attacked Harry Potter for promoting Satanism.
        – The author of the books is politically liberal.
        The actual contents of the books, from what I’ve heard of them, seem general enough that they could be taken to support liberal or conservative political positions.

        • A possibly significant factor not mentioned in that list is that the biggest Potter fans, apart from the notably apolitical primary school demographic, are nerdy high school kids who are into the arts, and these kids are extremely liberal. Perhaps a bit like how, while Wagner himself was anti-Semitic, though not to an especially unusual degree for his time, and while his operas have some political content, his music is associated with hatred mostly because of its most notorious enthusiasts.

        • The paper’s attempts to see the HP books as anti-Trump come off as pretty forced.

          For instance:

          Readers have long noted similarities between Voldemort and Hitler, and Rowling herself has acknowledged these parallels. Likewise, Trump’s fascist
          leanings have not gone unnoticed (see Chotiner 2016 ; Frank 2016 ; Kagan 2016 ). Consistent with authoritarian principles, he promises order as well as dominance over all potential threats. As does Voldemort, Trump portrays himself as a strongman who can bend others to his will, be they the Chinese government or
          terrorists. His open admiration for Vladmir Putin—“at least he’s a leader”—caused Joe Scarborough to point out that many of Putin’s opponents end up dead (Gass 2015 ).

          And this:

          As Gerson ( 2007 ) and others have noted, “Tolerance is one of the
          main themes of the Harry Potter books.” …

          The ongoing battle between good, as personified by Harry and his friends, and evil, as personified by Lord Voldemort, is at root about the importance of group purity. Lord Voldemort supports the eradication of mixed-blood wizards (so called “half-bloods”) as well as muggles (non-wizards), mudbloods (muggle-born wizards)
          and squibs (wizard-born non-wizards). During Voldemort’s time in power, people live in constant fear that they will be killed, either for having impure blood or for sympathizing with those who do. In Harry’s world, “ werewolves were subject to discrimination as if they had AIDS” (Gerson 2007 ).

          In comparison, Donald Trump has called for a temporary moratorium on Muslim immigration.1 Further, Trump falsely claimed that American Muslims in New Jersey celebrated the attacks on September 11, 2001 (Kessler 2015 ), and has suggested establishing a national database to register all Muslims (Hillyard 2015 ).
          Muslims are not alone in incurring Trump’s wrath. Trump has on several occasions stereotyped and insulted women, those with disabilities, and Asians, who according to his statement at an event in Iowa, dispense with introductory pleasantries to say, “We want deal!” He also has off ended Mexicans and immigrants by describing those crossing the United States’ southern border as “rapists.” Trump attacked federal Judge Gonzalo Curiel , the judge presiding over a lawsuit against Trump University, for having Mexican ancestry. So common are his degrading comments
          about outgroups that websites have become devoted to keeping long lists of them. 2

          These forced comparisons would be strong evidence that the article is tongue-in-cheek if this type of rhetoric were not so common.

        • Terry:

          Yes, I think those forced comparisons are indeed a joke. And, yes, one problem is that this sort of thing is said in earnest often enough that the parody can look like the real thing.

  5. I don’t understand the semi-bluff analogy. The semi-bluff is a strategic move in poker. It’s called “semi,” because you have some combination of pot equity (i.e., a draw is made by the river) and fold equity. This is as opposed to a standard bluff where there is no pot equity. The semi-bluff is a deliberate strategy based on board texture, position, etc.

    • Nn:

      My knowledge of poker is not deep and I could be using the term wrong. I’d understood a semi-bluff to be when you raise on a hand that will probably lose but has a chance of turning out best, as oppose to a pure bluff where you got nothing. In both cases, the “bluff” part is that you’re trying to get people with better hands to fold, but in the “semi” situation, you have some insurance in that if your opponent stays in, you could still draw out and win. For example, maybe you could semi-bluff with two low pairs.

      In the example of the above post, I was saying the paper in question is a bit of a bluff—do you really expect that reading Harry Potter would have such effects?—but maybe a semi-bluff in the sense that it could be real.

  6. Pete also asked about “obvious/good references critiquing causal interpretations of multiple regression.” Suggestions? Especially articles that critique the use of control variables?

Leave a Reply

Your email address will not be published. Required fields are marked *