Authors of AJPS paper find that the signs on their coefficients were reversed. But they don’t care: in their words, “None of our papers actually give a damn about whether it’s plus or minus.” All right, then!

Avi Adler writes:

I hit you up on twitter, and you probably saw this already, but you may enjoy this.

I’m not actually on twitter but I do read email, so I followed the link and read this post by Steven Hayward:

EPIC CORRECTION OF THE DECADE

Hoo-wee, the New York Times will really have to extend itself to top the boner and mother-of-all-corrections at the American Journal of Political Science. This is the journal that published a finding much beloved of liberals a few years back that purported to find scientific evidence that conservatives are more likely to exhibit traits associated with psychoticism, such as authoritarianism and tough-mindedness, and that the supposed “authoritarian” personality of conservatives might even have a genetic basis (and therefore be treatable someday?). Settle in with a cup or glass of your favorite beverage, and get ready to enjoy one of the most epic academic face plants ever.

The original article was called “Correlation not causation: the relationship between personality traits and political ideologies,” and was written by three academics at Virginia Commonwealth University. . . .

I had no recollection of this study but I forget lots of things so I decided to google my name and the name of the paper’s first author, and lo! this is what I found, a news article by Shannon Palus:

Researchers have fixed a number of papers after mistakenly reporting that people who hold conservative political beliefs are more likely to exhibit traits associated with psychoticism, such as authoritarianism and tough-mindedness. . . .

To help us make sense of the analysis, we turned to Andrew Gelman, a statistician at Columbia not involved with the work, to explain the AJPS paper to us. He said:

I don’t find this paper at all convincing, indeed I’m surprised it was accepted for publication by a leading political science journal. The causal analysis doesn’t make any sense to me, and some of the things they do are just bizarre, like declaring that correlations are “large enough for further consideration” if they are more than 0.2 for both sexes. Where does that come from? The whole thing is a mess.

He added:

It’s hard for me to care about reported effect sizes here…If the underlying analysis doesn’t make sense, who cares about the reported effect sizes?

Hey, now I remember! Oddly enough, Palus quotes one of the authors of the original paper as saying,

We only cared about the magnitude of the relationship and the source of it . . . None of our papers actually give a damn about whether it’s plus or minus.

How you can realistically expect to learn about the magnitude of a relationship and the source if it, without knowing about its sign, that one baffles me. And the author of the paper then adds to the confusion by saying,

[T]he correlations are spurious, so the direction or even magnitude is not suitable to elaborate on at all- that’s the point of all our papers and the general findings.

Now I’m even more puzzled as to how this paper got published in AJPS, which is a serious political science journal. We’re not talking Psychological Science or PPNAS here. I suspect the AJPS got snowed by all the talk of genetics. Social scientists can be such suckers sometimes!

Looking at the correction note by Brad Verhulst, Lindon Eaves, and Peter Hatemi, I see this:

Since these personality traits and their antecedents have been previously found to both positively and negatively predict liberalism, or not at all, the descriptive analyses did not appear abnormal to the authors, editors, reviewers or the general academy.

Wha??? OK, so you’re saying the data are all noise so who cares? You’ve convinced me not to care, that’s for sure!

Getting back to the original link above: I disagree with Steven Hayward’s claim that this is an “epic correction.” Embarrassing for sure, but given that it’s hard to take the original finding seriously, it’s hard for me to get very excited about the reversal either.

Good to see the error caught, in any case. I’m not at all kidding when I say that I expect more from AJPS than from Psych Sci or PPNAS.

P.S. I did some web search and noticed that Hatemi was also a coauthor of a silly paper about the politics of smell; see here for my skeptical take on that one.

8 thoughts on “Authors of AJPS paper find that the signs on their coefficients were reversed. But they don’t care: in their words, “None of our papers actually give a damn about whether it’s plus or minus.” All right, then!

  1. I find it irritating that un-ascribed variation in measurements is so often attributed to a “genetic basis.” It strikes me as such a cheap and easy fallback, especially since in many instances no one will ever actually evaluate that supposed genetic basis. What’s wrong with saying “I don’t know why there’s so much variation”?

  2. Not surprisingly, this type of research is very subject to context effects.

    “Meta-analyses published in 2003 and 2010 of dozens of studies using different measures revealed a consensus on “the rigidity of the right” – that is, people who hold more right-wing views tend to be more close-minded. Case closed? Or should we be open to other perspectives, such as the one offered in a new article published recently in Political Psychology. Produced by a research team lead by Lucian Conway of the University of Montana, it shows how classic measures of close-mindedness may be bedevilled by topic bias. When the subject matter is switched out, it’s the left who’re locked-in.”

    http://bps-research-digest.blogspot.com/2015/11/on-some-issues-liberals-are-more.html

  3. Do they realize they’ve said they’ve discovered there’s noise? My impression is there’s a general lack of understanding of the underlying meanings of abstractions such as “the magnitude of the cross-twin cross-trait covariation, and second moment of data” so they might think finding noise is actually finding something.

    • @Jonathan

      I’m having trouble understanding your comment. Do you perhaps mean

      … abstractions such as “the magnitude of the cross-twin cross-trait covariation” and “second moment of data” … ?

Leave a Reply

Your email address will not be published. Required fields are marked *