Thomas Basbøll pointed me to a discussion on the orgtheory blog in which Jerry Davis, the editor of a journal of business management argued that it is difficult for academic researchers to communicate with the public because “the public prefers Cheetos to a healthy salad” and when serious papers are discussed on the internet, “everyone is a methodologist.” The discussion heated up when an actual methodologist, Steve Morgan, joined in to argue that the salad in question was not so healthy and that the much-derided internet commenters made some valuable points. The final twist was that one of the orgtheory bloggers deleted a comment and then closed the thread entirely when the discussion got too conflictual.
In a few days I’ll return to the meta-topic of the discussion, but right now I want to focus on one thing Davis wrote, a particular statement that illustrates to me the gap between the rhetorical and the literal, the way in which a statement can sound good but make no sense.
Balazs Kovacs and Amanda Sharkey have a really nice article in the March issue of ASQ [Administrative Science Quarterly] . . . The paper starts with the intriguing observation . . . The authors do an extremely sophisticated and meticulous job . . . As is traditional at ASQ, the authors faced smart and skeptical reviewers who put them through the wringer, and a harsh and generally negative editor (me). This is a really good paper, and you should read it immediately to find out whodunit.
OK. Let’s look at this paragraph carefully. On a rhetorical level, it all holds together: it’s a bunch of strongly positive statements about the Kovacs and Sharkey paper, along with a judo-style endorsement: in praising a paper that came out in his own journal, he’s saying that he (Davis) was “harsh” and “generally negative”—but even so he thinks the paper is great.
But if you try to take the backhanded endorsement literally, it falls apart. If Davis was generally negative about the paper, why did he publish it at all? Presumably not under pressure from the “skeptical reviewers.” And, more to the point, what does it mean to be “harsh and generally negative” about a paper that is “really nice,” “extremely sophisticated,” and “really good”?
We all do this from time to time, so it’s not like I’m saying Davis is some sort of terrible person here. What happened, I’m guessing, is as follows. A paper was published in his journal and got positive publicity in the newspaper. This made him happy (as indeed it should; he works hard as a journal editor and it’s good to feel that your work is making an impact), but then he was unhappy to see many of the newspaper’s internet commenters criticizing the study, and he vented his annoyance on the orgtheory blog.
Davis hadn’t thought very hard about the paper in question (as explained above, I don’t think he was really being “harsh and generally negative” in his editorial role), and he just assumed that the article was high quality (after all, it was published in a top journal and included sophisticated methods such as a matched sample, “difference-in-difference regression,” and “several robustness checks”) and that the commenters didn’t know jack (in his words, “everyone is a methodologist”). For rhetorical reasons, he amped up the praise of this solid but unexciting article and described his own editing role as “harsh and generally negative” to heighten the contrast.
Sort of like how, in a letter of recommendation, I might write that student X is really good, does amazing work, and, “by the way, I don’t usually write this sort of thing. I’m a really demanding and critical advisor.” It sounds good. I pass around some compliments and then do the reader a favor and renormalize by emphasizing how usually negative I am.
In this case, the strategy backfired because many of the orgtheory commenters disagreed strongly with Davis, with some commenters objecting (rightly, I think) to his criticism of internet commenters and others pointing out issues with the Kovacs and Sharkey paper and writing things like “I think you overestimate the meticulous nature of peer review.” Again, I’m not saying that this paper was terrible, just pointing out the gap between incoherence of the praise that the journal editor is giving to it.
Nothing magic about publication
To bring this back to a subject discussed a lot in this space recently (for example, here), one problem I see is the perception of journal publication as a sort of plateau: the idea is that before a paper is published it is speculative but once it appears in a top journal, we are expected to take it seriously, with any criticism to be held to an extremely high standard.
It’s hard to get a paper published in a top journal—and, from an editor’s point of view, it’s hard to get media publicity—so, in either case, when you’ve achieved that goal, it’s natural to want to take a rest. Indeed, author Balazs Kovacs writes:
I have always been all for the transparency and truth and diversity of opinions that such an open discussion can bring out. I am still an advocate of that. But what I realized is that this can be really taxing on the authors. That is, if someone posts a comment / doubt about your paper, you as an author need to address that otherwise the last public record will be an unanswered doubt. . . . And that is not something I look forward to. I am a kind of person who gets tired of a project during the publication process (I guess I’m not alone!). The main reason that I love getting a paper published is that then I can close the process and move on to other new and exciting projects. The key is “moving on.” The fact that in such public debates of my previously published papers I’d need to go back to old stuff, essentially takes away the biggest satisfaction I derive from publishing a paper.
Wow! That’s just amazing. I mean, sure, I too get satisfaction out of having a project done, seeing the paper in reader’s hands, seeing the book on the shelf, grading the last final exam and saying goodbye to the semester—but “the biggest satisfaction”???
If your biggest satisfaction in a project is that it’s done and you get to “move on,” maybe you shouldn’t have done that project in the first place! For me, the biggest satisfaction of hiking in the mountains is that delicious moment when I get to take off the backpack and rest. Maybe that’s why I haven’t gone backpacking in 20 years.
Again, there seems to be this attitude that publication is a difficult challenge and once you’ve achieved it, the accomplishment shouldn’t be diminished in any way. You’ve sold it, as it were, to the journal or the press, and you’re not offering any refunds. Always be closing, then move on to the next lead.
What’s it all about?
The article in question, based on a counting of positive and negative reviews of 64 books, reports that “prizewinning books tend to attract more readers following the announcement of an award and that readers’ ratings of award-winning books tend to decline more precipitously following the announcement of an award relative to books that were named as finalists but did not win.” I assume that the prize motivates people who otherwise would not have read and commented on the book to read and comment on it, and that these marginal readers and commenters are less enthusiastic about the book, compared to people who would read and comment on it even in the absence of a prize. Or, as one of the much-derided anonymous internet commenters writes:
It doesn’t surprise me. You get it a lot on amazon where if a book has won an award or becomes famous it attracts a lot of negative reviews because people buy it because of the hype when it’s probably not their sort of book. Meanwhile less famous books by the same author get higher average reviews because they tend to be read only by people who are into that writer and maybe more on their wavelength. I’m a big fan of Roberto Bolaño for example and his two most famous books 2666 and Savage Detectives have some pretty negative reviews by people who bought them on the back of the hype and hated them because he’s simply not a writer for everyone. Meanwhile books like Nocturno de Chile (can’t remember English title sorry) or Distant Star get better averages because only Bolaño-philes are likely to read them.
Well put, but it’ll never appear in the Administrative Science Quarterly.
Neither of the two main findings of the Kovacs and Sharkey paper surprise me either, and Davis’s labeling of them as “intriguing” seems a bit of a stretch. I think what is happening, again, is that he’s getting carried away by rhetoric. Rather than writing in literal terms, he’s using words and phrases that sound good.
That said, it can be good to do a quantitative study even if only to confirm what makes sense. I say this as a person who, with collaborators, has published two papers on the advantage of incumbency in congressional elections, speaking of obvious topics.
Davis analogizes this research to a “healthy salad” and compares it to an unrelated paper “which found that people who walked down a hallway while texting on their phone walked slower, in a more stilted fashion, with shorter steps, and less straight than those who were not texting,” which he characterizes as “Cheetos.” Davis’s preferred paper is related to business management and the texting paper is related to public health, so I can see why he, as a business-school professor, would consider the former topic more important and worthy of serious coverage.
The horrible way in which publication and publicity are taken as a reason to close off discussion
And I better say this again: My point here is not to pick on Davis (who I’m sure is working very hard editing a journal, which is a form of service to the academic community) or Kovacs and Sharkey (whose paper seems reasonable enough, if unremarkable) but rather to emphasize the horrible (to me) way in which publication and publicity often seem to be taken, not as opportunities for further exploration, but as a reason to close off discussion and suppress the sharing of knowledge.
The idea that a claim, just because it’s published in a top journal, should go unquestioned, or that critics are “trolls,” or that anonymous commenters can’t be trusted (in comparison to anonymous journal reviewers, who are trusted even though their reports are forever hidden): that really bothers me. It seems contrary to the goals and practices of science and scholarship, and indeed contrary to the goals of publication itself.
I sort of understand the attitude on a professional level—once you’ve eaten the cake, you don’t want some critic going down your throat to pull it out of your stomach—but I don’t have to like it. And in some way it’s good to have this discussion with a paper that’s not particularly crappy, to emphasize that this is not just an issue with research that shouldn’t have been published at all; the concern also arises with solid, run-of-the-mill, another-brick-in-the-wall scholarship of the sort that we all would like to do, the sort of work that is central to the scientific enterprise, but which should not (I believe) be hyped and then held to be immune from criticism.