Literal vs. rhetorical

Thomas Basbøll pointed me to a discussion on the orgtheory blog in which Jerry Davis, the editor of a journal of business management argued that it is difficult for academic researchers to communicate with the public because “the public prefers Cheetos to a healthy salad” and when serious papers are discussed on the internet, “everyone is a methodologist.” The discussion heated up when an actual methodologist, Steve Morgan, joined in to argue that the salad in question was not so healthy and that the much-derided internet commenters made some valuable points. The final twist was that one of the orgtheory bloggers deleted a comment and then closed the thread entirely when the discussion got too conflictual.

In a few days I’ll return to the meta-topic of the discussion, but right now I want to focus on one thing Davis wrote, a particular statement that illustrates to me the gap between the rhetorical and the literal, the way in which a statement can sound good but make no sense.

Here’s Davis:

Balazs Kovacs and Amanda Sharkey have a really nice article in the March issue of ASQ [Administrative Science Quarterly] . . . The paper starts with the intriguing observation . . . The authors do an extremely sophisticated and meticulous job . . . As is traditional at ASQ, the authors faced smart and skeptical reviewers who put them through the wringer, and a harsh and generally negative editor (me). This is a really good paper, and you should read it immediately to find out whodunit.

OK. Let’s look at this paragraph carefully. On a rhetorical level, it all holds together: it’s a bunch of strongly positive statements about the Kovacs and Sharkey paper, along with a judo-style endorsement: in praising a paper that came out in his own journal, he’s saying that he (Davis) was “harsh” and “generally negative”—but even so he thinks the paper is great.

But if you try to take the backhanded endorsement literally, it falls apart. If Davis was generally negative about the paper, why did he publish it at all? Presumably not under pressure from the “skeptical reviewers.” And, more to the point, what does it mean to be “harsh and generally negative” about a paper that is “really nice,” “extremely sophisticated,” and “really good”?

We all do this from time to time, so it’s not like I’m saying Davis is some sort of terrible person here. What happened, I’m guessing, is as follows. A paper was published in his journal and got positive publicity in the newspaper. This made him happy (as indeed it should; he works hard as a journal editor and it’s good to feel that your work is making an impact), but then he was unhappy to see many of the newspaper’s internet commenters criticizing the study, and he vented his annoyance on the orgtheory blog.

Davis hadn’t thought very hard about the paper in question (as explained above, I don’t think he was really being “harsh and generally negative” in his editorial role), and he just assumed that the article was high quality (after all, it was published in a top journal and included sophisticated methods such as a matched sample, “difference-in-difference regression,” and “several robustness checks”) and that the commenters didn’t know jack (in his words, “everyone is a methodologist”). For rhetorical reasons, he amped up the praise of this solid but unexciting article and described his own editing role as “harsh and generally negative” to heighten the contrast.

Sort of like how, in a letter of recommendation, I might write that student X is really good, does amazing work, and, “by the way, I don’t usually write this sort of thing. I’m a really demanding and critical advisor.” It sounds good. I pass around some compliments and then do the reader a favor and renormalize by emphasizing how usually negative I am.

In this case, the strategy backfired because many of the orgtheory commenters disagreed strongly with Davis, with some commenters objecting (rightly, I think) to his criticism of internet commenters and others pointing out issues with the Kovacs and Sharkey paper and writing things like “I think you overestimate the meticulous nature of peer review.” Again, I’m not saying that this paper was terrible, just pointing out the gap between incoherence of the praise that the journal editor is giving to it.

Nothing magic about publication

To bring this back to a subject discussed a lot in this space recently (for example, here), one problem I see is the perception of journal publication as a sort of plateau: the idea is that before a paper is published it is speculative but once it appears in a top journal, we are expected to take it seriously, with any criticism to be held to an extremely high standard.

It’s hard to get a paper published in a top journal—and, from an editor’s point of view, it’s hard to get media publicity—so, in either case, when you’ve achieved that goal, it’s natural to want to take a rest. Indeed, author Balazs Kovacs writes:

I have always been all for the transparency and truth and diversity of opinions that such an open discussion can bring out. I am still an advocate of that. But what I realized is that this can be really taxing on the authors. That is, if someone posts a comment / doubt about your paper, you as an author need to address that otherwise the last public record will be an unanswered doubt. . . . And that is not something I look forward to. I am a kind of person who gets tired of a project during the publication process (I guess I’m not alone!). The main reason that I love getting a paper published is that then I can close the process and move on to other new and exciting projects. The key is “moving on.” The fact that in such public debates of my previously published papers I’d need to go back to old stuff, essentially takes away the biggest satisfaction I derive from publishing a paper.

Wow! That’s just amazing. I mean, sure, I too get satisfaction out of having a project done, seeing the paper in reader’s hands, seeing the book on the shelf, grading the last final exam and saying goodbye to the semester—but “the biggest satisfaction”???

If your biggest satisfaction in a project is that it’s done and you get to “move on,” maybe you shouldn’t have done that project in the first place! For me, the biggest satisfaction of hiking in the mountains is that delicious moment when I get to take off the backpack, pack away my trekking poles, and rest. Maybe that’s why I haven’t gone backpacking in 20 years.

Again, there seems to be this attitude that publication is a difficult challenge and once you’ve achieved it, the accomplishment shouldn’t be diminished in any way. You’ve sold it, as it were, to the journal or the press, and you’re not offering any refunds. Always be closing, then move on to the next lead.

What’s it all about?

The article in question, based on a counting of positive and negative reviews of 64 books, reports that “prizewinning books tend to attract more readers following the announcement of an award and that readers’ ratings of award-winning books tend to decline more precipitously following the announcement of an award relative to books that were named as finalists but did not win.” I assume that the prize motivates people who otherwise would not have read and commented on the book to read and comment on it, and that these marginal readers and commenters are less enthusiastic about the book, compared to people who would read and comment on it even in the absence of a prize. Or, as one of the much-derided anonymous internet commenters writes:

It doesn’t surprise me. You get it a lot on amazon where if a book has won an award or becomes famous it attracts a lot of negative reviews because people buy it because of the hype when it’s probably not their sort of book. Meanwhile less famous books by the same author get higher average reviews because they tend to be read only by people who are into that writer and maybe more on their wavelength. I’m a big fan of Roberto Bolaño for example and his two most famous books 2666 and Savage Detectives have some pretty negative reviews by people who bought them on the back of the hype and hated them because he’s simply not a writer for everyone. Meanwhile books like Nocturno de Chile (can’t remember English title sorry) or Distant Star get better averages because only Bolaño-philes are likely to read them.

Well put, but it’ll never appear in the Administrative Science Quarterly.

Neither of the two main findings of the Kovacs and Sharkey paper surprise me either, and Davis’s labeling of them as “intriguing” seems a bit of a stretch. I think what is happening, again, is that he’s getting carried away by rhetoric. Rather than writing in literal terms, he’s using words and phrases that sound good.

That said, it can be good to do a quantitative study even if only to confirm what makes sense. I say this as a person who, with collaborators, has published two papers on the advantage of incumbency in congressional elections, speaking of obvious topics.

Davis analogizes this research to a “healthy salad” and compares it to an unrelated paper “which found that people who walked down a hallway while texting on their phone walked slower, in a more stilted fashion, with shorter steps, and less straight than those who were not texting,” which he characterizes as “Cheetos.” Davis’s preferred paper is related to business management and the texting paper is related to public health, so I can see why he, as a business-school professor, would consider the former topic more important and worthy of serious coverage.

The horrible way in which publication and publicity are taken as a reason to close off discussion

And I better say this again: My point here is not to pick on Davis (who I’m sure is working very hard editing a journal, which is a form of service to the academic community) or Kovacs and Sharkey (whose paper seems reasonable enough, if unremarkable) but rather to emphasize the horrible (to me) way in which publication and publicity often seem to be taken, not as opportunities for further exploration, but as a reason to close off discussion and suppress the sharing of knowledge.

The idea that a claim, just because it’s published in a top journal, should go unquestioned, or that critics are “trolls,” or that anonymous commenters can’t be trusted (in comparison to anonymous journal reviewers, who are trusted even though their reports are forever hidden): that really bothers me. It seems contrary to the goals and practices of science and scholarship, and indeed contrary to the goals of publication itself.

I sort of understand the attitude on a professional level—once you’ve eaten the cake, you don’t want some critic going down your throat to pull it out of your stomach—but I don’t have to like it. And in some way it’s good to have this discussion with a paper that’s not particularly crappy, to emphasize that this is not just an issue with research that shouldn’t have been published at all; the concern also arises with solid, run-of-the-mill, another-brick-in-the-wall scholarship of the sort that we all would like to do, the sort of work that is central to the scientific enterprise, but which should not (I believe) be hyped and then held to be immune from criticism.

68 thoughts on “Literal vs. rhetorical

  1. Andrew: Thanks for the very helpful redirect of that discussion on orgtheory.

    Two notes:

    First, I think Jerry’s beef with the texting paper was that it was obvious (or tautological), not that it was on a public health topic. I haven’t read that paper though, so I can’t say whether his criticism is right. One can argue about whether the Kovacs/Sharkey paper is surprising or not relative to our intuition, but I would say as a social scientist who works in that area that the story seems less obvious in that there are intermediary mechanisms that one needs to work out to get from the event (award winning) to the result (more negative reviews) and one can a combination of mechanisms [and/or parameter values] that would lead to a different outcome. So there are puzzles to work out here. Perhaps that’s true in the texting paper, but at least based on Jerry’s summary of that paper, I agree with his take that it is uninteresting.

    Second, I actually wrote a quick and positive reaction to what Balazs wrote. This was despite the fact that I endorse everything you wrote about how we should regard published papers. (Here at the Economic Sociology PhD Program at MIT, it is a standard part of our training to get students to write mock referee reports on published papers. This is intended in part to get them not to give undue deference to a paper just because it has been published. Usually, we then have to address the opposite problem, which is to explain to them why they should not give up on the field even though most published papers are so weak). I was responding to a more narrow issue though, which I think is important– i.e., whether an author should be continually on the line (in the form of meeting active public challenges) to defend his/her work. That is, it can feel oppressive to have to face critics from the time of publication ad infinitum. I’m not saying in any way that this consideration is dispositive. Just that it is an important issue to consider.

    Thanks again for the great post.

  2. Jerry makes both a strong argument and a weak argument for elitism vis-a-vis the plebs of the comment threads.

    The strong version of his argument was that developmental peer review is a crucible of validity. Let us take as granted that this is wrong. There’s a probabilistic version of this that you and I both are less enthusiastic about than Jerry, but I don’t think even Jerry would say that developmental peer review is absolute proof of validity.

    The weak version of the argument is that internet comments are almost always lazy, essentially never reading the underlying paper but feeling license to opine about alleged flaws that are in fact addressed in the paper. We have all read comments along the lines of “this is obviously a spurious effect of X” (when in fact the authors are attentive to X and use a good specification to control for it) or just the mantra “correlation is not causation” (when in fact the authors have an RCT).

    I also think you’re not being fair to whether Kovacs and Sharkey have a compelling null. Like many social science findings it makes sense once it’s explained to you (especially if you have cultivated the tendency to think in terms of censorship, truncation, and colliders) but is not necessarily predictable in advance. I myself study awards and if you asked me prior to having read Kovacs and Sharkey whether awards raise or lower rankings I probably would have told you a story about status and the Matthew Effect and said they would go up, even though in retrospect it makes perfect sense to me. Collider type problems are generally the kind of thing that are hard to recognize but make sense once they’re pointed out to you. (I know this because I lecture my grad students on them and then watch them fail to recognize the issue when it’s presented in a word problem on the final). Basically, lots of social science findings are <intuitive but that’s a different thing from obvious except in retrospect when it becomes hard to tell them apart.

    • Gabriel in someways I agree but let me give an example.

      In a professional Epi course, given by someone I believe is relatively good and has studied (or even written on) colliders, in an example she made up for one of the final slides of her course contained a collider which upon being pointed out to her she responded “You are just being like a grad student looking for complications that aern’t there – oh OK right – that is a collider.”

      So I would not agree finding a collider that others did not think about makes a “really good paper” and that one “should read it immediately to find out whodunit”.

  3. I think you might be too harsh in your criticism regarding Davis’ comment. Another way to read it is that he acted as a harsh and negative editor from the outset towards the paper, bolstered by skeptical reviewers, but as his questions were answered and concerns addressed, his opinion shifted to regarding it as ‘a really good paper’. His final opinion was the result of the process of engagement that he had with the authors.

    That said, your general point is still valid- often statements are made that sound meaningful, but fall apart on closer inspection.

    • I agree. When I read, “As is traditional at ASQ, the authors faced smart and skeptical reviewers who put them through the wringer, and a harsh and generally negative editor (me),” I took it to mean that the ASQ reviewers approach papers skeptically (most reviewers do) and that the editor is typically negative (as most editors are), not that the reviewers were skeptical about this paper or that the editor was negative toward it. The qualifier “As is traditional” before the rest of the sentence seems like a clue, as does the “generally” before “negative”.

      Language is really vague. With no object for “skeptical”, we can’t tell whether it’s meant generically or if it’s meant specifically about this paper, and if the latter, we can’t tell if the skepticism and negativity were prior to reading the paper or posterior.

      • Bob:

        Take a look at the linked thread. Steve Morgan and other commenters list a bunch of criticisms of the paper in question. A “harsh and generally negative editor” would not heap such praise on a paper with so many issues.

        • Andrew:
          While the thread does reveal a number of criticisms concerning the paper, this differs from your initial criticism of his statement, which argued that Davis’ statement was inherently incoherent. Again, solely considering that statement on face value, it doesn’t appear to be as necessarily incoherent as you believe it to be.

        • Zachariah:

          Believe it or not, I did consider that alternative explanation and almost discussed the point when writing the original post, but then I decided not to because the presence of all these unresolved issues seemed to me to be contradictory to Davis’s claim of being “harsh and generally negative.” So I was interpreting that statement in the context of the entire discussion. I agree that there is no contradiction in the statements alone, without the context.

  4. I find it interesting that the commentary on the paper follows the pattern that they describe with book awards. Paper gets press -> Readership expands beyond intended cluster (where most of the empirical literature points to the opposite prediction) -> Paper gets derided as pedestrian and obvious by people outside the intended readership.

    I just hope this secondary commentary does not deter people from actually reading the paper and assessing it for themselves.

    • Craig:

      Maybe so, but it’s pretty sad if the result is obvious to laypersons but surprising to professional sociologists. I’d hope that a sociology result would go the other way!

      • This happens more often than we’re willing to admit. That’s why we need the Watts/Rossman (above) justification about overconfidence and how ‘everything is obvious’ to laypeople once the findings are presented. If you make people state their priors before reading the research, it is probably a lot less obvious to people than we might think from reading comments. It might be worth doing a survey (as did Lazarsfeld with the American Soldier study) to see how obvious this result is to laypeople ex-ante. In fact, this may be good as a general practice to see how much new information social science research is contributing to common sense.

        Of course, I agree with Gabe’s suggestion that people with certain areas of expertise are neither overconfident, nor surprised by this result – in which case this exposes an arbitrage opportunity for publication.

        • Consistent with my comment above, I agree with Craig and Gabriel that Andrew needs to consider what null he would have had before reading the paper. Also, the issue is not whether the paper by Kovacs/Sharkey has the most compelling null in the world, but whether the null of their paper is substantially more compelling than that of the paper that texting makes us walk more slowly (it does improve driving though; I can vouch for that…). I am with Gabriel and Craig that a random sample of Americans who were asked to anticipate the results of the two papers (assuming the texting one is as Jerry Davis describes it) would guess the results of the texting paper much more easily.

          And btw, if you look at research on how fashion cycles in part result from overshooting in situations where people are inferring quality from popularity (I review key models here [sorry if behind a fire wall; the key names in the literature are David Strang and Stanley Lieberson]: http://tinyurl.com/nsdvdc4), you will find that the processes involved are anything but straightforward. So in short, while I liked Andrew’s original post, I find his glib assertion that it is “sad” to hear what sociologists find interesting to be considerably less thoughtful.

          (I’m signing off this blog for the foreseeable future; will read what comes next at some point)

        • Ezra:

          Much of my concern is in the presentation. My impression is that Steve Morgan had similar concerns. The researchers found that “prizewinning books tend to attract more readers following the announcement of an award and that readers’ ratings of award-winning books tend to decline more precipitously following the announcement of an award relative to books that were named as finalists but did not win.” Davis describes this as a “puzzle.” From my perspective, part 1 of their finding is so obvious as to be a precondition for any further research on the topic, and part 2 is potentially interesting in that it fits in with one of the two obvious hypotheses ((a) that a prizewinning book gets a new set of readers, and (b) that reviewers follow the bandwagon; the finding is consistent with (a) rather than (b)). So it’s fine for this result to be presented as interesting and as supporting a hypothesis of general relevance. But is it a “puzzle” or an “intriguing observation”? I don’t think so. Did the authors really “figure out which mechanism was ultimately responsible?” No, of course not. Were the newspaper commenters really so wrong to point issues with the paper. I don’t think so either. Were the paper presented more calmly, without the hype, I’d be much happier with it. I think many newspaper commenters are like Steve and me: when we see extreme claims, we start to get skeptical, and rightly so.

        • Craig:

          I agree with you on the general point of watching out for “everything is obvious.” But let’s look at this particular paper:

          Finding #1 is that a book that wins a prize gets more readers. This is beyond obvious, to the extent that a finding in the other direction would not be news; rather, it would be clear evidence that something was wrong with the study.

          Finding #2 is that a book that wins a prize gets worse reviews, compared to a comparable book that doesn’t win the prize. That’s not surprising, but indeed I could imagine it going the other way (some sort of bandwagon effect). If it went the other way, it would be more newsworthy. In any case, I agree that the paper makes some contribution, just that it doesn’t live up to the hype that the journal editor was giving it. I think the analogy to my incumbency advantage papers is a good one. It’s no surprise that incumbent congressmembers running for reelection do better than nonincumbents in comparable districts. What we added was a quantitative controlled study. Quantitative controlled studies are good, and I agree with Steve Morgan that the Kovacs and Sharkey paper has some value. But (a) I don’t think it lives up to the hype, and, more importantly, (b) I’m bothered by the effort to disparage or shut off criticism of the paper.

        • I think you are picking up on the less interesting ‘headline’ findings. By my reading, the more interesting findings are the ones that show a) the post-award readers differ in terms of their genre preferences when compared with pre-award winners, which is associated with lower reviews, and b) that post-award reviews are 57% more likely to mention ‘expectations’ and that these reviews were approximately 1.2 stars lower. These factors (decreased fit, increased expectations), which are discussed in the mediation analyses section are the mechanisms that Ezra alluded to. These are not processes we have a lot of good micro-level evidence on from other domains.

          The watermark findings in this area come from the Salganik et al. music lab experiments. In that context, there was pretty strong evidence of a bandwagon effect when users were presented with rating or download count info. This paper is showing that award-winners do not follow the same pattern. I find it interesting that winning an award seems to disrupt this pattern.

          Of course, the paper is not perfect. A lot of the inferences hinge on the accuracy of the predicted rating model. The stronger that model of preferences becomes, the less likely they would find an effect. None of the criticisms I’ve seen thus far have been compelling enough to dissuade me from citing or recommending the paper though.

          p.s. The finding that winning a prize boosts readership is indeed very obvious. However, Azoulay, Stuart & Wang (Management Science 2014) show that this may not be the case in some contexts – HHMI prize winners only got a slight and temporary citation boost compared with matched nominees, suggesting this effect has been overestimated in prior studies. I think the Azoulay et al. paper fits more nicely with your analogy because they are revising previous estimates of Matthew effects downwards with a more rigorous test. Sharkey & Kovacs are presenting novel evidence of the effect going in the opposite direction (albeit on a different DV).

        • Greg:

          Good points. I was following Davis and focusing on the headline findings, but that wasn’t quite fair of me, as Davis himself recommend that we read the whole paper!

        • Andrew,

          Great post, I loved the connections you drew to Glengarry Glen Ross!

          I don’t want to go too off-thread here, but I read your everything-is-obvious post that you linked to. I would like to ask you please about a couple of comments you made in the section “Rationality != selfishness”. I see though that your previous post is vintage 2011, with no comments more than several weeks following.

          There’s discussion in the orgtheory post comments about how long of an interval to allow commenters to respond to published research (e.g. six months?) Apropos of that question, how do you feel about your own readers wanting to revisit years-old posts? Thank you for your consideration.

        • Brad:

          We set up the software to cut off comments on old posts after a certain point to control the spam. I love when readers revisit old posts. One cumbersome but effective way to do this is to send me an email and then I can repost and allow more comments.

  5. I cherish the moment when I reach the parking lot after a long hike, and I cherish the process that earns me that feeling. And I would hate to be told I had left my keys at the top of the mountain, forcing painful unintended effort for no new reward. So the sentiments aren’t unreasonable.

    That said, we function under inconsistent notions about publication. Since pubs are the currency of academia, they are thought of as prizes and once earned it’s natural not to want to be done with it. (After booking commission, the sales person has little incentive to revisit the transaction.)

    At the same time, the publication was likely “sold” under the pretense of making a contribution to the “literature”. So there is obviously an ongoing obligation to become part of that “literature”. 10 years ago, the vast majority of papers would fade away without causing a ripple, and any responses in the “literature” would only come at a glacial pace. Now the pace has picked up, and the feedback comes fast, and furious, unregulated and unfiltered. If you care about ideas, it’s a stimulating environment. If you only care about trophies, it’s sad to see them tarnish so quickly.

  6. Most reviews on Amazon for higher-brow nonfiction books tend to be positive four or five star ones from readers who appreciate the hard work the author put into an obscure subject. In contrast, reviews of more popular fiction books tend to be all over the place. I suspect a lot of reviews are driven by whether or not the amateur reviewer would like to be friends or not with the main character in the novel.

  7. Let me start by saying that the intention of this post is to agree with the argument that a journal’s publication of an article should be the start of the conversation, and not the end. It is a good thing for science to talk about articles that have been published, raise critiques, and allow authors and other to respond. (ASQ has created a blog exactly for this purpose, with exactly this stated rationale. Feel free to contribute.) It is also the case that some kinds of criticism are trolling rather than constructive, and that the boundlessness of the Internet means that we ought to think through what is the most productive forum for dialogue. That is where I will end up, so don’t be distracted by rhetoric along the way.

    So: agreed with Ezra, Gabriel, Zacharia, Bob, and Craig. Per Zacharia and Bob, “smart and skeptical” (the reviewers) and “harsh and generally negative” (me) were not conditional on this particular paper. Andrew’s rhetorical analysis, although ingenious and inventive, mis-interpreted the implication of that sentence. (Oops, check the superlatives. Might be mis-interpreted.)
    To better understand the intention of the original post (about how different journal formats, combined with incentives to do public-friendly research, can have bad consequences), read here: http://orgtheory.wordpress.com/2014/02/26/is-public-intellectual-oxymoronic/#comment-137391

    Per Gabriel, Guardian readers are welcome to write what they want. The Guardian gets smart readers (as some of the comments showed), and The Guardian itself includes many helpful features, including downloadable data repositories. In this case, they included a direct link to a pre-publication version of the article posted at SSRN. That is, anyone interested in this paper could download it in literally two clicks, with no paywall, in under 10 seconds. It was therefore frustrating to see comments describing the work as fluffy, iffy, obviously wrong, the product of publicists, etc., as well as asking easily-addressed questions like the sample size. Readers can hold these opinions, but if they care enough to kvetch, they might care enough to read the actual paper. The link is right there. (And Steve’s comments do not vitiate the basic correctness of the paper’s conclusions about what was going on.)

    Is the opening observation you cite from the paper “intriguing”? Suppose someone stated: “If your book wins a prestigious award, your readers are going to like it less.” The average person (that is, someone who’s not a statistician, and who has not incorporated ideas like “selection effects” and “regression to the mean” into their repertoire—cf. Gabriel) might respond “No way.” As Ezra suggests, ex ante there are a lot of possibilities that will seem obvious in retrospect if they turn out to be correct. You might expect a halo effect where ratings go up on average after the award, as reviewers want to be seen as the kind who admire admirable books. (That seems obvious.) You might expect the book to find a larger audience of people who like books like this, with average ratings staying the same as sales go up. (That seem obvious too.) You might expect ratings to go down because, as one Guardian reader (and a reviewer) suggested, cranks come out of the woodwork to trash things that receive recognition. (That seems obvious.) You might expect that awards lead teachers to assign books to kids who are likely to hate them and complain on the Internet. (This may be why the Twilight book collection is rated higher than Cather in the Rye on Amazon, where Catcher has hundreds of terrible reviews.) But people voluntarily buying books and then writing negative reviews? Perhaps not entirely obvious, particularly when set against these alternatives.

    Central to the journal review process is doing a reasonable job of considering alternatives. Quoting me: “Any of us could think of a half-dozen ways the data could have gone one way or another; if you assemble diverse reviewers, you get a lot of options (and perhaps even more post-publication). Through the review process, reviewers raised possible alternative explanations; authors responded; reviewers replied; authors responded more. Papers improve through this back-and-forth in the review process, via a virtual conversation with people having varied expertise.”

    That doesn’t mean that the final product is an impregnable fortress, immune to criticism. Three reviewers and one editor poked and prodded the paper, and judged that it was probably right; subsequent impromptu reviewers (some of whom might have actually read the paper before piling on) had additional “room for improvement.” (Andrew: since you’re a fan of rhetorical readings, let me note that the subtext of Steve’s comments was not “Don’t believe what’s in this paper” but “This editor is a sloppy, smug jerk, and if the paper were sent to my journal, it would have been even better.”)

    So, publication ought to be the beginning of the conversation, not the end. At Sociological Science, they have comment sections attached to each paper, and rules for civil and timely discussion. (See Ezra on this.) This is a good way to do things. At ASQ, we have a blog, intended primarily for doctoral students learning the craft of research, with author interviews and room for dialogue. This is another way to do things.

    But what this week has demonstrated to me is: “Write a blog post in haste, repent at leisure.” I wrote a hasty rant aimed at a specialized audience of organization theorists intended to spark a discussion of how different journal formats create incentives for different kinds of work, and prompted a completely different discussion, requiring many multiples of that initial writing time. In retrospect, obviously a mistake (hey, maybe there’s an ASQ article in this…), but I have tenure and don’t have to care much what “Anonymous” or “Steve” think.

    The authors of the article, however, got dragged into the discussion unbidden, by me. It’s one thing to give a seminar on a paper, send it for review, or have it published in a journal with room for commentary (like Sociological Science). In that situation, you expect push-back and dialogue. It’s quite another to find yourself in a knife fight somewhere on the web, during your heavy teaching semester… Now Balazs writes a couple of hasty paragraphs on how this experience has been disorienting, in the spare minutes accorded to junior faculty, and he gets crap for it. (The deleted Anonymous post said something to the effect of “If you want to stop responding to comments about your paper after 6 months, then the field should be able to stop citing it after 6 months.”) Note that the crap was NOT about the paper (that would be thin-skinned!), but about his hasty paragraphs. It’s not obvious how this kind of discussion advances science.

    And now this is fodder for yet more blog posts, in yet more venues. Andrew Gelman (wait, are you Susan’s brother? Didn’t I meet you at Adam’s bar mitzvah a few years ago?), Thomas Basboll, who knows where else. It’s like bamboo, and it’s not something any author can reasonably expect to address.

    Here’s a request: a good and useful discussion is “How can we create forums that allow published work to be critically discussed, with the participation of authors, in a way that the field can advance?” (I believe this was Ezra’s re-direct.) A not-at-all-useful discussion is “I don’t like what this untenured person said in a stray blog comment. Pile on!” Reasonable?

    • As one of the anonymous readers (not among the commenters on the orgtheory post), I was surprised that the authors didnt respond to the substantive methodological questions raised by the comments. Its their unwillingness to engage the valid concerns “trolls” (who might I add include un-tenured junior folks unwilling to make enemies once a paper has been anointed by ASQ) that I find surprising– a paper doesnt have to be perfect to be published, but acknowledging limitations (which the authors do to some extent in the paper) hardly constitutes a failure on part of the authors.

    • “How can we create forums that allow published work to be critically discussed, with the participation of authors, in a way that the field can advance?”

      Good question.

      One of the challenges here is balancing individual enquirer’s interests with the wider community of inquirers that includes anyone who desires/needs to appraise what is being claimed. Many tenure stream faculty currently have their interests so much are risk, it would be unwise or even almost unethical to just trust them to be doing and reporting only excellent work in a transparent manner. Many who have been involved in journal peer review (including me) believe it to be inadequate.

      On the other hand, flurries of mixed quality comments are likely to overwhelm faculty and cause them to panic. It is good when tenured faculty try to help tenure stream but again this has to be balanced against the interests of wider community of inquirers. Censorship and or disparagement of comments seems like the worst solution even when many of them are of very low quality.

    • Jerry:

      1. Yes, Susan is my sister. A couple years ago I gave a talk at Michigan on causal inference.

      2. Yes, it does seem like the original discussion went in unexpected ways. I think that’s good, though! It’s good to get these issues aired. I don’t see any of the original commenters (on the newspaper site or at orgtheory) to be trolls, at least not in the wikipedia definition of “a person who sows discord on the Internet by starting arguments or upsetting people, by posting inflammatory, extraneous, or off-topic messages in an online community.” Some of the comments might well have been mistaken, but I don’t think they were inflammatory, extraneous, or off-topic. In any case, I appreciate your calm, direct, and point-by-point responses to criticism.

  8. I’d like to respond to Jerry’s comment: … but I have tenure and don’t have to care much what “Anonymous” or “Steve” think.

    Let me just state for the record that, contrary to what Jerry may be attempting to imply, the only comments I wrote on the orgtheory thread in question were indeed signed “Steve Morgan.” The orgtheory folks have IP addresses and can confirm.

    I did not respond to the Kovacs comment after it appeared. By the time I saw the brouhaha that erupted following it, the comments were turned off. When they were turned back on, I decided not to respond. (I almost responded to the comment by “Lisa” since it implied that “motive” and “point” were the same thing. My motives were transparent, as I made clear. But my points of argument had scientific content that went beyond my motives. They were things that Jerry or the two authors could have responded to, and may yet do so. They stand as they are regardless of my motives for offering them. They may even help to clarify what the paper does accomplish, and if so that is an outcome that benefits the collective.)

  9. Andrew:

    A paper was published in his journal and got positive publicity in the newspaper. This made him happy (as indeed it should; he works hard as a journal editor and it’s good to feel that your work is making an impact)

    I think it is wrong to measure scientific impact by whether articles are reported in the press. Editors obsess about it, academics list press reports in their web pages, and CVs, and so on. This is somewhat surprising considering how in the 19th century academics considered it unprofessional to seek publicity. Now it is actively encouraged.

    It’s odd how even scientists now judge impact on the basis of press reports. Given how silly claims sell copy one can easily derive the impact this sort of incentives will have on editorial decisions, submissions, grants, funding, fishing, and even the research questions being asked. At my most cynic: Bread and circus.

    PS I am not picking on Andrew here, its just that over the past few weeks I have seen references to press and academic impact, and I think it is plain silly. Ok if the press picks up a finding, but whether it does so or not should in no way influence science (normatively speaking)

  10. I’m just seeing this thread now after most of the discussion has ended. Most points that I would have made have already been raised earlier. However, I want to respond specifically to Andrew’s point much earlier about whether what he calls “finding #1” (increase in readership after an award) and “finding #2” (decrease in ratings after award) are surprising. Of course, “surprising” is relative to one’s intuition and to one’s knowledge of the literature, so this assessment is going to vary somewhat across individuals. I can only tell you how my co-author and I perceived it and intended the paper to be read.

    “Finding #1” was not intended to be a finding, nor did my co-author and I find it surprising. We did not hypothesize about it. We used it to set up the story and establish that the setting wasn’t completely bizarre. On p. 2 of the paper, in discussing the example that we open the paper with, we write “As expected, sales of the book soared, but something surprising happened as
    well: readers’ ratings of the book entered a period of protracted decline even though the status of the book had increased.” Hopefully that clarifies and establishes that my co-author and I do have some level of taste as to what is “surprising.” That said, I can see how some of my own sloppiness (tiredness at 2am when I was doing the last copy-edits a few weeks ago, to be honest) in the abstract led to my/Balazs’ intentions being misunderstood on this one. I have now learned a lesson that is probably obvious to others who are more experienced — don’t work on the abstract last; it gets the most attention!

    “Finding #2” (decline in ratings) was intended as a finding and was surprising to us, based on our deep knowledge of the literature on status, including awards. The paper centered around explaining this finding. That the explanation for this finding is not obvious is, I think, underscored by the fact that one of Stephen Morgan’s questions about the paper is whether our proposed mechanisms actually explain this finding, or whether his proposed explanation – a backlash dynamic especially surrounding Booker Prize winners – was also occurring. In addition, the ASQ reviewers pushed us to examine was whether elevated expectations explained the results. The fact that we can come up with all these different possible explanations suggests to me that we had a genuine puzzle to explain and one whose answer wasn’t obvious.

    In any case – thanks for the comments and, more than anything, thanks for the tone in which they were conveyed. I am happy to engage about the content of the paper when the dialogue is respectful and seems motivated by genuine interest in the paper.

    • Amanda: What is obvious and interesting is clearly in the eye of the beholder. I write all kinds of papers that other people would presumably find obvious, if they noticed them. An example: high school sophomores who say they expect to end up in STEM occupations at age 30 are more likely to choose STEM majors in college. The only reason to even write up such a finding is that the recent academic literature seemed to ignore this point in its zeal to blame colleges for gender disparities in STEM when in fact the predispositions are (of course) shaped by K-12 education and rather powerful socialization mechanisms that continue to operate. Completely obvious stuff, but it took us a couple of years to generate the data to be able to say that with precision. Guardian readers, if they ever saw the paper, would presumably wonder why we bothered to write the paper at all (and would probably be horrified to learn that it required an NSF grant and a team of 8 research assistants).

      You are correct that I do think finding 2 is a puzzle worth analyzing, not something that is obvious. You have a distribution of individual-level effects to consider. There are at least three types of people who could generate the association: (1) those who generate your favored mechanism (new readers who are disappointed), (2) those who generate backlash because they are contrarians (readers who are perpetually angry at the Booker committee or who are going after status of their own by showing how tough they are), (3) those who generate backlash because they have taste for the particular other short listed books (the avid Booker shortlist readers who read all of them and then write a more negative review of the winner than they otherwise would because they want to restore balance of interest across all the finalists). You could further divide (2) and (3) in various ways. I would start from the assumption that these groups exist and then engage in analysis attempting to determine how large these groups are and then perhaps think through how their ratings of the book add up to an overall decline in ratings. You would probably end up saying that type-1 individuals are the largest group, but that you can only generate suggestive evidence based on an analysis of the low prevalence of types of backlash you can model, etc. And you obviously don’t have a design that allows you to identify the feelings about the book for each individual before and after the award is announced, and so overall you can’t identify the full distribution of effects and aggregate up from them. Starting from this overall “population” orientation, rather than a small set of matched pairs, might lead you to a wider range of informative analysis. You might still end up with a matched-sample-based result too, but you’d have other results in a supplementary appendix that show the broader patterns in the data and that you could point fair critics to in defense of your conclusions (i.e,. “Morgan thinks diff-in-diff is too rigid. He should go read Model 3 in Table S4, where he will see that the association holds also for a lag variable model, which is also not appropriate but shows the same basic result under an alternative set of implicit assumptions about the way in which regression to the mean operates.”)

      (This is meant to be friendly, although after all of what has transpired this surely rings hollow. I regret that the tone of my earlier comments made me appear less friendly to your scholarship than I am. I let the dismissive tone of Jerry get under my skin, both in the original post and in subsequent reactions. I am told that this is not an uncommon reaction to Jerry, and it is best to just ignore him. I’ll try to do that in the future.)

  11. Dear All,

    I guess at this point I should clarify what I meant in that post. I made a mistake using the expression “biggest satisfaction,” I apologize for that, it was misleading. What I was trying to refer to with my limited vocabulary is that gruelingly long review processes sometimes kill the excitement I derive from doing research. Maybe it’s only me, and maybe it only happens with junior scholars like me who are not really experienced in the publication process, but spending three years with 4-5 round of reviews is really tiring. Again, it might be only me who gets exhausted of this, but I envy the fields where this is not standard practice.

    All I meant to propose in my post is that Sociological Science, which (to a large extent) came about because its founders also got tired of the existing review practices in sociology, should add to one of its considerations that, to a some extent, post-publication discussions can have the same effect. I fully agree with Ezra’s comment, his interpretation is exactly what I was trying to convey.

    I’d like to emphasize that this was a *general* suggestion and not referring to the discussion in question. I didn’t mean to “close off the discussion.” Actually, when Steve Morgan raised the issues, Amanda and I started to put together a document with the results that would answer Steve’s comments. While we can answer some of Steve’s concerns with analyses we have already completed either during the review process or earlier, Steve’s comments also helpfully gave us some new ideas for analyses that we are running now – in between teaching classes, meeting obligations to co-authors on other papers, and raising families. We hope that a side benefit of our delay in posting is that more people will have time to read the paper, which from our perspective makes for a richer discussion, and that some of the heat from the earlier discussion of issues not related to our paper will have died down.

    Balazs

    • Balazs: Glad to hear you are working on additional results. Authors should revisit their results and defend their conclusions when they have additional results to back them up. Sounds like you have that, and so this will be a good outcome. I’d be happy to correspond with you by regular email if that would be helpful, although I completely understand if that is not appealing.

      Also, maybe you should submit your new results to Sociological Science! (I wouldn’t be the appropriate Deputy Editor anyway, and so you’d get a fresh read from a different Deputy Editor and likely a Consulting Editor too.)

  12. Hi Steve,

    I just wanted to say publicly – no hard feelings, if that’s the right way to put it. Just so you can appreciate my perspective, I’ll admit, I have a fair case of imposter syndrome and so having a tenured professor poke holes in the first paper I’d published out of grad school (meaning, it wasn’t started in grad school), was pretty much my worst fear realized. Actually, it was worse than that – I’d never actually imagined anyone would pay enough attention to my work to do something like that, so I’d never concretely thought of it as a fear. On top of that, I was a bit disappointed that some of the folks who have read the paper didn’t step out of the woodwork to defend it on orgtheory, especially the ones who know you and might have privately talked with you. I experienced how solitary academia truly is.

    But over the last few days, I’ve come to realize that maybe my case of imposter syndrome is not as bad as I think. I reacted indignantly to the idea that someone would feel entitled to pass judgment on my paper publicly after a 15-minute read. I think more of myself than that – if someone wants to comment, do me the justice of spending some time with the paper. Secondly, I realized that the reason people weren’t stepping out of the woodwork to defend the paper was that — holy #%$^, I’m an assistant professor at the University of Chicago. I can, should and will do it myself. This has been a valuable if painful form of professional socialization for me. Thirdly, this has made me realize that you truly can’t please everyone. Some of the critiques of the paper were ones that I view as a matter of taste – eg how much detail about the matching to include – rather than absolute rights or wrongs. In fact they were conscious decisions in that I didn’t want the paper to fetishize matching, which I agree with you is a fairly well developed technique these days. Probably we should have included a little more, but even then, it probably would have been less than you would have preferred. In any case, I realized I don’t need to feel bad that my taste (or the ASQ reviewers taste) isn’t the same as yours. So I truly thank you for all of that.

    That’s the end of my public therapy session, which I only posted for the benefit of others in similar demographic categories as myself who might look at this experience as pretty much like getting hit by a train. There have been good things that come out of it, not the least of which are that we will get to dig deeper into the data that was the basis of the paper. I’m very excited at some of the ideas Balazs and I have come up with as a result of your comments, especially on the backlash front. And the suggestion to try to publish it as a paper is great. While I agree with others who have said we would ideally not be tired of our papers post-publication and that we do have some obligation to engage with public critiques of them, the reality is that there is always a trade-off between doing those things, which I’m not sure will be valued as much when I come up for promotion, and writing another paper. This gets around that trade-off.

    Anyway, truly thanks again. i’m glad to end this part of the discussion on a positive note.

    • Great comment, Amanda. Should be required reading for assistant professors everywhere. And just great generally to have this discussion end in this way.

  13. Pingback: the orgtheory psychological contract | orgtheory.net

  14. Forest, tree-people, forest. The major issue is how crap all these books are nowadays. All the good stuff has been written. Peak literature. (/trolling)

    Or am I?

  15. Peer review can help, but it’s not a perfect way to screen out bad work and screen in good work. For a lovely lesson in this, read Art Stinchcombe & Dick Ofshe’s 1969 paper “Journal editing as a stochastic process,” published in the American Sociologist (stable jstor URL http://www.jstor.org/stable/27701478). Using a simple demographic model, they showed clearly that “Even if social scientists as editors are as accurate as people ever are in coding qualitative material, nearly half the good papers will be rejected and the journals filled with mediocrity.” (p. 117)

    FYI, I’m a long-standing member of the ASQ editorial board, and I have published a few papers there. I am also a consulting editor at Soc. Sci. and have a paper forthcoming there. But I’ve never submitted to a non-peer-reviewed open-access journal.

    • Heather: agreed with the general sentiment that journal reviews are an imperfect process. There are lots of great papers that don’t make it through, and occasional less-great papers that do. But I want to be perfectly, crystal clear here: the paper in question is in fact a really good paper, and even its most vocal post-publication reviewer agrees (as far as I can tell) that it is substantively correct and deserved publication. Observe that the commentators in this thread who have actually read it agree (Ezra, Gabriel, Craig, me). It would be great if people take the time to read the paper (and not just the abstract) before launching critiques.

      An analogy: my daughter tells me that “Catcher in the rye” is about a whiner who goes to boarding school, wears a red hat, and loses stuff on the subway, and that the book seems to inspire sullen loners to do crazy things. The book is sitting right there on the couch. I don’t think my daughter’s precis provides a great basis for a literary discussion of the merits of the book or its author’s skill as a novelist. We should not say, without reading the book (sitting right there on the couch!), “That Salinger is a terrible writer – I hate red hats” or after reading the description on the back cover, “What thin characterization! A bildungsroman set in a boarding school is cliché.” We might be able to discuss whether sullen-loner-inspiring books should be assigned in high school, given their possible consequences, but that’s not about the content of the book.

      Harshing on published papers is the very bread and butter of graduate education. Piling on is what we do. That does not mean that the paper is not persuasive or of high quality; indeed, seminars often feature papers that are widely influential and take them apart. Any grad student past the first year could write a brutal critique of almost any published paper in their field if they chose to do so. (There might be norms about the right time and place.) But shouldn’t doing the reading be the price of admission for a speaking role in the seminar?

      • The longer the read, the higher quality the comments and critique will be. In this case, my 15 minute read got the main points out. I read again, and I stand by the initial read, with some clarification that resulted from a closer read.

        I took no position at all on whether it should have been published in ASQ: obviously it met the standard and so should have been published. I did take the position that I thought it would have been published in SocSci but that, were I Deputy Editor, we would have done so faster and asked for some different things (we only have acceptance as is and conditional acceptance as options). But, I would not probably have been the Deputy Editor, and so the prediction may not mean much. Maybe we would have published as is. I very much doubt we would have rejected it.

        Now to Jerry’s implied question: I do think that the featured mechanism for the decline in ratings is almost certainly part of the story, and I think there is enough suggestive evidence that it is the main part of the story. I don’t actually know if it is “substantively correct,” since I don’t know enough about the analysis to tell and “correct” means something quite specific in my mind. If all it means to Jerry is that the overall conclusion (that the favored mechanism is probably the most important part of the story), then my sense is, yes, it is “substantively correct.” But, I also feel that the backlash story is more interesting and more important than is claimed in the article. And, I think there are methodological puzzles worth exploring and that the chosen design may have purged from the analysis a bunch of the variation that supports the favored mechanism. These limitations, as I see them, mean that the paper is not as correct as I think it could be. Probably all papers are not as correct as they could be, although I disagree with Jerry that any grad student past the first year would be able to tell quite as easily as he says. Yes, they can write a brutal critique, but not necessarily on the important issues that are relevant to whether a paper is correct, especially for papers that require methodological expertise to understand (and this one does).

        On the larger issue, I don’t think it is good for anyone to act as if people do not make judgments of papers, or their authors, based on reads of 15 minutes or less. It might be nice for editors and senior scholars to claim otherwise, so that junior scholars and graduate students feel that they are being treated with great care. It is just not true, and one could never get through a junior search, for example, by reading every paper in detail to appreciate every fine point that is made. The quality of decisions in such cases is determined by a wide read of the whole pool (not just a deep read of those who are from high status places) followed by a progressively more careful read as the list is winnowed. Good decisions rest on the ability to actually get to the strong and weak points in papers in less than 15 minutes, considering all of the papers a candidate has submitted. Bad decisions come from quick reads of papers that are well placed, but have holes in them, which the readers assume must not really be holes because the status conferred on the paper by the publication venue suggests that they are not holes. Or deciding to read only published articles, and not even giving 15 minutes to the unpublished papers in files or those that are published in second-tier journals. Given that this is how it really does work, junior scholars and graduate students should, in fact, assume that their papers will only get 15 minutes, and then a harsh evaluation will be made. Most people whose judgment I trust give the front end 3-5 minutes, then go straight to the core of the paper, try to figure out what is going on in the main argument in 5-10 minutes (which could be an empirical analysis or an analytic argument), and then only read the discussion/conclusion if they believe the design, results, or core analytic argument are solid. These people can process a 10 to 20 papers in an afternoon and give a ranking by quality. And, if it is a high stakes decision, like a hiring decision, they will then go back and do a careful read of the very best papers at the top to pick a winner. Even then, I doubt the papers of top candidates receive anything more than 45 minutes each.

        • Not to turn this into a love-fest, but: good comments. The third paragraph raises an appropriate question about the possibility of a backlash, and how it might be addressed with the data. One of the reviewers raised the same point. A nice feature of GoodReads is that people have an online “bookshelf” of books that they intend to read and review but had not gotten to yet, which was a clever way of distinguishing people who had bought or intended to buy the book before the award, but hadn’t gotten around to it. Backlash might suggest that these people would give harsher-than-expected reviews after the award (vs. new-audience). I’ll leave it to the authors’ forthcoming response to fill you in on what they found.

          A 15-minute read can be very informative, particularly by well-trained readers, and it is true that this is very often all one gets, particularly in hiring committees. My response here was not to your reading (which was ultimately detailed and thoughtful–even zealous), but to commentators who had clearly not read the paper at all, but responded to just the abstract, or just the Guardian’s write-up, or even how it was described in comments on a blog post. It’s fine if you read the abstract and decide not to read on; not so fine if you then write a review or blog comment trashing it. (Hey, it turns out that “Catcher in the rye” doesn’t take place in a boarding school at all!) The worst case would be if potentially interested readers skimmed posts, caught the tone, and concluded that this was not a good paper, or that it had some fatal flaw. That’s a different conclusion than “The following analyses would have been somewhat more informative, but even so the basic story is justified, and arguably interesting, and in any case merits publication.”

          And to be clear, I don’t think that all first-year grad students (or even ANY first year grad students) could have made the same kind of methodological comments that you did. Rather, any student could write up a brutal critique that would convince someone that a work was flawed or weak, and thus dissuade them from actually reading it for themselves. In fact, that sounds like our prelims. (“Can you believe the ineptness of Crick and Watson? Not worth the two pages it took in the journal.”) And human cognition being what it is, readers given strong priors going into a situation might well be guided by those priors, and read with that in mind. (I just learned about the confirmation bias, and now I’m seeing it everywhere.)

          My point with this last post is: read before piling on. The first post had a link to the article at ASQ; the Guardian had a link to SSRN in the second paragraph. You read the paper, which is good. I’d like others to do the same favor.

        • I don’t get it. You make a big deal over at orgtheory about how Morgan only read the paper for 15 minutes (which Amanda Sharkey, above, also seems to find unacceptable), and now you say that you never had a problem with Morgan, but only those (who?) who never read the paper? Then you passive-aggressively post on orgtheory just now about how mistakes were made and criticisms were made “by someone or other.” Good god, man, just pick one! Or if you regret being snarky about a relatively short read of a paper, then say so instead of pretending you never were.

        • Morgan’s self-described 15-minute reading received one in-kind comment from me, which is not that big of a deal. He did eventually read the paper in much more detail. The point in the post immediately above is that reading the paper should be the cover charge for commenting (especially if one is going to go off). It felt that the tone on this particular forum was getting more constructive, hence my measured response.

          But paying the cover charge does not mean that it is *appropriate* to use a particular discussion forum to harsh a paper. “There might be norms about the right time and place” (above). I did not expect Orgtheory to turn into a harsh-a-thon. I do not believe that inviting people to read a paper (along the way to a very different point) would be read that way by most people. (Go back and read the initial post if you disagree.) So it was irksome to be accused of putting authors in harm’s way by the loudest harsher, as if this were something exclusively caused by my post (which was about the hazards of doing work aimed explicitly at gaining media attention, and which, you may notice, is exactly the topic discussed in the early comments of Andrew’s next post). Even editors have feelings. (OK, only some editors.)

        • Jerry:

          I don’t think the authors of that paper were being put “in harm’s way” by commenters (whether here, or at orgtheory, or on that newspaper site. I agree with Thomas that, once someone publishes a paper, it’s appropriate for others to criticize it. Indeed, it’s hard for me to imagine it any other way. And I don’t think there is, or should be, any “cover charge” for criticizing a paper.

          I repeat: no cover charge. I do not think it appropriate for an audience (whether it be statisticians, scholars of business management, managers themselves, or the general public) to assume something is true and correct, just because it’s been published in a refereed journal, even a high-ranking journal. The abstract of the paper contains some claims, and this abstract can be read in 1 minute, not 15 minutes. After reading the abstract for 1 minute, a reader has every right to start asking questions (ideally prefacing such questions with, “I only read the abstract, but . . .”).

          That said, I feel your pain, as Bill Clinton would say. I have had papers of my own criticized based on points that I explicitly replied to. Most irritating was a case where I blogged on a paper, linked to the paper, and in the blog post explicitly mentioned a point of confusion and linked to our one-page discussion of why this was not a problem with our paper. Still, people criticized us for making an obvious error. This was frustrating, but of course it did reveal a point of miscommunication that I’d rather hear about than not.

        • Andrew:
          I will have to respectfully disagree here. Discussion of published papers is of course essential to the advancement of our enterprise; it’s fine and appropriate to be critical, and this makes subsequent work better. But discussion of work among people who have not read it is not productive, and can actually be destructive if it’s based on others’ (inaccurate) characterization of the work. Can we talk about “Catcher in the rye” based on my daughter’s summary? Obviously not — more precisely, we could talk about it, but it would not advance understanding of the topics in “Catcher in the rye,” or inspire better work going forward, and it might lead onlookers either to not read the book, or read it guided by unhelpful priors about the book and its merits.
          There are hazards of drawing conclusions based on a newspaper summary (unless a study is so completely straightforward that none could doubt it; e.g., “texting while walking slows you down,” shown with highly elaborate motion-tracking equipment and video capture). In this case, I called out Guardian readers who not only didn’t read the study, but completely misunderstood the brief write-up in The Guardian. Is it uncool to (anonymously) quote silly things written by someone who concludes “I’ll wait to fully read the actual research in case it’s been badly reported or incorrectly written up but right now I feel the publicist who managed to get this headline and space for what is essentially a non story about flawed research with iffy conclusions is more creative than the research team” ?
          In your own comments above, it appears that you might not have gotten any farther than the abstract in this paper, or maybe even my brief summary. That’s fine; this is your blog, and you can write whatever you want. But it appears that your understanding of the paper is that it reports two findings: awards lead to sales increases, but declines in average ratings. You describe the first as totally obvious, and the second as fairly obvious. True (or truish): but these were not the *findings* of the paper; they were the *premise*, reported in the opening paragraph, and the entire point of the paper was to figure out how this might have happened. (See my earliest post for plausible alternatives addressed. Or, hey, read the paper!)
          You’re certainly welcome to write about how “Catcher in the rye” is set in a boarding school, and compare it to “Dead poet’s society” and “Harry Potter” and the literature of boarding schools, without reading it–just as many high school English students do. But it’s not set in a boarding school, and I don’t see how this can be a productive discussion. Obviously, people should not reflexively believe everything published in journals, and absolutely no one has actually advocated this position. It is a red herring. (Balazs did, in a moment of frustration, note that there might be some conceivable limits on the obligations of authors to respond, but no one said “Once it’s published, it’s true forever and can not be revisiting.”)
          (As for whether alluding to a publication puts authors in harm’s way, your disagreement here is with Steve.)

        • Jerry:

          I think our disagreement here is that that you describe the comments by Steve and others on that paper to be “harsh,” and I find the comments to be quite reasonable.

          But I accept your main point in your recent comment. I agree that uninformed comments will tend to be a bit . . . uninformed! And I’m guilty of this, having often reviewed the promotional materials, as it were, rather than the original article (which I did glance at but did not read carefully).

  16. I really hated Catcher in the Rye and I think the simple answer is the clearest. It’s wimpy poseur literature. And I effing read it. What a waste.

    • IOW your daughter nailed it. And who cares about the boarding school. That’s not the main point, the main point is the narrator is a whiner and that the cult of reading this book is about lionizing whiners and about some sort of ingroup wimpy crap. Go daughter, beat father!!!

  17. You know what I like doing? “Reading the rips” on Amazon. IOW, go look at some book that everyone is supposed to like, but that I find to be crap. And then reading the most helpful 1, 2 or 3 star reviews. Often they are more detailed and analytical than the positive reviews.

  18. I’m with Andrew on this: once you choose to put your thoughts in the public domain you are fair game, whoever you are. Nobody is forced to write anything. You choose to do so, hopefully because you believe you have something worth saying. I think Kovacs & Sharkey have something worth saying & they got the opportunity to say it, so does Steve Morgan & most of the other contributors. I also think that SM makes some entirely reasonable critical points.
    What I can’t understand is why anyone should get professionally upset about what Joe Public says in the Comments section of the Guardian website. If you read that paper every day, as I do, you’ll see Joe and his mates making a lot of dumb and ill-informed comments about all sorts of things. So what: its a newspaper not a scientific journal. JP & his friends and relations are not going to be spending their leisure time reading the footnotes of academic articles even if a non pay-walled pre-print is easily available. Let’s face it, if a full Professor at Cornell is only going to spend 15 minutes or so on a paper he’s professionally equipped to appreciate it’s a bit of a stretch to expect the average Guardian commenter to hold their tongue before letting forth an opinion. Let them get it off their chest, no one is forced to read it and what possible harm can it do (or do Ivys now make tenure decisions on the basis of newspaper reader’s comments?). I should think that post-review debate in a journal like Sociological Science will be a lot more worth reading.

  19. Colin: no one is professionally upset at the comments of Guardian readers. The original post said that responses from online readers favor some kinds of research (straightforward and hard to misinterpret, like “texting while walking slows you down”) over others (e.g., puzzle solving, like “Why do reviewers turn negative after awards?”). Open access journals like PLoS One are well-suited to the first kind of research, but may inadvertently encourage publications specifically intended to be media-friendly rather than substantive. It would be bad if this skewed the kinds of research people did toward the first kind at the expense of the second kind.

    [This has been mis-interpreted to be saying “The public is stupid” or “Anything published in a good journal is thereby true forever and should never be revisited or criticized.” Guardian readers are welcome to say whatever they want, but (a) if research is undertaken to be immediately accessible or sensational to the public, it will be bad, as a subsequent long string on this blog agrees, and (b) dialogue among people who have not read, or have mis-read, a piece of research is not likely to be scientifically productive.]

  20. 1) If we grant you your “may inadvertently encourage publications specifically intended to be media-friendly”, you may also have to grant open-access lovers “traditional peer reviewed publications may inadvertently encourage publications specifically intended to be reviewer friendly”. Furthermore, Plos one publishes thousands and thousands of papers and still has an impact factor close to ASQ (true, one shouldn’t compare). It is a fantastic success story, unless the research community is also stupid for citing these public loving studies.
    2) Furthermore, Eliza has offered a very intelligent post on the potential value of the texting-walking story which you might want to take into account – http://orgtheory.wordpress.com/2014/02/26/is-public-intellectual-oxymoronic/#comment-137484
    3) And I still don’t understand why the ASQ paper is taken to contrast media-friendly stories, given that one can explain the basic research design and theoretical mechanism involved within 90 seconds. Which is one of the great features of the study and one of the reasons it was featured in the Guardian.
    4) About the stupidity of the public: When one writes sentences like “But that would require two clicks of a functioning mouse, and an attention span greater than that of a 12-year-old.” + indicates that the public can only deal with Cheetos research, I am not sure one should be surprised if people interpret it as indicating the stupidity of the general public.

    • @Anon: (1) Agreed. Traditional journals have pathologies, and PLoS One has many great features. (See here for tl;dr version: http://orgtheory.wordpress.com/2014/02/26/is-public-intellectual-oxymoronic/#comment-137391) “Success story” remains to be seen, as the post-publication tools do not yet do what we want (i.e., sort through the 92,000 papers to find the ones worth reading), and the site itself highlights the “most downloaded,” which is clearly wrong and seems to favor sensationalism or pandering. (See my original post.) It’s possible to imagine post-pub tools that would help this (cf. Medium), but they are not there yet.
      (2) Eliza makes an entirely fair point. The texting study is quite meticulous, and I did not say it was “dumb,” but straightforward and hard to doubt. An up-or-down review process favors such work, and strongly dis-favors work that requires colloquy with reviewers – you just can’t have the kind of back and forth about alternatives and doubts that is the strength (and bane) of traditional reviews. There is room for such studies as part of a balanced diet. Not every study needs to be a theoretical breakthrough, but a review process premised on the idea that a submitted paper either meets a quality threshold or not is unlikely to favor such papers (and seems ill-suited to ethnographies and theoretical papers).
      (3) The mechanism that turned out to be (most likely!) correct can be explained in 90 seconds, but it takes longer than that to address the plausible alternatives that arise in peoples’ minds (halo effect, reaching a larger market of people with the same taste, backlash, regression to the mean). If Andrew’s reaction is (paraphrasing from above) “Duh” based on a cursory reading of someone else’s summary of the premise of the paper, then it’s only media friendly if readers either read the paper (which they won’t), or trust that the authors probably addressed that. (And we’ve learned from Andrew not to reflexively trust what’s in journals.) So: if it requires addressing plausible doubts that are too long for a media story, it may not be so media-friendly.
      (4) (a) Those comments were directed at specific harsh commentators who raised questions that either reflected an obvious mis-reading of the Guardian’s own story [compared Times reviews to GoodReads??] or could be answered in under a minute by clicking through to the study [sample size, how selected]. They do not stand in for “the public.” What is your preferred adjective for mis-guided and easily-remedied silliness? (b) Fair.

      • Jerry:

        I just wanted to thank you for participating in this discussion. You could easily have just taken my original post as a sort of disagreement that would best be ignored, but instead you engaged with me and with other commenters. That’s what this is all about.

      • Jerr:

        Hit the carriage return twice when making a para break. I’m not having a go at you. It really is hard to parse text in a block. People are reading fast on the net. It’s for your own good, if the “kindness to readers” is insufficient rationale.

        See.

        Like this.

        :-)

  21. Andrew: My pleasure. Sort of. The discussion did cause me to think through my positions on a bunch of issues around how best to discuss articles post-publication, cover charges (if any), and the merits of different reviewing formats.

    I was going to politely ask about the blog post retraction policy. If one reads through the whole discussion, it seems that the original post is pretty comprehensively wrong, at least from my perspective. (The point of my original post was about the kinds of work that different reviewing formats promote, not about the difficulty of conveying work to the public; the whole unpacking of my paragraph on K&S hinged on a misreading of the sentence starting “As is traditional;” I don’t get excited about press coverage because the majority of the papers we publish get significant press coverage [see http://www.sagepub.com/asq-news/%5D; I had thought hard about the paper, because I was the handling editor; what you describe as the paper’s two main findings were actually the premise, not the findings; and absolutely no one was claiming that papers, once published, should never be criticized, only revered, particularly by the public. There is more, but I’ll quit there.)

    Unfortunately, those who don’t read through the entire discussion (i.e., anybody ever) might read the opening post and conclude: “Sloppy editor publishes crappy paper and brags about it. What a jerk.” For reasons detailed above, I think this is not an entirely accurate summary of the facts. But perhaps this conclusion reflects my jaundiced view of the blog-reading public.

    Nony: Thanks for the tip. It locked right on my screen!

    • Jerry:

      I don’t think there’s anything wrong with what I originally wrote; I still think that the article in question was a solid contribution but did not quite merit all the praise you gave it, and I still think the discussion indicates that, whether or not you are “a harsh and generally negative editor” in general, this harshness and negativity was not evident in your consideration of that particular paper. That’s ok, I don’t think it’s necessary for editors to be harsh or negative; I just felt (and still feel) that you were engaging in the same sort of rhetoric that is often used in letters of recommendation, of placing praise amidst a backdrop of saying that such praise is unusual.

      I also think that some of the discussion at orgtheory went in the direction of saying that, once a paper is published, it should be immune from criticism (unless that criticism followed certain rules, such as requiring critics to read a certain amount of the paper or to have certain professional qualifications). One thing that bothers me about such an attitude is that we can all agree that it would be senseless for unpublished claims to be protected in that way. But the line between published and unpublished is somewhat arbitrary; basically, you have to make 3 referees happy. I had the same reaction to some of the orgtheory discussion as I did to that biologist’s position that nobody should try to replicate or criticize or work until they’d first talked with her and visited her lab. This didn’t seem quite right, given that she didn’t seem to mind that 3 journal reviewers could accept her paper for publication without a conversation or a lab visit.

      I still value your comments here, and I certainly don’t hold you responsible for some of the more extreme comments (such as this one) on the orgtheory blog. I think much of this discussion resolves around differences of emphasis. Uninformed criticism by outsiders is a real problem and that’s what you’ve addressed; unwarranted defensiveness by insiders is a real problem and that’s what I’ve addressed. Ultimately there are tradeoffs here, as neither problem is the whole story.

      • What’s wrong with “requiring critics to read a certain amount of the paper”? That sounds sensible though perhaps hard to enforce.

        • Rahul:

          I object to the sense that it’s considered ok for journalists to promote a paper without having read it, but that it’s not considered ok for outsiders to criticize the press release, as it were, without reading the paper. I think that it’s often appropriate to criticize extreme claims made in the press release (which often appear as quotes from the authors of the paper). I’m not saying the authors of this particular paper did any exaggeration; here I’m just speaking more generally, that I object when people put up barriers to criticism that are higher than barriers to uncritical celebration.

        • That’s like saying two wrongs make a right. I think it behooves both sides to read the material before promoting it or criticizing it.

        • Rahul:

          But we’re all busy (not so busy that we can’t have a blog discussion, but you know what I mean). I agree that comments are better after having read the paper, and I think that people who comment without reading the paper should refrain from saying various stupid things such as, Why didn’t the paper try X, when the paper actually tried X. But I think it is only reasonable to expect reactions from people who’ve only read the press release, only read the abstract, etc. Especially considering that many news reports do not link to the research article at all, and many research articles (although not the one under consideration here) are behind paywalls. In that case, it’s not a matter of “two wrongs make a right,” it’s a matter of the critic making do with what he or she has.

        • Actually I agree. But with the caveat that if you do say something stupid & then someone points it out, the mistake is all yours.

          In fact, only when you do say something stupid for lack of reading the original do you get called out for it (mostly). You’ve got to say something that makes your not reading the original patently obvious.

          What annoys me then is the “Oh! But I only read the abstract” excuse. If you only read the abstract & ended up saying something stupid, serves you right.

  22. Pingback: Pre- vs. post-publication review and procedural justice | Dynamic Ecology

  23. Pingback: Beyond the Valley of the Trolls « Statistical Modeling, Causal Inference, and Social Science Statistical Modeling, Causal Inference, and Social Science

Comments are closed.