Skip to content
 

“Groundbreaking or Definitive? Journals Need to Pick One”

Sanjay Srivastava writes:

As long as a journal pursues a strategy of publishing “wow” studies, it will inevitably contain more unreplicable findings and unsupportable conclusions than equally rigorous but more “boring” journals. Groundbreaking will always be higher-risk. And definitive will be the territory of journals that publish meta-analyses and reviews. . . .

Most conclusions, even those in peer-reviewed papers in rigorous journals, should be regarded as tentative at best; but press releases and other public communication rarely convey that. . . .

His message to all of us:

Our standard response to a paper in Science, Nature, or Psychological Science should be “wow, that’ll be really interesting if it replicates.” And in our teaching and our engagement with the press and public, we need to make clear why that is the most enthusiastic response we can justify.

9 Comments

  1. Farrel says:

    That is why I repeatedly state, “Be careful. The null hypothesis is truer than you think”

  2. K? O'Rourke says:

    It would be interesting to speculate how long this takes to be widely accepted and practiced.

    This ancient paper suggests it will take a generation or two.

    http://www.jameslindlibrary.org/illustrating/records/meta-analysis-in-medical-research-strong-encouragement-for-high/whole_articles

    p.s. It was written as a response to fellow graduate stats students who were convinced it was very wrong headed of me to assist researchers doing (mega-silly) meta-analysis.

  3. CM says:

    This is already what I think when I read a paper in Science or Nature or Psych Science.

  4. lost grad student says:

    It seems he is talking about political science and economics journals …. and their communities; sometimes when I go to job talks in political science I get the felling that most people don’t really care a lot about the world (in my cases, I attend presentations mostly about other, typically poor countries). The community mostly care about a nice stories, a “theoretical question” ; careful description, for instance, is just for looses, so that sometimes they are trying to make causal inferences and/or write “theories” about patterns that don’t even exist! is quite ridiculous … they want poetry over facts. Of course, many wil deny that and anyway, not everybody is like this but… at the end of the day nice stories rules over science and facts; it looks quite an infantile attitude for university professors.

  5. zbicyclist says:

    Aren’t there a substantial number of journals that are neither groundbreaking nor definitive?

  6. K?, my money’s on never. But I hope I’m wrong.

    zbicyclist, good point. I was thinking of journals that try to be both, so I was focused on the maximum. But it’s possible for some journals to be neither (and in fact a good thing, if you believe in things like replication).

    Andrew, thanks for linking.

    • K? O'Rourke says:

      It has not quite been never in clinical research (~ 20 years)

      I am not as up to date but there are requirements with penalties for not registering RCTs and peer reviewed on line journals that do focus on publishing any completed RCT.

      Still many statisticians do not think about the need to replicate or even ask their clients “how many studies like this have you or others done?”

      How many have heard, “I would not have come to see you about this study but the p_value was small” ;-)

  7. David Duffy says:

    An editor of a certain journal described the higher impact journals (Nature stable etc) as the “tabloids” ;)

  8. [...] comments #2,3,4 here. I guess that’s why Science and Nature are known as “the tabloids.” As the commenter writes, “you can’t have people look at too [...]