“Groundbreaking or Definitive? Journals Need to Pick One”

Sanjay Srivastava writes:

As long as a journal pursues a strategy of publishing “wow” studies, it will inevitably contain more unreplicable findings and unsupportable conclusions than equally rigorous but more “boring” journals. Groundbreaking will always be higher-risk. And definitive will be the territory of journals that publish meta-analyses and reviews. . . .

Most conclusions, even those in peer-reviewed papers in rigorous journals, should be regarded as tentative at best; but press releases and other public communication rarely convey that. . . .

His message to all of us:

Our standard response to a paper in Science, Nature, or Psychological Science should be “wow, that’ll be really interesting if it replicates.” And in our teaching and our engagement with the press and public, we need to make clear why that is the most enthusiastic response we can justify.

9 thoughts on ““Groundbreaking or Definitive? Journals Need to Pick One”

  1. It would be interesting to speculate how long this takes to be widely accepted and practiced.

    This ancient paper suggests it will take a generation or two.

    http://www.jameslindlibrary.org/illustrating/records/meta-analysis-in-medical-research-strong-encouragement-for-high/whole_articles

    p.s. It was written as a response to fellow graduate stats students who were convinced it was very wrong headed of me to assist researchers doing (mega-silly) meta-analysis.

  2. It seems he is talking about political science and economics journals …. and their communities; sometimes when I go to job talks in political science I get the felling that most people don’t really care a lot about the world (in my cases, I attend presentations mostly about other, typically poor countries). The community mostly care about a nice stories, a “theoretical question” ; careful description, for instance, is just for looses, so that sometimes they are trying to make causal inferences and/or write “theories” about patterns that don’t even exist! is quite ridiculous … they want poetry over facts. Of course, many wil deny that and anyway, not everybody is like this but… at the end of the day nice stories rules over science and facts; it looks quite an infantile attitude for university professors.

  3. K?, my money’s on never. But I hope I’m wrong.

    zbicyclist, good point. I was thinking of journals that try to be both, so I was focused on the maximum. But it’s possible for some journals to be neither (and in fact a good thing, if you believe in things like replication).

    Andrew, thanks for linking.

    • It has not quite been never in clinical research (~ 20 years)

      I am not as up to date but there are requirements with penalties for not registering RCTs and peer reviewed on line journals that do focus on publishing any completed RCT.

      Still many statisticians do not think about the need to replicate or even ask their clients “how many studies like this have you or others done?”

      How many have heard, “I would not have come to see you about this study but the p_value was small” ;-)

  4. Pingback: The tabloids strike again « Statistical Modeling, Causal Inference, and Social Science

Comments are closed.