A template for future news stories about scientific breakthroughs

Screen Shot 2016-05-04 at 12.58.50 AM

Screen Shot 2016-05-04 at 12.59.17 AM

Yesterday, in the context of a post about news media puffery of the latest three-headed monstrosity to come out of PPNAS, I promised you a solution. I wrote:

OK, fine, you might say. But what’s a reporter to do? They can’t always call Andrew Gelman at Columbia University for a quote, and they typically won’t have the technical background to evaluate these papers by themselves.

But I do have a suggestion, a template for how reporters can handle PPNAS studies in the future, a template that respects the possibility that these papers can have value.

I’ll share that template in my next post.

And here we are.

I’ll give you three for future news stories about scientific breakthroughs. You’d have Story #1, followed a year later by Story #2a or Story #2b or, if necessary, Story #2c.

Story #1:

Can a spray improve your memory?

A spray can improve your memory, claims a paper just published in the Proceedings of the National Academy of Sciences. Mary Smith and John Lee, psychologists at Miskatonic University, reported a 34% improvement in short-term memory performance for 91 students subject to a chlorine spray, as compared to 88 students given a placebo control. Participants in the studies were randomized to the two groups. Asked about these findings, Smith says, “We suspected the spray could be effective, but we were surprised at how well it worked, and we’re planning further studies to understand the mechanism.”

Sara Yossarian of Harvard Medical School, who has no affiliation with the study, was guarded in her assessment: “This could be real, but I’m not believing it until I see a preregistered replication.” Smith and Lee confirmed that the hypotheses and data analysis plan for their study were not preregistered.

The Proceedings of the National Academy of Sciences is a prestigious journal but in recent years has had its credibility questioned after its publication of several articles with seriously flawed statistical analyses on topics ranging from suicide rates to life expectancy to air rage to hurricane naming.

Do the memory-boosting effects of chlorine imply that you should be spending more time inhaling the spray at your local pool? Smith says, “I can’t say yes for sure. But I don’t recommend you cancel your gym membership.” We’ll wait for the replication.

A year later, Story #2a:

Yes, it seems that a spray can improve your memory.

A spray can improve your memory, claimed a much-publicized paper published last year in the Proceedings of the National Academy of Sciences. Mary Smith and John Lee, psychologists at Miskatonic University, reported a 34% improvement in short-term memory performance for 91 students subject to a chlorine spray, as compared to 88 students given a placebo control. Participants in the studies were randomized to the two groups. Asked about these findings, Smith said, “We suspected the spray could be effective, but we were surprised at how well it worked, and we’re planning further studies to understand the mechanism.”

The Proceedings of the National Academy of Sciences is a prestigious journal but in recent years has had its credibility questioned after its publication of several articles with seriously flawed statistical analyses on topics ranging from suicide rates to life expectancy to air rage to hurricane naming.

Since then, two laboratories have conducted independent replications of the study, and these were just published in the journal Psychology Research Replications. The replication studies in this issue were performed in the U.S. and France, and both showed the expected positive results.

Sara Yossarian of Harvard Medical School, who had no affiliation with the original study or the replications and who expressed skepticism about this research last year, says, “I’m impressed with the preregistered replications. I’m still not sure how effective this treatment will be in the real world—I’d like to see some evidence outside campus and hospital settings—but this is definitely worth further study.”

Do the memory-boosting effects of chlorine imply that you should be spending more time inhaling the spray at your local pool? Smith says, “I can’t say yes for sure. But I don’t recommend you cancel your gym membership.” At the very least, this is one more reason to do some daily laps.

Or, Story #2b:

Follow-up: No evidence that a spray can improve your memory.

A spray can improve your memory, claimed a much-publicized paper published last year in the Proceedings of the National Academy of Sciences. Mary Smith and John Lee, psychologists at Miskatonic University, reported a 34% improvement in short-term memory performance for 91 students subject to a chlorine spray, as compared to 88 students given a placebo control. Participants in the studies were randomized to the two groups. Asked about these findings, Smith said, “We suspected the spray could be effective, but we were surprised at how well it worked, and we’re planning further studies to understand the mechanism.”

The Proceedings of the National Academy of Sciences is a prestigious journal but in recent years has had its credibility questioned after its publication of several articles with seriously flawed statistical analyses on topics ranging from suicide rates to life expectancy to air rage to hurricane naming.

At the time of its publication, the conclusions of the chlorine-memory study could only be considered tentative because its hypotheses and data analysis plan for their study had not been not preregistered. Since then, two laboratories have conducted independent replications of the study, and these were just published in the journal Psychology Research Replications. The replication studies in this issue were performed in the U.S. and France, and they showed no pattern of success.

Asked about the failed replications, Smith says, “We thought we found something there, but apparently we were overconfident. We’re glad that other groups went to the trouble to replicate our study, and we plan to start over and think more carefully about how a memory spray should work.”

Sara Yossarian of Harvard Medical School, who had no affiliation with the original study or the replications, says, “The unsuccessful replications confirm my original skepticism. Yes, it’s possible that a spray could improve your memory, but it could also make your memory worse, or have no effect at all. This episode is a reminder not to overreact to preliminary research findings—even if they’re published in a prestigious journal.”

If chlorine really has no memory-boosting effects, does this imply that you should spend less time at your local pool? Smith says, “I don’t recommend you cancel your gym membership. But go for the exercise, not for the chlorine.”

Or, if necessary, Story #2c:

Follow-up: No evidence that a spray can improve your memory.

A spray can improve your memory, claimed a much-publicized paper published last year in the Proceedings of the National Academy of Sciences. Mary Smith and John Lee, psychologists at Miskatonic University, reported a 34% improvement in short-term memory performance for 91 students subject to a chlorine spray, as compared to 88 students given a placebo control. Participants in the studies were randomized to the two groups. Asked about these findings, Smith said, “We suspected the spray could be effective, but we were surprised at how well it worked, and we’re planning further studies to understand the mechanism.”

The Proceedings of the National Academy of Sciences is a prestigious journal but in recent years has had its credibility questioned after its publication of several articles with seriously flawed statistical analyses on topics ranging from suicide rates to life expectancy to air rage to hurricane naming.

At the time of its publication, the conclusions of the chlorine-memory study could only be considered tentative because its hypotheses and data analysis plan for their study had not been not preregistered. Since then, two laboratories have conducted independent replications of the study, and these were just published in the journal Psychology Research Replications. The replication studies in this issue were performed in the U.S. and France, and they showed no pattern of success.

Asked about the failed replications, Smith says, “We stand by our originally-published finding. These so-called replications differed in important ways from our original study, which is why they failed to find any effects.”

Sara Yossarian of Harvard Medical School, who had no affiliation with the original study or the replications, says, “The unsuccessful replications confirm my original skepticism. Yes, it’s possible that a spray could improve your memory, but it could also make your memory worse, or have no effect at all. This episode is a reminder not to overreact to preliminary research findings—even if they’re published in a prestigious journal.”

If chlorine really has no memory-boosting effects, does this imply that you should spend less time at your local pool? Yossarian says there’s no reason for to you cancel your gym membership. “But go for the exercise, not for the chlorine.”

OK, these aren’t the only possible template. In the above stories I’ve emphasized preregistration, but (a) not every study can be replicated so easily or at all, and (b) when we criticize studies, it’s typically not for being non-preregistered but for specific statistical or conceptual flaws. To put it another way, the power-pose paper had many statistical weaknesses, to the extent that I would see no reason to believe it even in the absence of preregistration. The failed replication is just another nail in its coffin. The air-pollution-in-China is a one-time thing and the study can never be repeated—but my criticism is that the statistical analysis is so weak that the conclusions can’t be trusted.

But these sorts of criticisms are hard to make in general, hence I think it’s safest to present a general skepticism in line with PPNAS’s demonstrated willingness to publish unsubstantiated claims garnished with statistically significant p-values.

If you’re writing about a study such as air rage or himmicanes or air pollution in China which can’t be easily replicated, then remove the bits about replication in the above templates and replace with statements such as the researchers’ hypotheses being open-ended and the data being unavailable and that you don’t really believe the claims until the data have been reanalyzed by outside groups.

For example, in Story #1:

Sara Yossarian of Harvard Medical School, who has no affiliation with the study, was guarded in her assessment: “This could be real, but I’m not believing it until I see a reanalysis of the raw data by an independent research group.” Smith and Lee confirmed that the hypotheses and data analysis plan for their study were not preregistered. Their data and detailed experimental protocol are not yet available online.

And then, in the concluding line to Story #1, instead of “We’ll wait for the replication,” you can have:

We’ll wait until other research teams have had had a chance to analyze these data.

I think the above stories represent a first start at a reasonable template that journalists can use when reporting science news. Perhaps the actual journalists out there can refine these.

8 thoughts on “A template for future news stories about scientific breakthroughs

  1. “the power-pose paper had many statistical weaknesses, to the extent that I would see no reason to believe it even in the absence of preregistration.”

    Even in the presence of preregistration? That would be even stronger.

  2. > a reasonable template that journalists can use when reporting science news
    Hmm, not if they are overly dependent on there current source of income.

    Hopefully, there are enough who are not overly dependent on there current source of income to make it noticeable difference by following such advice.

    That takes time, but as Don Rubin used to say, “smart people hate being repeatedly wrong” [but they first need to notice how often they are being wrong].

  3. I like this. I suspect it would be helpful to journalists for you Andrew to post a list of questions that they might ask researchers of various kinds about published studies, like

    – How does the reported effect compare to other effects in the literature of this treatment?
    – How does the reported effect compare to other effects in the literature on this outcome?
    – Was it preregistered?
    – Could it be preregistered?
    – What evidence is there of a causal effect per se?
    – How specific are the implications to the exact testing conditions?
    – …

    For each thing you think of, you might include a link (or paragraph) about why it is important, in nontechnical language.

  4. Clearly, Andrew has gone literary, leaving some of us behind trying to catch up. Miskatonic University–>

    https://en.wikipedia.org/wiki/Miskatonic_University

    “Miskatonic University is a fictional university located in Arkham, a fictitious town in Essex County, Massachusetts. It is named after the Miskatonic River (also fictional). After first appearing in H. P. Lovecraft’s 1922 story ‘Herbert West–Reanimator’, the school appeared in numerous Cthulhu Mythos stories by Lovecraft and other writers.”

    With respect to “Sara Yossarian” is that an anagram for something or other or merely a reference to those of us who believe “Catch 22” is the funniest and greatest American novel ever composed?

    But never mind chlorine sprays, a University of Minnesota researcher is working on a spray to cure most everything mental:

    http://www.startribune.com/minnesota-researcher-brainstorms-a-new-approach-to-treating-alzheimer-s/375770341/

    “Now, 27 years after Frey made a groundbreaking discovery, a new treatment is undergoing a clinical trial in St. Paul that could help not just Alzheimer’s, but Parkinson’s, stroke, traumatic brain injury and, potentially, countless other issues, from mood swings to cravings.”

      • The spray which Frey is using is not chlorine. Frey uses insulin. From the Minneapolis Star Tribune article:

        “He pressed on with his work, and after scientists in other parts of the world caught on, he won his patent in 1997. In 2001, he developed a specific treatment using his intranasal method: sending insulin into the brain.”

        “By getting insulin into the brains of patients with Alzheimer’s, Frey found, energy could spike, improving memory in just 15 minutes without altering glucose levels in the blood.”

        “And, thanks to Frey’s work, researchers in Europe have found that intranasal insulin can sharpen memory in healthy adults, as well. Perhaps one day, we could all carry a spray that we use twice a day to help us live, think and work better.”

  5. Andrew: This post has the wrong title.

    May I suggest a more catchy title: “Three ways a scientific finding report can kill your social media shares and get you fired”.

Leave a Reply to Alex P Cancel reply

Your email address will not be published. Required fields are marked *