A link from the comments here took me to the wonderfully named Barfblog and a report by Don Schaffner on some reporting.
First, the background: A university in England issued a press release saying that “Food picked up just a few seconds after being dropped is less likely to contain bacteria than if it is left for longer periods of time . . . The findings suggest there may be some scientific basis to the ‘5 second rule’ – the urban myth about it being fine to eat food that has only had contact with the floor for five seconds or less. Although people have long followed the 5 second rule, until now it was unclear whether it actually helped.” According to the press release, the study was “undertaken by final year Biology students” and led by a professor of microbiology.
The press release hit the big time, hitting NPR, Slate, Forbes, the Daily News, etc etc. Some typical headlines:
“5-second rule backed up by science” — Atlanta Journal Constitution
“Eating food off the floor may be OK, scientist says” — CNET
“Scientists confirm dad’s common sense: 5-second rule totally legit”
OK, that last one was from the Christian Science Monitor, a publication that I don’t think anyone will take very seriously when it comes to health issues.
Second, the take-home point from Schaffner:
If you don’t have any pathogens on your kitchen floor, it doesn’t matter how long food sits there. If you do have pathogens on your kitchen floor, you get more of them on wet food than dry food. But in my considered opinion, the five-second rule is nonsense. I’m a scientist, I’ll keep an open mind. I know what some people in my lab will be working on this summer. . . .
Third, the rant from Don Schaffner on barfblog:
I [Scaffner] can tell when something is a big news story.
First, I read about it in my news feed from one or more sources. Second, friends and family send it to me. By these two criteria, the recent news about the five second rule qualifies as a big news story. . . . And it’s a story, or a press release, not a study.
The press release is apparently based on a PowerPoint presentation. The study has not undergone any sort of peer review, as far as I know. Science by press release is something that really bugs me. It’s damned hard to do research. It’s even harder to get that research published in the peer-reviewed literature. And when reputable news outlets publish university press releases without even editing them, that does a disservice to everyone; the readers, the news outlet, and even the university researchers. . . .
A review of the slide set shows a number of problems with the study. The researchers present their data as per cent transfer. As my lab has shown repeatedly, through our own peer-reviewed research, when you study cross-contamination and present the results as percentage transfer, those data are not normally distributed. A logarithmic transformation appears to be suitable for converting percentage transfer data to a normal distribution. This is important because any statistics you do on the results generally assume the data to be normally distributed. If you don’t verify this assumption first, you may conclude things that aren’t true.
The next problem with the study is that the authors appear to have only performed three replicates for most of the conditions studied. Again, as my own peer-reviewed research has shown, the nature of cross-contamination is such that the data are highly variable. In our experience you need 20 to 30 replicates to reasonably truly characterize the variability in logarithmically transformed percent transfer data.
Our research has also shown that the most significant variable influencing cross-contamination appears to be moisture. This is not surprising. Bacteria need moisture to move from one location to another. When conditions are dry, it’s much less likely that a cell will be transferred.
Another problem that peer-reviewers generally pick up, is an awareness (or lack thereof) of knowledge of the pre-existing literature. Research on the five-second rule is not new. I’m aware of at least three groups that schaffnerhave worked in this area. Although it’s not peer-reviewed, the television show MythBusters has considered this issue. Paul Dawson at Clemson has also done research on the five-second rule. Dawson’s research has been peer-reviewed and was published in the Journal of Applied Microbiology. Hans Blaschek and colleagues were, as far as I know, the first lab to ever study this.
When I first read this, I was like, Yeah, you go guy! If only all the journalists did it as well as Mary Beth Breckenridge of the Beacon Journal, in a news article headlined, “Study supports five-second rule, but should you? Probably not”:
A new study appears to validate what every 12-year-old knows: If you drop food on the floor, you have five seconds until it becomes contaminated. Biology students at Aston University in Birmingham, England, tested the time-honored five-second rule and claim to have found some truth to it. The faster you pick food up off the floor, they discovered, the less likely it is to contain bacteria. . . .
But don’t go picking fallen Fritos out of the rug just yet.
The study contradicts findings of earlier research at Clemson University, where scientists tested how fast Salmonella Typhimurium bacteria made their way from flooring surfaces to bologna and bread. It happened instantly, the researchers found.
What’s more, the British study apparently hasn’t been published yet in a scientific journal, noted Jeffrey T. LeJeune, a food safety expert at the Ohio Agricultural Research and Development Center in Wooster Township.
Since the data aren’t available to other researchers, he said, there’s no way to replicate the study or determine whether the results are legitimate. “I would be very skeptically cautious about the results, and even more about the interpretation,” he said. . . .
But then I got a bit worried. What exactly is the take-home message? It can’t just be, “don’t report a study that hasn’t been peer-reviewed,” since (a) even if a study is published in a peer-reviewed journal, it could be crap (recall all those papers published in Psychological Science), and (b) if a topic is sufficiently important, it could well be newsworthy even before the grind of the peer review process.
This particular study does seem shaky, though: a student project that is not backed up by shared data or a preprint. The press release seems a bit irresponsible: “Although people have long followed the 5 second rule, until now it was unclear whether it actually helped,” which implies that now all is clear. But journalists should know better than to trust a press release! Don’t they teach them that in day 1 of journalism school?? The reports typically do express some skepticism, for example the NPR report says, “The team hasn’t published the data yet. So the findings are still preliminary and need to be confirmed” and later on quotes a biologist stating an opposite position. Even so, though, it seems like all these news outlets are taking the press release a bit too uncritically.
Some of this is simple envy: I’d love for my research to be discussed on NPR and I’m sure Don Schaffner wouldn’t mind this sort of exposure either. But it does seem to me that this sort of science-reporting-by-press-release creates the worst sort of incentives for researchers. I don’t blame the university researcher for promoting his students’ project (his quote: “The findings of this study will bring some light relief to those who have been employing the five-second rule for years, despite a general consensus that it is purely a myth”) but I do blame the reporting system for hyping this sort of thing, which seems like the flip side of the notorious proclivity of media organizations for scare stories. (As Jonathan Schoenfeld and John Ioannidis found, it seems like just about everything has been said to cause cancer at one time or another.)
P.S. This all got my attention not because I care about the so-called five-second rule but because I was attracted by the name of the barfblog.