Ticket to Baaaaarf

A link from the comments here took me to the wonderfully named Barfblog and a report by Don Schaffner on some reporting.

First, the background: A university in England issued a press release saying that “Food picked up just a few seconds after being dropped is less likely to contain bacteria than if it is left for longer periods of time . . . The findings suggest there may be some scientific basis to the ‘5 second rule’ – the urban myth about it being fine to eat food that has only had contact with the floor for five seconds or less. Although people have long followed the 5 second rule, until now it was unclear whether it actually helped.” According to the press release, the study was “undertaken by final year Biology students” and led by a professor of microbiology.

The press release hit the big time, hitting NPR, Slate, Forbes, the Daily News, etc etc. Some typical headlines:

“5-second rule backed up by science” — Atlanta Journal Constitution

“Eating food off the floor may be OK, scientist says” — CNET

“Scientists confirm dad’s common sense: 5-second rule totally legit”

OK, that last one was from the Christian Science Monitor, a publication that I don’t think anyone will take very seriously when it comes to health issues.

Second, the take-home point from Schaffner:

If you don’t have any pathogens on your kitchen floor, it doesn’t matter how long food sits there. If you do have pathogens on your kitchen floor, you get more of them on wet food than dry food. But in my considered opinion, the five-second rule is nonsense. I’m a scientist, I’ll keep an open mind. I know what some people in my lab will be working on this summer. . . .

Third, the rant from Don Schaffner on barfblog:

I [Scaffner] can tell when something is a big news story.

First, I read about it in my news feed from one or more sources. Second, friends and family send it to me. By these two criteria, the recent news about the five second rule qualifies as a big news story. . . . And it’s a story, or a press release, not a study.

The press release is apparently based on a PowerPoint presentation. The study has not undergone any sort of peer review, as far as I know. Science by press release is something that really bugs me. It’s damned hard to do research. It’s even harder to get that research published in the peer-reviewed literature. And when reputable news outlets publish university press releases without even editing them, that does a disservice to everyone; the readers, the news outlet, and even the university researchers. . . .

A review of the slide set shows a number of problems with the study. The researchers present their data as per cent transfer. As my lab has shown repeatedly, through our own peer-reviewed research, when you study cross-contamination and present the results as percentage transfer, those data are not normally distributed. A logarithmic transformation appears to be suitable for converting percentage transfer data to a normal distribution. This is important because any statistics you do on the results generally assume the data to be normally distributed. If you don’t verify this assumption first, you may conclude things that aren’t true.

The next problem with the study is that the authors appear to have only performed three replicates for most of the conditions studied. Again, as my own peer-reviewed research has shown, the nature of cross-contamination is such that the data are highly variable. In our experience you need 20 to 30 replicates to reasonably truly characterize the variability in logarithmically transformed percent transfer data.

Our research has also shown that the most significant variable influencing cross-contamination appears to be moisture. This is not surprising. Bacteria need moisture to move from one location to another. When conditions are dry, it’s much less likely that a cell will be transferred.

Another problem that peer-reviewers generally pick up, is an awareness (or lack thereof) of knowledge of the pre-existing literature. Research on the five-second rule is not new. I’m aware of at least three groups that schaffnerhave worked in this area. Although it’s not peer-reviewed, the television show MythBusters has considered this issue. Paul Dawson at Clemson has also done research on the five-second rule. Dawson’s research has been peer-reviewed and was published in the Journal of Applied Microbiology. Hans Blaschek and colleagues were, as far as I know, the first lab to ever study this.

When I first read this, I was like, Yeah, you go guy! If only all the journalists did it as well as Mary Beth Breckenridge of the Beacon Journal, in a news article headlined, “Study supports five-second rule, but should you? Probably not”:

A new study appears to validate what every 12-year-old knows: If you drop food on the floor, you have five seconds until it becomes contaminated. Biology students at Aston University in Birmingham, England, tested the time-honored five-second rule and claim to have found some truth to it. The faster you pick food up off the floor, they discovered, the less likely it is to contain bacteria. . . .

But don’t go picking fallen Fritos out of the rug just yet.
The study contradicts findings of earlier research at Clemson University, where scientists tested how fast Salmonella Typhimurium bacteria made their way from flooring surfaces to bologna and bread. It happened instantly, the researchers found.
What’s more, the British study apparently hasn’t been published yet in a scientific journal, noted Jeffrey T. LeJeune, a food safety expert at the Ohio Agricultural Research and Development Center in Wooster Township.
Since the data aren’t available to other researchers, he said, there’s no way to replicate the study or determine whether the results are legitimate. “I would be very skeptically cautious about the results, and even more about the interpretation,” he said. . . .

But then I got a bit worried. What exactly is the take-home message? It can’t just be, “don’t report a study that hasn’t been peer-reviewed,” since (a) even if a study is published in a peer-reviewed journal, it could be crap (recall all those papers published in Psychological Science), and (b) if a topic is sufficiently important, it could well be newsworthy even before the grind of the peer review process.

This particular study does seem shaky, though: a student project that is not backed up by shared data or a preprint. The press release seems a bit irresponsible: “Although people have long followed the 5 second rule, until now it was unclear whether it actually helped,” which implies that now all is clear. But journalists should know better than to trust a press release! Don’t they teach them that in day 1 of journalism school?? The reports typically do express some skepticism, for example the NPR report says, “The team hasn’t published the data yet. So the findings are still preliminary and need to be confirmed” and later on quotes a biologist stating an opposite position. Even so, though, it seems like all these news outlets are taking the press release a bit too uncritically.

Some of this is simple envy: I’d love for my research to be discussed on NPR and I’m sure Don Schaffner wouldn’t mind this sort of exposure either. But it does seem to me that this sort of science-reporting-by-press-release creates the worst sort of incentives for researchers. I don’t blame the university researcher for promoting his students’ project (his quote: “The findings of this study will bring some light relief to those who have been employing the five-second rule for years, despite a general consensus that it is purely a myth”) but I do blame the reporting system for hyping this sort of thing, which seems like the flip side of the notorious proclivity of media organizations for scare stories. (As Jonathan Schoenfeld and John Ioannidis found, it seems like just about everything has been said to cause cancer at one time or another.)

P.S. This all got my attention not because I care about the so-called five-second rule but because I was attracted by the name of the barfblog.

29 thoughts on “Ticket to Baaaaarf

  1. All studies done on the ‘5-second rule’ are at the present time the scientific equivalent of troll bait.

    To design the appropriate study is difficult but not impossible; but the study is not being designed or
    conducted appropriately. The current studies use inappropriate proxies for the key variables.

    1. Cultured bacteria are being used to contaminate environmental surfaces. They come from an inappropriate
    growth state and are distributed in an unrealistic fashion.

    2. The best studies use multiple surfaces and different adhesion/adsorption and drying times. Not all do.

    3. The best studies use a variety of foodstuffs and somewhat realistic food handling.

    4. The foods would normally then be consumed – this is a complicated process. Most studies simply aim to
    recover the bacteria by culture. They do not consider culturability vs viability and infectivity. They use
    very basic proxies for infectious dose and do not consider the impact of the food matrix.

    5. There is huge inter-host variability in susceptibility/infectious dose. This is never considered.

    To do the study ‘most accurately’ – food items of differing kinds should be exposed to household surfaces for
    different durations and then consumed. Rates of illness should be recorded and any statistical difference
    in rate and severity of illness explained by the duration of the food on the household surface. This is not
    a difficult study to plan, but it would be large and difficult to execute ethically.

    A related study which might be better managed could involve animals. One could start with a cage of mice that
    are ill, gastrointestinally with a defined pathogen, and a cage that are not ill. Then food of several sorts
    could be exposed to one of these two cages for a defined period of time and returned to a cage of otherwise
    healthy animals. Rates and severity of illness can be recorded by molecular detection of infection and physiological
    monitoring. A relationship between exposure time and transmission of infection would be the product.
    An additional variable might be exposure only to a cage that had held infected animals vs a cage currently
    housing infected animals. Conducting this several times for different defined pathogens, a few strains of
    mice, etc, would give a rather robust association. Still a large study, but a big step in the right direction.

    These sort of studies constitute heavy lifting in science. Very little of our practical biology is so well-grounded.
    A single very well-done study, such as the ones that related the kill-time/temperature/pH of pathogens in food, can
    redefine the parameters of safe food handling, for example. One might think that huge safety margins are the right
    way to go, but for every increase in the margin of safety, food that doesn’t meet the standard must be discarded (ton upon
    ton of food waste), additional cooking times destroy nutrition, and so on. This is critically important practical
    science.

    • I think this illustrates a great example of a situation where external validity is so difficult to achieve (because there really is no “typical” food, nor surface, nor susceptibility dose etc.) that the usefulness of any study is highly questionable. No matter how good the repeatibility or internal validity.

      In a lot of disciplines external validity is the big hidden question mark.

  2. More important than any “sterile” study of how much gets on what when is how much more likely you are to get sick. In that regard, who is better off: the person who eats off the floor or the person who doesn’t but touches his or her eyes often? (I’d bet the former because the gut is built to destroy what can hurt you while the eye is a channel for infection.) What if the person who eats off the floor washes hands often and the person who doesn’t eat off the floor rarely washes hands? (I’d again bet the floor eater is better off than a non-hand washer.)

    An example I love is in my gym. People are gun-shy about touching things in the bathroom area because, of course, they think everything in there is “dirty”, which really means associated in the mind with poop and pee. Toilet seats are pretty clean, especially if you simply wipe them off, and of course you can always just wash your hands. But how to explain the people who take a face towel from the rack inside the bathroom? I know that flushing expels small amounts over a large area. You ingest some in the air but is it a good idea to wipe your face with it. I use towels from outside the room.

  3. National Geographic did better as did Scientific American.

    Actually, this is a good event for Internet media researchers to study, since it originates from a few press releases, and spread across the net.
    Google: 5-second rule aston university or five-second rule aston university

    One might study:
    a) The headline … often written by someone other than the reporter
    b) The extent to which the story went beyond the press release. Some at least talked to other reserchers.
    c) The balance or lack thereof in the story.
    (This sort of thing inspired What to do about poor science reporting years ago.

    d) Include blogs
    e) Examine the propagation pattern through the net

        • Andrew,

          It’s true that Christian Science is a religion that doesn’t “believe” in modern medicine. But the Monitor is actually a really good news source. From Wiki:

          “The paper has been known for avoiding sensationalism, producing a “distinctive brand of nonhysterical journalism”.”

          Here’s a link to some health reporting, and in particular one of those stories it would be easy to Fox-News-ify: Swine Flu vaccine. Sure, they quote some skeptics of vaccines – but they give one sentence to what sounds like a real anti-vax person, another sentence to someone worried about vaccines and mutations, and then they come straight back to:

          http://www.csmonitor.com/USA/2009/1007/p02s01-usgn.html

          *
          “There’s no doubt there’s some benefit” to vaccines, William Schaffner, an epidemiologist at Vanderbilt University, told American Medical News earlier this year.

          In aggregate, even a small reduction in flu cases could have a huge impact on overall transmission rates across the country, says Dr. Gerrard of the University of Michigan. That is the main reason why the $3 billion vaccination program, despite variations in effectiveness, has become the country’s primary strategy to fight the swine flu.
          *

          You could quibble with some of the wording, in that they don’t say “Flu shots clearly work!”. But then again, the flu shot is actually of varying effectiveness from year to year (they use last year’s flu in Asia as a base, I believe) and person to person and strain to strain, and so they note the importance of the marginal prophylactic effect on the aggregate.

          I’d say you might be judging a book (newspaper) by its cover here.

        • The paper was started by a “Christian Scientist” and due to this early founding grant and soforth has to have “Christian Science” in the name, but it is not actually espousing the beliefs of “Christian Science” and typically produces pretty good REAL science journalism. As I said, it’s unfortunate.

  4. I don’t agree with Andrew’s argument (a) i.e. “even if a study is published in a peer-reviewed journal, it could be crap”

    Sure. So what? Shouldn’t we go by probabilities? So long as we have reason to believe that peer reviewed stuff is, on average, more reliable (or higher quality) than non-peer reviewed stuff I don’t see any problem in sticking to peer reviewed stuff. Unless there are overwhelming reasons to do otherwise. Which I don’t see here.

    • Rahul,

      Do you have “reason to believe that peer reviewed stuff is, on average, more reliable (or higher quality) than non-peer reviewed stuff”? Sure, on the surface it seems like peer review must help somehow (or else why would everyone be doing it), but I have searched for evidence and could find none.

      • Biggest evidence is the fact that we are still doing it. The alternative for me is to believe hundreds of thousands of smart scientists and researchers are wasting time at a dumb ritual. I’m no academia hater. These are smart people. I refuse to believe that they will continue doing something of zero value.

        No doubt peer review has its problems. But I still think it is better than no review.

        • There are huge monetary incentives to “keep doing it” and these cause huge survivorship biases (labs that didn’t bother to publish peer reviewed stuff and just shoved things wholesale onto ArXive or whatever would have their funding wither and would go out of business).

          So while there may be reasons to believe that peer review is overall helpful for improving science quality, “we’re still doing it” doesn’t seem like one of those reasons to me.

        • I see. Unfortunately I already “believe hundreds of thousands of smart scientists and researchers are wasting time at a dumb ritual” when it comes to strawman NHST. Once one accepts this type of failure can happen in one instance it opens the doors to many more. It was not easy for me to accept, it took me ~2 years of study to convince myself that one really did not make sense. I suspect peer review is in the same boat.

        • Another pointless ritual I think is the calculation of pH. The use of negative logarithms just makes concentrations hard to understand and confusing to talk about.

        • The reason for pH rather than [H+] is that you don’t need to write 10^-7. In the early 1900s negative exponents were considered ugly. It is really that dumb.

        • Doesn’t matter why it was adopted; it’s well-justified in information-theoretic terms. In the vast majority of cases where one wants to communicate H+ concentration, the “10” and the negative sign are redundant. Only the pH part of the expression conveys unknown information, so we might as well just write that.

        • Well this is case of user friendliness vs encoding info efficiency. You choose the latter. Any doctor you go to will have learned less other stuff because they were trying to figure out wtf is going on with logs due to that. Most won’t have an intuitive grasp of what it means by the end anyway.

          If there where two societies exactly the same except one used negative logs for proton concentration and the other not. I think the latter would win out.

        • I don’t really understand your position, but this is just one of a thousand cuts that will result in you dying earlier than need be.

        • Sorry, once I posted that I realized it might sound like a threat. “This” referred to opportunity cost of doctors learning to understand logarithms to understand pH, when they will basically never need to use that concept otherwise.

        • “Sorry, once I posted that I realized it might sound like a threat. “This” referred to opportunity cost of doctors learning to understand logarithms to understand pH, when they will basically never need to use that concept otherwise.”

          I’m not sure that is true. Doctors should have an understanding of odds and log odds among other things. There is a well known BMJ article explaining logarithms for doctors for instance.

  5. Pingback: "The Europeans and Australians were too eager to believe in renal denervation" « Statistical Modeling, Causal Inference, and Social Science Statistical Modeling, Causal Inference, and Social Science

  6. Pingback: Data-based ways of getting a job - Statistical Modeling, Causal Inference, and Social Science

Leave a Reply to Corey Cancel reply

Your email address will not be published. Required fields are marked *