Alzheimer’s Mouse research on the Orient Express

Paul Alper sends along an article from Joy Victory at Health News Review, shooting down a bunch of newspaper headlines (“Extra virgin olive oil staves off Alzheimer’s, preserves memory, new study shows” from USA Today, the only marginally better “Can extra-virgin olive oil preserve memory and prevent Alzheimer’s?” from the Atlanta Journal-Constitution, and the better but still misleading “Temple finds olive oil is good for the brain — in mice” from the Philadelphia Inquirer) which were based on a university’s misleading press release. That’s a story we’ve heard before.

The clickbait also made its way into traditionally respected outlets Newsweek and Voice of America. And NPR, kinda.

Here’s Joy Victory:

It’s pretty great clickbait—a common, devastating disease cured by something many of us already have in our pantries! . . . To deconstruct how this went off the rails, let’s start with the university news release sent to journalists: “Temple study: Extra-virgin olive oil preserves memory & protects brain against Alzheimer’s.”

That’s a headline that surely got journalists’ attention. It’s not until after two very long opening paragraphs extolling the virtues of the nearly magical powers of extra virgin olive oil that we find out who, exactly this was tested on.

Mice. . . .

It’s never mentioned in the news release, but this hypothesis was tested on only 22 mice, just 10 of which got the olive oil rich diet, and 12 of which got a standard diet.

As in: The sample size here is so small that we can’t be very sure what the results are telling us.

The release does at least make it pretty clear that these were also genetically modified mice, who had their DNA fracked with so that they developed “three key characteristics of the disease: memory impairment, amyloid plaques, and neurofibrillary tangles.”

Victory continues:

Stating the obvious, here, but genetically modified mice are a far, far cry from people. . . .

In fact, even drugs that apparently do a great job of getting rid of amyloid in thousands of actual humans don’t seem to have much effect on the symptoms of Alzheimer’s disease. . . .

There are other limitations of the study that should have been in the news release, but we’ll stop here.

I looked briefly at the published research article and am concerned about forking paths, type M errors, and type S errors. Put briefly, I doubt such strong results would show up in a replication of this study.

“No actual interviewing took place.”

Victory continues her article with a discussion of the news coverage:

With a news release like this, journalists were primed to do a poor job writing about the study.

To its credit, the Inquirer was upfront about this being a mouse study. . . . Yet, the story included no independent experts to offer some perspective—something we saw across the board in the stories we read.

USA Today and the Atlanta Journal Constitution didn’t even bother to disclose that all the quotes in their stories come directly from the news release. As in: No actual interviewing took place.

Ouch.

Victory concludes:

This is all great news for Temple’s PR team—their messaging made it out to the public with very little editing. But this isn’t great news for the public.

Countless people across the country who have loved ones with dementia likely read these stories with a whole lot of hope.

What they get instead is a whole lot of hype.

It’s ok to publish raw data and speculation—just present them as such.

I have no objection to this research being published—indeed, a key reason for having a scientific publication process in the first place is so that these sort of preliminary, exploratory results can be shared, and others can replicate.

The key is to present the results as data rather than as strong conclusions, and to avoid this sort of strong conclusion:

Taken together, our findings support a beneficial effect of EVOO consumption on all major features of the AD phenotype (behavioral deficits, synaptic pathology, Aβ and tau neuropathology), and demonstrate that autophagy activation is the mechanism underlying these biological actions.

That’s from the published paper. Can they really conclude all that from 22 genetically modified mice and a pile of bar graphs and p-values? I don’t think so.

The ethics question

Is it unethical for a team of researchers to overstate their results? Is it unethical for a university public relations office to hype a statistical study about 22 genetically modified mice with the following inaccurate headline: “Extra-virgin olive oil preserves memory & protects brain against Alzheimer’s”?

I feel like everyone’s passing the buck here:
– Newspapers don’t have the resources to spend on science reporting;
– Science writers are busy and feel under pressure to present good news, maybe even attach themselves to the kind of “breakthrough” they can write about in more detail;
– Lots of pressure to get “clicks”;
– Public relations offices are judged on media exposure, not accuracy;
– Academic researchers know that it’s hard to publish in top journals without making big claims.

Somewhere along the line, though, something unethical seems to be happening. And it seems worth discussing in a context such as this: legitimate (I assume) research on an important topic that is just being majorly hyped. This is not the Brian Wansink show, it’s not power pose or himmicanes or ovulation and clothing or beauty and sex ratio or all the other ridiculous noise-mining exercises that we’ve used to sharpen our intuition about understanding and communicating variation. It’s potentially real research, being unethically hyped. But the hyping is done in stages. It’s a Murder on the Orient Express situation in which . . . . SPOILER ALERT! . . . all the players are guilty.

And all this is happening in a media environment that has less journalism and more public relations than in the past. Fewer eyes on the street, as Jane Jacobs would say.

36 thoughts on “Alzheimer’s Mouse research on the Orient Express

  1. They say the person was blinded during testing but are silent about during analysis or housing (feeding, changing cages, weighing, etc). Then look at those sample sizes… I see n = 12/10; n = 11/10; n = 8/8; n = 4/4; n = 6/6; n = 5/5. They say:

    Starting at 6 months of age, mice were randomized into two groups: one fed with standard diet (CTR, n = 12), the other with EVOO-enriched diet (EVOO, n = 10) for 6 months.

    […]

    After perfusion brains were removed, gently rinsed in cold 0.9% phosphate-buffered saline and immediately dissected in two halves. One half was immediately stored at −80°C for biochemistry; the other half was fixed in 4% paraformaldehyde in phosphate-buffered saline, pH7.4 for immunohistochemistry studies.

    So, anything besides n = 12/10 and n = 6/5 needs to be explained, perhaps I missed it.

  2. Wow, this indeed looks like an extreme example of how journalists rely on university press releases quite uncritically.

    Andrew, you write:
    “– Newspapers don’t have the resources to spend on science reporting;
    – Science writers are busy and feel under pressure to present good news, maybe even attach themselves to the kind of “breakthrough” they can write about in more detail;”

    This is probably true, but luckily, this is not inevitable, journalists can choose a different approach. I encountered one example on one of the major German news sites (Spiegel Online), where a news article reported very critically about a paper published in Science about the relation between migration and temperature (http://science.sciencemag.org/content/358/6370/1610)

    The journalist – upon reading the paper in Science – reached out to other researchers and collected their opinions and wrote about that. These became headline news on the news site for several hours. This looks like a pretty good example of how it should be done to me.

    Here’s the Google translated news article:
    https://translate.google.com/translate?hl=de&sl=de&tl=en&u=http%3A%2F%2Fwww.spiegel.de%2Fwissenschaft%2Fnatur%2Fmehr-fluechtlinge-durch-klimawandel-asyl-studie-entsetzt-wissenschaftler-a-1184640.html

    Here’s a bit more details.
    http://axelbojanowski.de/ausfuehrliche-kritiken-an-der-asyl-prognose-in-science/#more-804

    Whatever is correct here, I like the approach that the journalist chose: very important topic, it did not smell proper to him, he checked, and wrote about it.

  3. “…this hypothesis was tested on only 22 mice….”

    I find this type of comment irritating and I demand clarification (because we get these kinds of comments all the time).

    What exactly is enough mice? What exactly is “enough data”?

    Would it matter if the analysis, conducted with classical methods, was suitably and sufficiently powered with 22 mice?
    Would it matter if the analysis, conducted with Bayesian methods, incorporated other sources of knowledge and published studies?
    Would it matter if the study was conducted on an extremely rare disease? Or in vulnerable populations that are extremely difficult to recruit?

    None of this seems to matter to the skeptical reader. The “gut test” says that 22 mice isn’t enough. Neither is 23 or 28. What about 50 mice?
    It sounds like the Strom Thurmond principle: I don’t know what a big enough sample size is, but I’ll know it when I see it.

    The only logic that I can think of is that, in certain cases, small samples are based on highly motivated and non-representative participants so that the small sample won’t generalize well to the target population. Obviously, this is irrelevant for cloned mice, but this is the best explanation I can think of for human studies. But if we believe this is true, we could model these enrollment timepoint effects on the outcome.

    • Garnet:

      I don’t think it’s appropriate for you to “demand clarification” from someone who expresses skepticism about the headline, “Extra virgin olive oil staves off Alzheimer’s, preserves memory, new study shows,” based on an experiment on 22 mice. I think you should be demanding clarification from the person who wrote that headline, the person who wrote the associated news article, the public relations office that promoted the claim, the editors who published the journal article, and the authors of the journal article. They’re the ones making strong claims based on very little evidence. I think the burden of proof should be on the people who make these over-the-top claims, not on a journalist who expresses reasonable skepticism.

      • I guess my morning coffee took over my typing hands. The point of the post is well taken and an important lesson for us all.

        However I, and anyone who has published in scientific research, have heard the same comment over and over: “the sample size is too small.” Indeed, this comment is repeated several times in the blog post. What does that mean?

        • Sample size is a question of economy of research for which the bottom line is how to accelerate getting less wrong about things we think we know. Kinda of a how long does a piece of string need to be – out of context.

          One of the most convincing studies I analysed had 6 dogs in the exposed and 6 in the treated group. The means were 6 SDs apart and the next step in research would fail if the effect did not persist – so direct replication was not felt necessary.

          Now, the experimenter was kicked out of the statistical consulting office of the stats dept they first went to for help. So I do get your point.

        • Any sample size (greater than zero) can be useful for some purposes. As Seth Roberts used to point out, many important medical discoveries were based on, or at least originated with, a single researcher experimenting on himself.

          22 mice could be enough to be pretty convincing. If 11 mice got the treatment and X happened to all of them, and 11 mice didn’t get the treatment and not-X happened to all of them, that would be at least strongly suggestive. Indeed, sample size wouldn’t even be the thing I’d look at, in that case: I’d rather see the same experiment performed with only, say, 12 mice (6 treated, 6 untreated) at a different institution, than to see the original research team do the same thing with 60 mice or something.

          But I feel like this misses the forest for the trees. The problem isn’t just that the sample size is too small, or that the experiment was performed on mice rather than humans but extrapolated to humans, or that the mice are genetically engineered in a way that may or may not be directly relevant to human Alzheimers sufferers, or any of the other things Joy Victory (great name) mentions. It’s all of those, and a whole bunch more.

        • “It’s all of those, and a whole bunch more.”

          +1

          And when people say, “the sample size is too small,” they often mean, “The sample size is to small given [things like the measure, the design, the variance, the conclusion stated, etc., etc]

        • You know, I get all of that, and it is the basis for the Strom Thurmond test of sample size requirements. But sample size requirements are, in principle, based on a reasonably well defined calculus. If the sample size calculations are done properly, then the statement “The sample is too small” is meaningless, though the other points you make may be correct. If the reviewer is skeptical given these other issues, then they should just say so, instead of “The sample size is too small.”

          Ultimately, though, I’m convinced that reviewers simply don’t believe the sample size calculations in the power analysis sections of papers/proposals, and prefer the Strom Thurmond test. In light of that, I wonder why we even bother with formal sample size calculations.

        • Martha, Garnett:

          Beyond all this, the low sample size indicates a lack of seriousness in the project. If it’s really true that “Extra virgin olive oil staves off Alzheimer’s, preserves memory,” then this is a big deal, no? Maybe worth the trouble to gather more convincing evidence.

        • Andrew:

          I agree, but I can’t help but ask:

          Is a person who studies drug effects with small sample sizes in an extremely rare disease any less serious?

        • Garnet said, “Ultimately, though, I’m convinced that reviewers simply don’t believe the sample size calculations in the power analysis sections of papers/proposals, and prefer the Strom Thurmond test. In light of that, I wonder why we even bother with formal sample size calculations.”

          Perhaps in some fields it is common practice to include power calculations — but I’ve not seen them often, and when I have, they are often based on something like “small”, “medium” or “large” power definitions, rather than on practical significance for the particular question being studied.

  4. What exactly is the ethical problem here? The ultimate outcome, maybe, is that some group of newspaper readers will start adding more olive oil to their diet, believing that this will reduce Alzheimer’s risk. I doubt that this is harmful to anyone — we’d be in different territory if the claim was that punching people prevents Alzheimer’s or something, but this is pretty benign.

    There’s also a pretty big difference between “overhyped” and “false.” I’d say it’s unethical to willfully or negligently pass along false health claims (e.g., vaccines cause autism) but I don’t see an ethical problem with passing along speculative claims (unless there’s something inherently problematic contained in those speculative claims).

    Another way of thinking about this is that researchers often are not “overstating” their claims, relative to their own sincerely held beliefs. The guys that ran the olive oil study (probably) had a reasonably strong prior that it would work, it apparently did work (though in a small sample), and so they rationally/ethically/whatever have a sincerely held posterior belief that it works. And so they write up the results in a way that reflects those sincere beliefs. This is only overstating relative to the way a disinterested third party would interpret the results, but I don’t think you can fault someone for writing up conclusions that reflect sincerely-held beliefs. (If there’s a lapse anywhere, I think it’s when reviewers/editors who ought to act as that disinterested third party fail to check those claims).

    • Yeah, I thought the larger narrative here (missed by many reporting on it) was that the Mediterranean diet seems to be reduce Alzheimer’s and these researchers are looking into the mechanism for why that might be happening, in which case their results, while overblown in the media, are more of a continuation of prior work than some revolutionary new claim that olive oil is a magic elixir against Alzherimer’s.

      https://www.mayoclinic.org/diseases-conditions/alzheimers-disease/expert-answers/alzheimers-disease/faq-20058062

    • The guys that ran the olive oil study (probably) had a reasonably strong prior that it would work, it apparently did work (though in a small sample), and so they rationally/ethically/whatever have a sincerely held posterior belief that it works.

      This is NHST thinking again. In no way is it rational to conclude the treatment works from a study like this.

      Why not conclude the effect they saw is due to bias during the analysis (eg check out the sample size variations for each study I pointed out above)?

    • Somehow this got posted while I was still trying to write it (klutzy fingers again?), so here’s another try:

      Joe said,

      1. “I don’t see an ethical problem with passing along speculative claims (unless there’s something inherently problematic contained in those speculative claims).”

      I think it’s OK to pass along speculative claims *as long as* they are clearly stated as speculative claims. However, my understanding is that in this case they were presented as scientific results, not speculative claims.

      2. “The guys that ran the olive oil study (probably) had a reasonably strong prior that it would work, it apparently did work (though in a small sample), and so they rationally/ethically/whatever have a sincerely held posterior belief that it works.”

      I’m not convinced that the conclusion was either rational or ethical (I am not accusing them of doing anything deliberately unethical, but of perhaps behaving unethically out of ignorance. It happens all to often, regrettably.)

      3. “Another way of thinking about this is that researchers often are not “overstating” their claims, relative to their own sincerely held beliefs.”

      That’s fine as long as the claims are stated as their own sincerely held beliefs. But that’s different from stating them as a scientific conclusion.

      3.”This is only overstating relative to the way a disinterested third party would interpret the results”

      Scientific results are supposed to be the conclusions that a disinterested third party would make.

    • Joe, please confirm that you’re geunine in your response; if so, I will take the time to respond. I think you might instead be doing a clever parody or indulging in some trolling, in which case I think you’ve done an excellent job that needs no response.

      • This is Joe. Yes, I’m sincere (although for some reason my responses keep getting swallowed). I don’t see an ethical problem here. To elaborate, and respond to Martha, I’ll give an analogy.

        I’m a bad cook. I make lots of mistakes. I think we can agree that making mistakes while cooking does not, in and of itself, present an ethical problem. If I overcook some fish and then serve it to my guests, am I really to be ethically faulted for that?

        On the other hand, if I dangerously undercook some chicken, then we start to enter different territory. If I know it’s undercooked and might give my guests salmonella, but I serve it to them anyway, then we have a clear ethical problem. It was ethically wrong of me to do that. But what if I *think* I cooked it appropriately? Then it’s hard to assign an ethical fault to my action — we all make mistakes. But maybe, we can make an argument that I knew I was doing something dangerous and so I had an extra obligation to ensure doneness.

        In any case, I think this research is *much* more similar to the overcooked fish scenario (it offends the tastes of someone like Andrew) than either of the undercooked chicken scenarios (it might actually hurt someone).

        • Joe:

          I think that the claims coming from bad research practices are the statistical equivalent of stating 2 + 2 = 5. Yes, if people start saying 2 + 2 = 5, this will offend my taste. It’s also wasting a lot of people’s time, it’s wasting government funds, it’s sucking up some of the attention that could be going to real science, and it’s polluting the channels of scientific communication.

          Regarding ethics etc., see my post on Clarke’s law.

        • But it isn’t the equivalent of 2+2=5. It’s not even close to that. We know that the claim 2+2 =5 is wrong. We don’t know, one way or the other, if olive oil does something for Alzheimer’s. Maybe it does — maybe olive oil is the magic bullet. And, if you sincerely believe that to be true, then I think you have, if anything, an ethical obligation to share that news (even if you happen to be wrong). Of course it would be better if the research design were stronger. But that’s true regardless of the strength of the design we’re discussing.

        • Joe:

          The 2+2=5 equivalent is not “Extra virgin olive oil staves off Alzheimer’s, preserves memory.” The 2+2=5 equivalent is “Extra virgin olive oil staves off Alzheimer’s, preserves memory, new study shows.” If they wanted to say, “Extra virgin olive oil staves off Alzheimer’s, preserves memory, some scientists believe,” I’d have no problem with that.

        • I’m not quite sure what the difference between “new study shows” and “some scientists believe” is. Studies only show what scientists believe they show.

        • Joe:

          You demonstrate “new studies show” with scientific evidence. You demonstrate “some scientists believe” with interviews. I don’t care if Daryl Bem believes in ESP. I would care if he did experiments providing convincing evidence for ESP, but he didn’t.

        • There are many ethical components to shoddy medical research, eg:

          1) Performing unnecessary medical procedures on animals
          2) Consuming funding that could have been used on helpful research (or anything else)
          3) Generating distrust of science in the layperson
          4) Driving people who would have done good research away (eg someone who knows about multiple comparisons will publish less than someone who doesn’t)
          5) Misuse of taxpayer dollars, thus lowering trust in the state and tending to destabilize your society
          6) Inappropriate medical treatments may follow from the health claims leading to personal injury or death

    • I wrote a reply to Joe’s comment last night, but it apparently got lost, so I’ll try again.

      “Another way of thinking about this is that researchers often are not “overstating” their claims, relative to their own sincerely held beliefs.”

      Just because a belief is sincere doesn’t mean it is scientific.

      “The guys that ran the olive oil study (probably) had a reasonably strong prior that it would work, it apparently did work (though in a small sample),”

      If they indeed used a prior, good scientific practice requires that they justify it.

      “and so they rationally/ethically/whatever have a sincerely held posterior belief that it works.””

      Given the above, I’m not convinced about the “rational” nor the “ethical”.

      ” This is only overstating relative to the way a disinterested third party would interpret the results”

      If something is “overstating relative to the way a disinterested third party would interpret the results”, then it is not good science. Good science needs to be convincing to a disinterested third party.

        • Doing good science is not equivalent to acting ethically — but doing good science requires acting ethically.

          Doing bad science is not equivalent to acting unethically, but in many cases doing (and especially publishing) bad science is acting unethically.

    • Joe,

      I did not comment earlier on your last statement, “(If there’s a lapse anywhere, I think it’s when reviewers/editors who ought to act as that disinterested third party fail to check those claims).”, but will do so now.

      Are you aware that reviewers/editors are usually not paid for their work as reviewers/editors? The usual standard of publishing is that authors are responsible for their work. (I do think the system would be better if we could change it so that reviewers/editors have the time, resources, and other incentives to do more critical reviewing. But we’re not there yet.)

  5. I blame Uncle Sam for blindly shoveling money at any research founded on the beta amyloid model. Something important might have been stumbled upon here – e.g. olive oil alters the gut microbiota which in turn ameliorated some brain inflammatory response – but no; they can’t let curiosity lead them to question their money-maker. They gotta shake it, shake, shake it.

    • Am I reading that paper correctly? It looks like n-sizes of 4 in each group. No it seems that CTR, n = 12 CO, n = 10. Of course, the control group is the same control group reported in the study that Andrew links to. “The control mice used in this study have been previously described” (Ref 15).

      So this is a single study split into two papers or the Canola Oil study was some kind of follow up or afterthought to the Olive Oil study?

  6. I know without looking at the article that the authors are either Spanish or Italian or some combination of the above. These two countries are the leading producers of olive oil.

    • So, just to provide data re this, all three have Italian sounding last names, two of them are at Temple University in Pennsylvania USA and the middle author is from Rome, Italy. One suspects that they all probably grew up in Italy and did their graduate work there, then the PI moved to Temple U and took the student with them or some such thing.

      Looking at the last author “Domenico Pratico” (usually senior/PI author in biomedicine publications) his publication record is all about oxidative stress, lipid oxidation, Vitamin E, and “Mediterranean diet” related issues.

      https://www.ncbi.nlm.nih.gov/pubmed/?term=domenico+pratico

Leave a Reply to Martha (Smith) Cancel reply

Your email address will not be published. Required fields are marked *