Skip to content

So little information to evaluate effects of dietary choices


Paul Alper points to this excellent news article by Aaron Carroll, who tells us how little information is available in studies of diet and public health. Here’s Carroll:

Just a few weeks ago, a study was published in the Journal of Nutrition that many reports in the news media said proved that honey was no better than sugar as a sweetener, and that high-fructose corn syrup was no worse. . . .

Not so fast. A more careful reading of this research would note its methods. The study involved only 55 people, and they were followed for only two weeks on each of the three sweeteners. . . . The truth is that research like this is the norm, not the exception. . . .

Readers often ask me how myths about nutrition get perpetuated and why it’s not possible to do conclusive studies to answer questions about the benefits and harms of what we eat and drink.

Good question. Why is it that supposedly evidence-based health recommendations keep changing?

Carroll continues:

Almost everything we “know” is based on small, flawed studies. . . . This is true not only of the newer work that we see, but also the older research that forms the basis for much of what we already believe to be true. . . .

The honey study is a good example of how research can become misinterpreted. . . . A 2011 systematic review of studies looking at the effects of artificial sweeteners on clinical outcomes identified 53 randomized controlled trials. That sounds like a lot. Unfortunately, only 13 of them lasted for more than a week and involved at least 10 participants. Ten of those 13 trials had a Jadad score — which is a scale from 0 (minimum) to 5 (maximum) to rate the quality of randomized control trials — of 1. This means they were of rather low quality. None of the trials adequately concealed which sweetener participants were receiving. The longest trial was 10 weeks in length.

According to Carroll, that’s it:

This is the sum total of evidence available to us. These are the trials that allow articles, books, television programs and magazines to declare that “honey is healthy” or that “high fructose corn syrup is harmful.” This review didn’t even find the latter to be the case. . . .

My point is not to criticize research on sweeteners. This is the state of nutrition research in general. . . .

I just have one criticism. Carroll writes:

The outcomes people care about most — death and major disease — are actually pretty rare.

Death isn’t so rare. Everyone dies! Something like 1/80 of the population dies every year. The challenge is connecting the death to a possible cause such as diet.

Carroll also talks about the expense and difficulty of doing large controlled studies. Which suggests to me that we should be able to do better in our observational research. I don’t know exactly how to do it, but there should be some useful bridge between available data, on one hand, and experiments with N=55, on the other.

P.S. I followed a link to another post by Carroll which includes this crisp graph:

Screen Shot 2016-04-06 at 10.44.03 AM


  1. jrkrideau says:

    Nina Teicholz in her book The big fat surprise covers, at a general level, many of the issues and resulting quality of dietary research.

    Let’s say it is not encouraging—in fact, it often sounded like the Wild West of research. Much of the “basic” research is funded by corporate interests and it is just maddeningly difficult to do any decent long term controlled research as Carroll points out.

    If Teicholz is correct the entire Mediterranean diet rests essentially on one proselytizer and some very shaky observational data.

    Epidemiological studies may be the best we can do but they are not very sexy compared to the War on Cancer etc. and probably are hard to fund.

    • Paul Alper says:

      “If Teicholz is correct the entire Mediterranean diet rests essentially on one proselytizer and some very shaky observational data.”

      For more on that observation, see

      which states, “This study had 18 authors some of whom served on the board of the Research Foundation on Wine and Nutrition, received support from the California Walnut Commision, the International Nut and Dried Food Council, Nestle, PepsiCo, the Beer and Health Foundation and Danone.”

      Another substantive critique of the Mediterranean diet by way of a NYT article where Dr. Esselstyn, a noted vegan, remarked:

      “those in the Mediterranean diet study still had heart attacks and strokes. So, he said, all the study showed was that ‘the Mediterranean diet and the horrible control diet were able to create disease in people who otherwise did not have it.'”

      And, no discussion of diets would be complete without this obvious historical truth:

      “Let us not forget the famous phrase popular among our forebearers: when it comes to diets, there are really only two: food and no food.”

      • Z says:

        ““those in the Mediterranean diet study still had heart attacks and strokes. So, he said, all the study showed was that ‘the Mediterranean diet and the horrible control diet were able to create disease in people who otherwise did not have it.’”

        This contains a horrible point. The fact that people had strokes while on the Mediterranean diet does not “show” that the Mediterranean diet “created” (or caused) any of those strokes compared to any other diet. (I guess it caused strokes compared to an all-arsenic diet which would kill you by other means before you could have a stroke.) The argument that the control diet in the study was bad might be reasonable, but I’m not inclined to take this guy’s word for it.

  2. mark k says:

    As far as I know there’s not any particular evidence that high fructose corn syrup (which is around 50% fructose and 50% glucose, like table sugar) is worse than sugar. Or honey for that matter which is also basically the same thing.

    Corn syrup is the sugar of choice for manufacturers though. Cutting out high fructose corn syrup would probably lead to lot less simple sugar intake, which is good for most people’s health and appetite control (including me) so it probably deserve some of the it’s bad rep.

    I agree with larger point about nutrition science.

    • It’s one of those awkward arguments that has nutritionists, epidemiologists, economists, and statisticians, all talking past one another.

      HFCS isn’t necessarily bad, in moderation. Except it could be, maybe, but there aren’t enough studies. But the price is cheaper, due in part to subsidies, which increases consumption for constant demand. There is also a nutrition-consumption trend over decades to put more sugar in cheap foods — naturally — they pick the cheapest sugar. Sugar is then sometimes called addictive, specifically HFCS, except that’s observationally equivalent to increased consumption due to lower prices.

      And somewhere amongst all of those is, without a doubt, some high-dimensional nonlinear interaction and feedback mechanism.

  3. BenK says:

    It’s a matter of scale. In a two-week study with 1000 generally healthy people, new onset of disease and death are each rare outcomes.

  4. Clyde Schechter says:

    “Death isn’t so rare. Everyone dies! Something like 1/80 of the population dies every year.”

    Yes, of course. But the kind of dietary advice people would like to have is about how they should eat while they are still young and healthy so as to remain healthy as they get older. If you enroll a cohort of young healthy people in a study, death *during the study observation period* will indeed be too rare to analyze meaningfully. So to look directly at mortality outcomes you either need a gigantic cohort or you need to follow a moderate sized cohort for many decades. Neither approach is practical or affordable, especially since there are no hypotheses that look sufficiently promising at this point to justify that kind of investment.

    Dietary research suffers from many obstacles. It is impossible with existing methods to obtain accurate data on what people actually eat unless you enroll them in a feeding study. But such studies can only be sustained for short periods of time: you can’t lock people up in a metabolic ward forever! Perhaps somebody will invent a micro-sensor that can be bonded to the lingual surface of a molar and that will identify and wirelessly transmit accurate information about food intake, but nothing like that exists now. As noted above, the outcomes of interest would need to be studied in impractically large cohorts or unrealistically long follow-up periods.

    Moreover, much nutritional research is predicated on the assumption that individual nutrients are the proper units of study. But, except for people who ingest lots of nutritional supplements, people don’t eat nutrients. They eat foods. And it may be that there are effects of ingesting particular foods or classes of foods that are different from the “sum” of the effects of their nutrients: there may be interactions among them. Current nutritional research mostly ignores this kind of model.

    Even if we had perfect information about what people eat and the resources to study a large cohort for a long period time, people’s diets change considerably over time, so teasing out the effects of particular aspects of diet that come and go would be quite a challenge.

    In short, I think that for the foreseeable future this situation is not going to improve and we will have nothing to go on but animal studies and poor quality research on humans.

    • Martha (Smith) says:

      The take-away message: Nutritional studies need to be diligent in reporting their limitations and the uncertainty in any conclusion. (Hmm — just like those power pose, himmicane, etc. folks should be, but aren’t.)

  5. Garnett says:

    “…there should be some useful bridge between available data, on one hand, and experiments with N=55, on the other”

    This perspective has motivated scientists to combine animal studies, where invasive experiments are possible, with observational/epidemiological studies in humans. In doing this kind of work we need to be crystal-clear about how the two types of studies, and organisms involved, are similar and how they might be different in the form of thoughtful priors. This is very, very challenging to get past reviewers, but I see a great deal of interest in this approach in biomedicine.

  6. beamish says:

    The mortality rate is more like 1/120 in the US and a little lower in the world as a whole.

  7. Oncodoc says:

    The factuality of dietary studies is a big problem. It has been too many years to dig up the actual study, but I recall someone who went through the garbage of participants in a dietary study and found lots of wrappers from stuff that the subjects denied consuming. Likewise, based on interviews the average American admits to consuming 3-4 lbs of chocolate per year while 8-10 lbs per person is actually sold. Doing a dietary study well requires locking a large number of people into a secure facility for years. Some low quality correlation type studies are the best possible on this earth.
    I do insist on lots of red wine to maintain my vital resveratrol levels and walk around naked to get my vitamin D.

  8. Salomon says:

    If this is so difficult to do, why does it keep being done? Is anything coming out of such studies useful? How about correlations between cholesterol and heart disease?

    • Clyde Schechter says:

      The initial association between cholesterol (subsequently made more specific to low density lipoprotein cholesterol) derives, I believe, from the Framingham study. That study has been actively following its participants since the late 1940’s and is into the second and third generation of participants as well. It has enrolled a large number of participants as well. It is the kind of study I alluded to in my earlier post. But they did not obtain dietary information (at least not originally). So it is an association between blood levels of low density lipoprotein cholesterol and heart disease. This same association has been repeatedly observed in other more recent large (though not as long duration) studies. Studies of heart disease in mid-life are comparatively easy to carry out because the heart disease endpoint is not nearly so rare in this age group as death is.

      If you are thinking about an association between dietary cholesterol intake and heart disease, that was debunked a long time ago. There is a subset of the population with certain mutations for whom this association prevails; but for the population as a whole the two are unrelated. The main reason is that dietary cholesterol intake and blood low density lipoprotein cholesterol levels are not associated with each other. The vast majority of the cholesterol circulating in the blood is synthesized in the body from saturated fat; dietary intake contributes a negligible amount beyond that (in most people).

      So with respect to cholesterol and heart disease, the following two associations have been well established in large cohort studies:

      1. High levels of low density lipoprotein cholesterol are associated with increased risk of heart disease.
      2. High levels of intake of saturated fat over long periods of time are associated with incresaed levels of low density lipoprotein cholesterol.

      Most of the medical profession also accepts that decreasing dietary intake of saturated fat will cause a decline in risk of heart disease. There is a body of evidence for this, but it is no as strong as the two conclusions listed above.

      There is, however, very strong evidence from clinical trials that a combination of reducing dietary saturated fat and taking statin medications leads to both a reduction in blood low density lipoprotein cholesterol levels and reduction in risk of heart disease in people who begin with high levels of low density lipoprotein cholesterol.

  9. Islio says:


    The lack of “factuality” in dietary & nutritional health studies obviously does not hinder the government, medical establishment nor media from confidently issuing endless pronouncements on this subject.

    How is it that all these “experts” are so easily led astray?

    If they are so wrong on nutrition, why trust them in other areas of health & medicine?

    Seem highly likely that most medical research lacks objective “factuality”.

    • Anoneuoid says:

      Yep, most people won’t admit it yet, but the rational thing to do at this point and time is to avoid all people claiming to know how to help you regarding health.

      I go by the principle that its much easier to break things you don’t understand than fix them. But maybe that is only a justification. When I think about all those people with sharp scalpels, concentrated chemicals, and a brain fogged up by statistical significance… I get fear, much more than of any disease.

    • Martha (Smith) says:

      Oh, yes — just do a search on “outcome switching” to scratch the surface.

    • Oncodoc says:

      I think that a certain skepticism is always appropriate, but cynicism should be avoided. The quality of studies vary, and a good way to recognize the good studies is to point out the problem areas.
      When it comes to government proclamations regarding diet, I think a case can be made that the Department of Agriculture has acted as a promoter of some producers like the dairy farmers. The stellar position of dairy products on the famous diet pyramid in the past reflects this. Political and cultural considerations impacts are considerable. I doubt that a vegetarian will ever be Secretary of Agriculture.

      • Keith O’Rourke says:

        My former director (chief of medicine) put it this way – of the stuff done in medicine, 1/3 is pretty much know for sure, 1/3 has evidence and 1/3 has no evidence.

        Today, they might add that of the 1/3 with evidence, only a 1/3 of that is of adequate quality.

        For instance, when they starting tracking “outcome switching” via research ethics board documents, the prevalence of it was surprisingly high. There was a sense that most clinical researchers knew to avoid such poor practices and would.

        Too bad most research practices are not as easily assessed as outcome switching.

  10. jrc says:

    In the past several months, I have read Economics of Nutrition papers that claim:

    Vitamin C improves agricultural productivity for poor farmers.

    Vitamin A protects against the negative effects of tornadoes.

    Poor children who eat more are healthier and taller (ok, I believed this one, but that was literally the entire argument of the paper – top field journal).

    A school feeding program taken up by less than 1/3 of children increased average child weight-for-age by 0.75 sd (half the difference between being born in the USA and being born in sub-Saharan Africa)… even though it somehow couldn’t be detected at the population level.

    Month of birth can have a 0.4 sd effect on child height via health and nutrition environment… or roughly the same effect as being born during the Rwandan genocide (assuming you survived).

    Economic growth does not improve child nutritional status,… you know, despite the fact that stunting rates in the USA are (a statistically-natural) 2.5% and they are 30-50% in the world’s poorest countries.

    …. essentially, much of the economics of nutrition literature is one big example of the statistical significance filter, the exaggeration factor, and the garden of forking paths rolled into one. And so is much of the experimental nutritional biology work upon which it is based (the low-N poor studies mentioned in the post). I mean, you could google Vitamin A and find people that say it fixes everything, and then find a series in the Lancet saying to stop focusing on it.

    But there is also some real progress. Hundreds of thousands of children are not blind today because of Vitamin A fortification/distribution efforts. Iron-defiicency anemia really does hurt women’s productivity at a large scale in poor households around the world and is easily treated (and is being treated, at least more than before). Folic Acid supplementation during pregnancy really does massively reduce incidence of spina bifida. And of course, if you don’t eat, you die.

    So I dunno… yeah, we are way behind where we’d like to be. And yeah, ridiculous stuff is said all the time. But it is a hard field – everything is over-determined and under-identified, nutritional intakes are almost impossible to measure and totally correlated with each other in every statistically difficult way, and the pathways through which nutrition affects health outcomes are complex and difficult to represent mathematically.

    The sad thing is that, so far, the observational/epidemiological work seems at least as bad as the poorly-run experiments. But I agree that we are gonna have to go there. We might end up getting some traction using major public health campaigns in the past (iron fortification, salt iodization). Daniel Lakeland will chime in in a minute explaining how to model all the nutritional inputs at once (but I’ll argue that it isn’t feasible). And there might be some insights from really-careful quasi-experimental designs based on weird characteristics of the world that make people living very similar lives eat diets based on different foods. But mostly, I’m not sure how far we can go with the statistical toolkit we have. What is the causal effect of increasing consumption of a particular micro-nutrient, holding everything else constant? That doesn’t even make sense as a thought experiment… it isn’t like vitamins magically enter you without anything else changing, and it is in no way clear that micro-nutrients from supplements are even absorbed in the same manner as more natural delivery mechanisms (like an apple).

    • I won’t argue that we can model the individual effects of all the nutritional inputs at once, but I’d be happy to chime in with we can model aggregate effects of some small number of classes of nutritional inputs. So for example I’d be in favor of a model that uses:

      1) total calories as a fraction of some experimentally established baseline of calories needed for healthy growth as a function of body size, age, sex

      2) A simplex of how the calories break down on 3 categories Fat, Carbs, Protein, and how different this balance is from the range of things seen to be adequate in developed societies / experiments.

      3) Quantity available of several classes of micro-nutrients. I’m thinking “major micronutrients” like iron, calcium, magnesium, vitD, vitC, vitA, the vitB complex, and a separate category for minor micronutrients like selenium, manganese, whatever

      Now, I’m sure we could sit down with a coffee and think up some way to model growth and health as a function of these 3-4 aggregates if we could get ahold of the data. At least, we could make some progress on things to do. And my concern is that so much effort is spent on searching out individual studies of ivermectin deworming or selenium supplements or whatever to get “statistically significant” results to publish, that no-one is actually doing the *science* because if we wrote down some ideas for a model, and scraped together some meager data, we’d be likely to find “people who get nutrition more like what upper middle class kids in Los Angeles get do better” and “the lack of data makes it statistically impossible to figure out very much more, we need millions of dollars in measurement efforts across the developing world to find out what would be best, but if you gave that info to us, we’d be able to give you actual *science based policy* with uncertainty, but what you want is *smells like science based policy but actually gives money to our cronys in the agriculture industry* “

  11. A.G.McDowell says:

    As in most applications of statistics, there is another source of information available – theoretical knowledge of the underlying mechanism being studied. One example (unfortunately paywalled at least at its primary source) is “F stands for Fructose and Fat” by Lyssiotis and Cantley. This claims to find an underlying difference between the metabolism of fructose and that of glucose which suggests that sources of Fructose – whether sucrose, honey, Agave, or HFCS – should be treated with suspicion. Unfortunately, of course, the real thing we care about is what actually happens when real people eat the stuff and this does require experimental evidence.

    One curious exception to the difficulty of finding statistically satisfactory effects of diet is the effect of vegetable sources of nitrate on athletic performance. See e.g. “The most striking finding of the present investigation was
    the significant improvement in 4- and 16.1-km TT performance
    after the ingestion of a single 0.5-L BR beverage,
    with all nine individuals completing both distances faster
    after BR supplementation.”
    It would appear that even in dedicated athletes who you would expect already have relatively sensible diets the addition of vegetable sources of nitrate increases performance. If statistical plausibility was to be the deciding factor, we would apparently be seeing a massive increase in the widespread consumption of vegetables as described e.g. at (Irish Sports Council technical document on nitrates describing vegetable sources of nitrate and purported mechanism and incidentally why this also suggests that you discontinue antibacterial mouthwashes)

  12. Keith O’Rourke says:

    Someone had recently done a study of biological markers and was trying to relate them to dietary diary records.

    It was a complete mess and the low quality of the dietary diary records seemed obvious.

    All through their talk they kept of trying to get suggestions for any way to salvage the work.

    Anything at all!

    If anything, they should have been writing a short note on how the use of dietary diary records was currently useless.

    The leaders in the field need to start being honest about the limitation of their fields empirical work.

  13. Jon says:

    In comparing honey, table sugar, and high fructose corn syrup, we can look to one other source of information other than randomized control studies—the chemistry of those substances and our knowledge how the body processes the sugars. While this won’t tell us how harmful any of them are, this information suggests that there should be little or no difference between the three as they all continue rather similar percentages of fructose: 40-50%.

    • Martha (Smith) says:

      I would guess that “how the body processes the sugars” would be a crucial factor — e.g., (probably the simplest scenario) if each source of sugar were mixed with water to give the same resulting percentages of sugar vs water, I would suspect the results on the body to be pretty much the same.

      One problem with trying to base conclusions on recorded dietary intake is potentially that e..g, different sugars might be used in different foods, so effects could result from the difference in the other ingredients (or the different proportions of “pure” sugar) in the foods eaten.

Leave a Reply