Absence of evidence is evidence of alcohol?

Arho Toikka writes:

I ran across what I feel is a pretty peculiar use of statistical significance and p-values, and thought I’d send you a message and see if you find it interesting too or if I’m just confused about something:

I read a news story about a study that showed that previous studies on moderate alcohol use and its association with lower mortality have been false – biased by erroneous selection of the control group (as they are observational studies). Being a risk-averse moderate alcohol user and knowing how news media treats press releases, I had to go and read the study.

Here’s the reference:
Stockwell et al. (2016). Do “Moderate” Drinkers Have Reduced Mortality Risk? A Systematic Review and Meta-Analysis of Alcohol Consumption and All-Cause Mortality. Journal of Studies on Alcohol and Drugs, 77(2), 185–198 (2016).

They put forward three meta-analytic models to claim:
Estimates of mortality risk from alcohol are significantly altered by study design and characteristics. Meta-analyses adjusting for these factors find that low-volume alcohol consumption has no net mortality benefit compared with lifetime abstention or occasional drinking.

Thus, for each of the three strategies, evidence for reduced mortality risk among low-volume drinkers largely disappeared once design and methodological issues were controlled for directly in the analysis or by study selection.

One of their strategies was to do a stratified meta-analysis for four groups, one with the correct control group and three with different biases (table 4 in the paper). The risk ratios for moderate drinking were 0.9 (vs true abstainers), 0.91 (vs abstainers + those who quit drinking + those who drink very little), 0.86 (vs. abstainers + those who quit drinking) and 0.86 (vs. abstainers + those who drink very little).

At face value, it seems that changing the control group did little to change the effect. But there are only 13 studies with the correct control group! And this brings the p-value to .19, widens the confidence interval to 0.72 to 1.06.

To me, their claims do not follow from the data. At best, it seems that they should say that this strategy shows little or no difference between the studies with different control groups, and too few studies with the correct one are in to conclude much.

Am I missing something? Isn’t this arguing absence of evidence as evidence of absence? Is this an appropriate use of frequentist inference? Is this a common use?

My reply: I guess I’d recommend that the researchers step back and be willing to acknowledge uncertainty. Even if results aren’t statistically significant, they can still inform decisions. But I don’t have any clean answers here. I’m not sure exactly what I would do in this situation.

9 thoughts on “Absence of evidence is evidence of alcohol?

  1. These alcohol studies are very political. I submitted one a few years back and we had a discussion about it. It had similar issues. in essence the researchers saw a typical “J” shaped curve that is described in the literature with a minimum risk for relatively moderate consumption. Then, they broke up the data into sub-categories until the uncertainties were large enough to lose statistical significance, then they claimed that since they couldn’t find statistical significance in the sub-groups that the effect must be zero….

    See that discussion here:

    http://statmodeling.stat.columbia.edu/2015/09/30/an-unconvincing-analysis-claiming-to-debunk-the-health-benefits-of-moderate-drinking/

  2. I’ll tell ya a story.

    A few years ago I had some heart rhythm problems and my cardiologist recommended I drink a glass of red wine every day. About a year after that, I was working in France for a year and as a condition of employment I had to go for a medical checkup. The doctor asked if I drank alcohol. I said, yes, I drank about one glass of red wine every evening with dinner, on the advice of my cardiologist. She told me this was bad for my liver and she recommended I stop.

    I don’t know exactly what was going on—maybe just variation between doctors—but my guess was that the French doctor, upon hearing me say I drink one glass a day, assumed that I was underreporting, that I really drink like a fish, and that was using the cardiologist’s recommendation as an excuse. So I’m guessing she said what she said as a way of telling me to moderate my heavy drinking. But I can’t be sure; I didn’t think of asking her at the time.

    • (From what I recall from about 15 years ago) The French have lower cardiac death rates and higher liver cancer rates compared to the USA. In America they treat for the cardiac rates, in France they treat for liver cancer rates.

    • The clinicians I used to work with told me they just doubled the amount reported.

      When David Speigelhalter looked into this about a year ago he argued that one alcohol unit a day was medicine (beneficial) and any more was poison (harmful).

      For even large effects, epidemiological studies leave a lot of uncertainty that is very hard to puzzle out – small effects either way I would take with a grain of salt – at least until salt is found to be harmful at that dose ;-)

      I do agree with Sander Greenland who commented about 20 years ago that it is totally irresponsible to encourage patients to consume alcohol based on epidemiological studies which such weak and uncertain results.

      If its really one glass a day and not a chore/imposition – it should be an easy decision.

    • > I don’t know exactly what was going on—maybe just variation between doctors

      Or maybe it is possible to do something that is good for your heart and bad for your liver and you have a trade-off on your hands. Eating beets, almonds, and spinach can have health benefits in a lot of areas and health costs in others, like getting kidney stones from the oxalates. The proper decision of how much to eat of high-oxalate food depends on how prone you are to oxalate-caused kidney stones.

  3. I found a statement from Boston University on this systematic review.

    They claim that its results are not valid on selection bias grounds.

    Furthermore, that there is information on animal studies that was not taken into account when they made the inferences.

    Unluckily, I could not download the article to give a more informed opinion.

    http://www.bu.edu/alcohol-forum/critique-183-an-unusual-analysis-of-the-association-of-alcohol-consumption-with-mortality-24-march-2016/

Leave a Reply

Your email address will not be published. Required fields are marked *