Question of the week: Will the authors of a controversial new study apologize to busy statistician Don Berry for wasting his time reading and responding to their flawed article?

Posted on by

Aaron Carroll shoots down a politically-loaded claim about cancer survival. Lots of useful background from science reporter Sharon Begley:

With the United States spending more on healthcare than any other country — $2.5 trillion, or just over $8,000 per capita, in 2009 — the question has long been, is it worth it? At least for spending on cancer, a controversial new study answers with an emphatic “yes.” . . .

Experts shown an advance copy of the paper by Reuters argued that the tricky statistics of cancer outcomes tripped up the authors.

“This study is pure folly,” said biostatistician Dr. Don Berry of MD Anderson Cancer Center in Houston. “It’s completely misguided and it’s dangerous. Not only are the authors’ analyses flawed but their conclusions are also wrong.”

Ouch. Arguably the study shouldn’t be getting any coverage at all, but given that it’s in the news, it’s good to see it get shot down. I wonder if the authors will respond to Don Berry and say they’re sorry for making so many mistakes and wasting his time.

P.S. Just to take this out of a left-right political rut, I’ve heard that the sorts of mistakes made in this article—attributing inappropriate policy implications to unadjusted comparisons—have also been done by anti-smoking activists when arguing on the purported economic and health effects of cigarettes.

22 thoughts on “Question of the week: Will the authors of a controversial new study apologize to busy statistician Don Berry for wasting his time reading and responding to their flawed article?”

  1. Andrew,
    Do you have any specific articles on the inappropriate policy implications for the health effects of cigarettes. I have done extensive research in the past on the policy implications of cigarettes, but haven’t seen those specific papers (maybe I was not careful enough). Any guidance would be much appreciated.

  2. But Don is right it is dangerous and so very important to address.

    Yes the _miss-use_ of time is frustrating as it was for me doing this effort http://statmodeling.stat.columbia.edu/2010/04/when_engineers/ to prevent the same kind of error statisticains all too often make.

    “the design and analysis apparently failed to consider the _usual_ and expected lag in a screening effect here (perhaps worth counting the number of statisticians in the supplementary material given)”

  3. Rena Conti is nobody’s fool. I’m surprised they weren’t asked about the article. Mortality data has its own peculiarities. The appendix explains why they did both; I can’t say why the presentation seemed to focus on the survival data:

    “A limitation of examining population mortality rates alone is that these rates will be affected if cancer incidence changes for reasons other than improved detection, including changes in prevention or risk factors for disease. Taken together, however, analyses of mortality rates and survival produce more robust conclusions. If US data show both survival gains and mortality reductions, this is likely to reflect real improvements in health; the alternative interpretation requires a scenario in which the diagnosis rate is rising in the US while the actual number of cancer cases is falling.”

    I suppose you could also have improved diagnosis and a delta in reclassification of deaths, although one would expect the sign to go the other way (having known that they have prostate cancer, their death could be called secondary to cancer or treatment when without that knowledge it wouldn’t have been) or a delta in competing risks.

    Their results show some of the weirdness associated with pure differences in differences. For example, stomach cancer had the US losing lots of ground (it started the period out way ahead) and isn’t broken out separately in the survival data (the chart says it was non-significant, but that doesn’t mean small). Oddly enough, colorectal cancer is broken out and is weird: huge US gain in mortality with a significant loss in survival time. I don’t know why the p is so large for that one.

    The result does appear driven mostly by prostate cancer, which seems dangerous.

    • “…If US data show both survival gains and mortality reductions, this is likely to reflect real improvements in health; the alternative interpretation requires a scenario in which the diagnosis rate is rising in the US while the actual number of cancer cases is falling.”

      This is phrased as though the “alternative interpretation” is considered unlikely but in fact this is believed to be the case at least with stomach and colon cancer in women, and stomach and lung cancers in men. For example http://www.cancer.org/acs/groups/content/@epidemiologysurveilance/documents/document/acspc-032006.pdf shows the age-adjusted death rate from various cancers among women. Nobody thinks the steady decrease in those cancers, which started decades ago, is entirely due to improved medical treatment: it’s that fewer people are getting those cancers. But also, nobody thinks diagnosis probability is going anywhere but up. In short, it is known or at least generally believed that for some cancers the diagnosis rate is rising while the number of cases is falling. That’s not the “alternative interpretation” it’s the expected one!

      • Good graph, likewise, the equivalent for men. The male/female differences in size and timing of peaks seem to make sense.

        Was there an explanation somewhere why lung&bronchus wasn’t included in the original article, given that they seem to account for ~40% of the deaths? Of course, maybe data in this are related.It seems hard to get adult smoking rates much below 20% without special efforts in a few places.

        I don’t know the equivalents of these graphs for Europe, but on principle, it seems odd to ignore a big chunk of the data.
        How might the interactions among death rates work? Suppose tobacco were retroactively disappeared 40 years ago?
        Would the death rates for other causes go up or down? It might be interesting to look at UT statistics, as the US state with lowest smoking rate.

      • Ok, but stomach and colorectal cancer actually go the other direction in their estimates (Europe gained more ground). Thier result seems to really just be driven off prostate and breast cancer. What basis is there for thinking that _differential_ changes in risk patterns account for changes in mortality? The decades old decline in breast cancer mortality starts in 1990. When did tamoxifen start widespread adujant use, the late 80’s?

        • “What basis is there for thinking that _differential_ changes in risk patterns account for changes in mortality?”

          I don’t think that, I’m just *asking* if anything is known, as I have no idea of the extent to which these death rates are positively/negatively correlated or independent. If I recall right, some think there is a link between smoking an prostrate cancer.

          If a study ignores long/bronchial cancer, 40%+ of the cancer deaths, I’d sure want to know it was independent before I drew conclusions about the rest of it. For instance, if the rate of auto deaths went up for every age group, I’d guess the death rates for cancer would go down. That would be an independent cause, but would kill people before they could die of something else.

        • Smoking is associated with a few cancers and with alcohol consumption etc. – also associated with a few cancers, so I’m guessing lots of potential for correlation.

  4. You should have a look at healthnewsreviews.org, a website that monitors health/medical reporting in about 65 media. Reports are graded by experts on a set of 10 criteria and given a 1-5-star rating. The reviews and ratings are fed back to the media and the individual reporters with the aim of encouraging better science reporting.
    The site has been supported by the Foundation for Informed Medial Decision Making (fimdm.org), whose name has recently been changed to Informed Medical Decisions Foundation.
    full disclosure: I am a member of the Board of Directors of the Foundation.

  5. Is anyone skeptical about one researcher criticizing another when they haven’t had a publication since ’08 and really just focus on their (slightly obvious politically leaning) blog. I have yet to read the paper that is being criticized, but it seems one party is attacking rather loudly.

      • Yeah he is fine, but Carroll is not a great source for this type of stuff, and he tends to yell louder than most and therefore get his name out more than most.

    • Jonathan:

      My prior is that Don is right (90%+)

      OK its argument by authority – but read the arguments and don’t assume publication rate should be highly related to sensible and purposefull

    • Who are you talking about? I think both Don Berry and Aaron Carroll are active researchers. A quick Pubmed search confirmed both publish quite a bit.

  6. Not talking about Berry, he I believe.

    Also publication rate does not mean too much to me depending on the source (again not talking about Berry).

    Sorry if I was being vague.

  7. My first thought was the risk stuff that Gigerenzer published. Comparing European and US survival rates makes zero sense.

    http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2733256/

    Also, given the present health systems in the two countries I’d immediately wonder how the US could give vastly superior _average_ treatment in terms of cost. If there was an expensive treatment that actually dramatically improved on existing treatments, the UK system would fairly quickly mandate its introduction across England and Wales (and Scotland would do likewise). So the US could only really outdo this if i) they got the treatment much cheaper (whereas US consumers tend to get identical treatmant at much higher cost) or ii) got coverage nationwide more quickly (rather than, say, getting the treatment to those with adequate insurance).

Comments are closed.