“Behind a cancer-treatment firm’s rosy survival claims”

Brett Keller points to a recent news article by Sharon Begley and Robin Respaut:

A lot of doctors, hospitals and other healthcare providers in the United States decline to treat people who can’t pay, or have inadequate insurance, among other reasons. What sets CTCA [Cancer Treatment Centers of America] apart is that rejecting certain patients and, even more, culling some of its patients from its survival data lets the company tout in ads and post on its website patient outcomes that look dramatically better than they would if the company treated all comers. These are the rosy survival numbers . . .

Details:

CTCA reports on its website that the percentage of its patients who are alive after six months, a year, 18 months and longer regularly tops national figures. For instance, 60 percent of its non-small-cell lung cancer patients are alive at six months, CTCA says, compared to 38 percent nationally. And 64 percent of its prostate cancer patients are alive at three years, versus 38 percent nationally.

Such claims are misleading, according to nine experts in cancer and medical statistics whom Reuters asked to review CTCA’s survival numbers and its statistical methodology.

The experts were unanimous that CTCA’s patients are different from the patients the company compares them to, in a way that skews their survival data. It has relatively few elderly patients, even though cancer is a disease of the aged. It has almost none who are uninsured or covered by Medicaid – patients who tend to die sooner if they develop cancer and who are comparatively numerous in national statistics. . . . Accepting only selected patients and calculating survival outcomes from only some of them “is a huge bias and gives an enormous advantage to CTCA,” said biostatistician Donald Berry . . .

What I really like about this article is how it combines quantitative information with qualitative interviews:

Carolyn Holmes, a former CTCA oncology information specialist in Tulsa, Oklahoma, said she and others routinely tried to turn away people who “were the wrong demographic” because they were less likely to have an insurance policy that CTCA preferred. Holmes said she would try to “let those people down easy.” . . .

The ads also challenge viewers to “compare our treatment results to national averages.” Doing so, on the company’s website, shows that CTCA’s reported survival outcomes regularly beat those averages.

Experts in medical data who reviewed CTCA’s claims for Reuters say those claims are suspect because of what they called deviations from best practices in statistics – in particular, comparing its carefully selected patients to those nationwide.

“It makes their data look better than it is,” said Robert Strawderman, professor and chairman of biostatistics at the University of Rochester. “So the comparisons used to suggest that CTCA has better survival rates are pretty meaningless.”

This is horrible! Of course, we do something similar at Columbia—we accept only the very best students—but that’s a bit different, I think, in that we are providing an educational experience that is designed to work with the most-prepared students. In contrast, I doubt that there’s any particular reason for CTCA to restrict its cancer treatments to the least-sick patients.

24 thoughts on ““Behind a cancer-treatment firm’s rosy survival claims”

  1. I don’t see a specific problem with the statistics here. It seems to me that any unethicality is taking place upstream, when the decision to treat people or not based on their financial resources is taken. What CTCA is saying is, “If you can afford to come here, these are your plausible outcomes.”

    If you were to read an article saying that people who stayed at a specific $1200-a-night luxury resort were happier than other tourists, you would probably think “yep, sounds like a nice place, but I’ll bet it’s expensive.” I’m not sure how this is fundamentally qualitatively different, unless people really imagine that all places that treat cancer in the United States have admission processes that are blind to financial considerations and other social status factors.

    Of course, being an Old European and therefore a dangerous socialist, I don’t like the idea of survival being dependent on money (although I’d bet that even in Europe, comfortably-off middle-class people live a lot longer after cancer treatment than public housing tenants). But I’m not sure it’s fair to focus on an incidental factor (the reporting of their survival rates) rather than the underlying cause.

    • It depends whether one considers causal inference as being part of statistics or not. Their marketing is implying that the correlation between better patient outcomes and attendance at CTCA is a causal effect attributable to CTCA’s care.

      @Rahul – I’m not convinced 3rd party metrics will solve this. Here we’re lucky that the selection effect is documented, but in many cases it may not be.

    • The article also mentions that the numbers reported by CTCA are based on 4 types of cancer and fewer than 100 patients for each type. This at a time when CTCA was treating thousands of patients. So we do not know how representative their sample was. Maybe it was just people who did well.

      How can you judge the statistics they offer without seeing their methodology and datasets?

    • Their statistics might be okay, but they’re comparing their statistics to more general numbers and touting the difference. It would be like if Columbia said that they have the best professors because their students are more likely than the general population to get a PhD. Their students performance is more likely an indication of the selection process of becoming a student than because their professors are exceptional.

  2. This is one good reason why third party metrics are what we should be trusting. Never trust too much what a firm says about itself.

    Andrew: Which part is horrible? The fact that they reject “hard” cases? Or the way they mis-portray their survival metrics?

    • CTCA is a heavy local advertiser (Chicago). It’s disappointing and fraudulent, but hardly surprising, that they mis-portray their survival metrics. They show up in the same advertising pods with the personal injury lawyers and the discount lasik surgery people. Here’s the wikipedia page on the founder/chairman: http://en.wikipedia.org/wiki/Richard_J._Stephenson

      Personally, I don’t find it offensive that their restrict themselves to “easy” cases. Maybe that’s all they are competent for? If they aren’t in the same league as the Mayo Clinic [etc.], maybe they shouldn’t pretend they are.

  3. I still don’t understand what’s misleading about this. Conditional on accepting you, the statistics are correct, right? Of course, it might be true that your life expectancy would be higher at some other place which has lower unconditional survival rates, but what we *should* be asking *every* institution for is a conditional survival rate, not an unconditional one. It’s not that CTCA is misleading, it’s that everyone else is!

    • If all CTCA did is claim that 64 percent of its prostate cancer patients are alive at three years, it would not be misleading. It would be information. But, it is incredibly misleading to then compare this survival rate (64%) to the national average (38%) and then implicitly tell consumers that they are more likely to survive cancer if they choose CTCA as their treatment center. That is a bogus comparison. Based on the interview information, CTCA patients are more likely to survive because of their demographic profile and certain characteristics of the cancer itself. If CTCA wants to compare its survival rate to the national average it should be a national average of people who are similar to the people treated at CTCA. That would be fair. The comparison cited above is bogus.

      • I don’t disagree with anything that you say. But I also don’t see what’s misleading. If people don’t want to make sensible comparisons, that’s not my problem. If I hit .400 in Little League, I can certainly say, “Compare my batting average with the MLB batting average of .260.” If people don’t want to make the adjustment for the relevant population, why is that my problem? To take a more relevant example: in the health care debate US mortality rates are constantly being compared to mortality rates in other countries as an index of the quality of the healthcare system. I assume you agree with me that this ia almost completely nonsensical without some adjustment for population composition and other non-health-care-system effects. But to say it is to understand it. So it isn’t misleading.

        • Asymmetric information is the real problem here. CTCA has detailed information about the specifics of what it’s patient population is like, and is therefore in a position to do the proper comparison, pretty much no-one else has this information and so no-one else *can* compare properly. The advertisement is misleading because it implies that a straightforward comparison is justified.

        • I agree, but…. CTCA doesn’t know how to adjust the numbers for the other patients either. In essence, no one should ever care about the average rate at facility X… so the daa everywhere else is worthless as well. What you want to know is your own condition expectation at each possibility facility, possibly adjusted for cost. There is no shorthand for that either in the CTCA stats or in anybody else’s.

          I’m not saying that CTCA might not be trying to mislead people. But they’re really in no position to do so. It’s like a car repair place who said: “Our average repair costs $300.” No one would be misled by that when their front end was bashed in, whether or not they made the comparison that the average car repair costs $500. At least I don’t think anybody who thought about for 2 seconds would. And if it’s not worth two seconds of thought you deserve what you get.

        • Also, this, from the news article:

          What sets CTCA apart is that rejecting certain patients and, even more, culling some of its patients from its survival data lets the company tout in ads and post on its website patient outcomes that look dramatically better than they would if the company treated all comers. These are the rosy survival numbers that attract people like the Hilborns.

          Observational data (such as raw country-level mortality rates) are subject to misinterpretation, but when data are selected in the way described here, that introduces a whole new level of bias.

  4. You write: “I doubt that there’s any particular reason for CTCA to restrict its cancer treatments to the least-sick patients.”
    Perhaps you missed or doubted this bit from the article: “Today, CTCA – with hospitals in Illinois, Oklahoma, Pennsylvania, Arizona and Georgia, plus an outpatient clinic in Washington state and headquarters in Schaumburg, Illinois – is the only hospital system in the country that specializes solely in complex and advanced cancers.”

    You did quote from the article: “It has almost none who are uninsured or covered by Medicaid”
    I suspect that CTCA has much lower outstanding receivables and fewer rejected payment claims than it would with less restrictive patient acceptance.

  5. The good news is that if CTCA accepts you, you do have a better chance of survival … because you already had a better chance of survival … so you can go elsewhere if you prefer.

  6. Any selective university advertises the fact that it’s selective. You the applicant know that going in the door and indeed it’s part of reason why you are there. (There are other bait ‘n switches that unis engage in but this isn’t one of them.) CTCA is implying that their patients are essentially the same as the general patient *except for the fact that they went to CTCA*. Not so much….

  7. Universities like Columbia are ranked by the SAT scores of incoming freshman. The change in performance on the part of outgoing seniors as a result of that educational experience doesn’t seem to be a topic of as much interest.

    • Wonks:

      Columbia applicants and their parents are not idiots. I’m pretty sure they come here in large part because they think the can get a good educational experience. Some of the goofy things that happen on campus are what makes the news, but we have lots of solid courses and many research opportunities for students at all levels.

  8. Back in 1998 I had Stage IV non-Hodgkins lymphoma. I barely got into a Phase 3 trial of Rituxan, which in the 2000s became the biggest dollar volume cancer drug in the world, because my prognosis was just a tiny bit better than the minimum cut-off for the trial.

  9. Doctors weren’t very good at curing patients until about 100 to 150 years ago, so being good at “prognosis” was the most valuable ability a doctor could have. Darwin’s grandfather Erasmus Darwin was the most eminent doctor in England, in part because he was very good at picking out sick people who were likely to get better on their own to take on as his patients. Erasmus refused to let King George III become his patient. And indeed, the King didn’t get better on his own, underling Erasmus’s skill at prognosis.

  10. “For instance, 60 percent of its non-small-cell lung cancer patients are alive at six months, CTCA says, compared to 38 percent nationally.”

    Ok, I can take just this one line and, unless it is completely out of context, the comparison is so misleading as to be fraudulent. It implies that their treatment is why the six month survival rate is one and a half times the national average. A direct comparison implies that the two groups can be reasonably compared. Picking a subset of patients means you should compare to patients who would be eligible for the exposure of interest.

    But where this crosses the line is here:

    ” What sets CTCA [Cancer Treatment Centers of America] apart is that rejecting certain patients and, even more, culling some of its patients from its survival data lets the company tout in ads and post on its website patient outcomes that look dramatically better than they would if the company treated all comers.”

    The culling of certain patients from the dataset makes even the unconditional estimate incorrect. It is by far the more serious problem (and people seem to be focusing on the invalid comparison). In what other context would selectively deleting participants count as fair play?

    What if I did a study of snake oil and myocardial infarction — but found excuses to delete 50% of the exposed cases (without really reporting this in a clear way). Would that be a useful estimate?

  11. My recollection from researching where to get a stem cell transplant in 1998 (which, thankfully, I didn’t turn out need) was that the key statistic was the percentage of patients who didn’t die from opportunistic infections while your immune system was wiped out. It wasn’t so much that the distinguished Fred Hutchison Clinic at the U. of Washington was more likely to cure you of cancer with its stem cell treatment than your local hospital, but that Fred Hutchison was less likely to kill you while it was trying to cure you than it’s less expert competitor.

  12. this is why I love this blog.

    Those of us who studied statistics as part of our main degree or even post degree can read this and understand.

    it means our skills do not atrophy.

    many thanks

  13. nottrampis, by reading this blog either A) prevents his skills from atrophying, or B) identifies themself as someone whose skills have not atrophied.

Comments are closed.