Skip to content
 

“Mainstream medicine has its own share of unnecessary and unhelpful treatments”

I have a story and then a question.

The story

Susan Perry (link sent by Paul Alper) writes:

Earlier this week, I [Perry] highlighted two articles that exposed the dubious history, medical ineffectiveness and potential health dangers of popular alternative “therapies.”

Well, the same can be said of many mainstream conventional medical practices, as investigative reporter David Epstein points out in an article co-published last week by ProPublica and The Atlantic.

“When you visit a doctor, you probably assume the treatment you receive is backed by evidence from medical research,” writes Epstein. “Surely, the drug you’re prescribed or the surgery you’ll undergo wouldn’t be so common if it didn’t work, right?”

Wrong, as Epstein explains:

For all the truly wondrous developments of modern medicine — imaging technologies that enable precision surgery, routine organ transplants, care that transforms premature infants into perfectly healthy kids, and remarkable chemotherapy treatments, to name a few — it is distressingly ordinary for patients to get treatments that research has shown are ineffective or even dangerous. Sometimes doctors simply haven’t kept up with the science. Other times doctors know the state of play perfectly well but continue to deliver these treatments because it’s profitable — or even because they’re popular and patients demand them. Some procedures are implemented based on studies that did not prove whether they really worked in the first place. Others were initially supported by evidence but then were contradicted by better evidence, and yet these procedures have remained the standards of care for years, or decades.

Even if a drug you take was studied in thousands of people and shown truly to save lives, chances are it won’t do that for you. The good news is, it probably won’t harm you, either. Some of the most widely prescribed medications do little of anything meaningful, good or bad, for most people who take them.

Epstein describes the results of several recent reviews of common clinical practices that found such practices were often unnecessary, unhelpful and/or potentially harmful — “from the use of antibiotics to treat people with persistent Lyme disease symptoms (didn’t help) to the use of specialized sponges for preventing infections in patients having colorectal surgery (caused more infections).”

Many of these treatments were hailed as being “breakthroughs” when they were first approved, but were found in subsequent research to be inferior to the practices they replaced.

By then, though, the treatment had become so ubiquitous that doctors — and patients — were reluctant to accept the evidence that it didn’t work.

As Alper points out, this echoes ESP, power pose, and the collected works of Brian Wansink: publicized mind hacks that don’t seem to hold up under scrutiny.

The question

So, what to think about this? I’m not asking, “What can we do about all this published research, widely accepted by doctors and patients, which turns out to be largely wrong?”, nor am I asking, “What’s a good Edlin factor for clinical research literature?”

Those are good questions, but here I want to ask something different: Should we care? What’s the cost? These largely ineffectual medical treatments are, I assume, generally no worse than the alternatives. So what we’re doing as a society, and as individuals, is to throw away resources: costs in the development and marketing of drugs, doctors’ time effort, medical researchers’ time and effort, patient time that could be better spent in conversation with the doctor or in some other way, various mountains of paperwork, and so on. Ultimately I think the costs have to be put on some dollar scale, otherwise it’s hard to know what to make of all this.

Another way to think of this is in terms of policy analysis. We currently have an implicit policy—some combination of laws, regulations, company policies, and individual behaviors which result in these treatments being performed but not those. So I’d be interested in Susan Perry’s suggested alternative, or the alternatives proposed by others. I’m not saying this as a challenge to Perry, nor is it any sort of requirement that she comes up with an alternative—her reporting already is doing a valuable service. What I’m saying is that it’s hard for me to know what to make of these stories—or even hard numbers on treatment effects—without going the next step and thinking about policies.

I know there are people working in this area, so I’m not at all trying to claim that I’m making some stunning deep point in the above post. I’m just trying to shake the tree a bit here, as I’d like to see the connection between individual studies and the big picture.

36 Comments

  1. Preregistration and replication should be mandatory in fields which demand both. Otherwise this situation will continue as is. Statisticians should read public health journals & doctors should be exposed to controversies within statistics.

  2. Clyde Schechter says:

    “Statisticians should read public health journals & doctors should be exposed to controversies within statistics.”

    Every statistician I know who works in a medical center, and I know a lot of these, already does read public health journals. They also usually read clinical journals in those fields they work in.

    As for doctors being exposed to controversies within statistics, I would suggest that this, itself, is a treatment that requires testing and validation. Consider the following preliminary evidence. For several decades, medical schools have included a biostatistics and epidemiology course in part of the curriculum; the response from most medical students is often either fear and loathing or shrugging it off. The courses offered may, in fact, be part of the problem. Many, nay most, clinicians I know took away only one thing from that course: p < 0.05 is the arbiter of truth. You can't really learn about the controversies within statistics until you know enough statistics to understand them–and efforts to achieve that have so far been, in my view, ineffective. I can also attest that clinical medicine is driven by a desire for certainty. Certainty about diagnosis, and certainty about treatment. Up to a point, that can be a good thing. But it is part of the medical culture often to react to uncertainty with denial, rather than with inquiry. I suspect that exposure to the controversies might well drive many to the wrong side! Finally, the medical school curriculum is already jam packed. While I could rattle off a fairly long list of things that I think could be dropped to make room for better statistical training, the medical education leadership would surely disagree. In any event, it does raise the question, if we make them learn more about statistics, what do we then have them learn less about? Even if there are good answers to that question, I'm not sure there are any that will draw substantial support in the medical education community.

    In any case, tweaking the statistical education of doctors strikes me as unlikely to really help. I could be wrong, and I hope I am. But before making this a policy, it needs a well-designed, well-executed, well-interpreted trial.

    • Clyde Schechter says:

      Oops, sorry. I meant to post that as a reply to Sameera Daniels.

    • Kyle MacDonald says:

      “I can also attest that clinical medicine is driven by a desire for certainty.” That’s a key point. Long before attempting any kind of quantitative education, statisticians should count it as a victory if they could get clinicians (and even medical researchers) to embrace variation and uncertainty even a little bit. That would be a huge achievement, given that they would be embracing that uncertainty in the face of a mother screaming at them to fix her baby’s fever.

      More broadly, from following discussions here and elsewhere, and from spending a lot of time with medical and pre-med students, I’m totally cynical about any kind of improvement in our society’s statistical literacy. As far as I’m aware, more people are reasonably numerate today than ever before, but I’m pretty sure that many to most educated people will always hate and fear mathematics and be really bad at it. Every prediction I’ve ever seen to the contrary starts with “If we could just fix the education system”. There can be some marginal improvements, and those are worth the effort, but I think pretty much any plan that involves widespread statistical literacy among non-statisticians is dead in the water.

    • Kyle C says:

      +1 to all these comments.

      Is it really “certainty,” though? My impression, from conversations with doctor friends whom I knew before they became doctors, is that they were trained, not so much to value certainty, but to eschew *doubt* and speculation as unprofessional and inefficient. They all recall, “When you hear hoofbeats, think horses, not zebras.” (Meaning, if you are in America.) If treatment A is “indicated” in the literature based on test result X, you recommend the treatment and move on to the next patient, not because you are certain, but actually out of humility and deference to the common fund of wisdom.

      Also, in my own interactions with doctors, I have at least as much trouble with noise-chasing as with undue certainty. Too often, [expensive and invasive] test Y is indicated based on a borderline troubling result of test X — which never should have been run on me in the first place, because the false positive rate is so much higher than the background rate of the worrisome condition in the general population, and I am asymptomatic.

      • Seth says:

        Noise-chasing is the result of not understanding uncertainty. Test Y is ordered because the doctor does not understand (or chooses to ignore) just how weak was the evidence provided by test X. Same in Psychology, where a single weak study can launch a thousand follow-up studies under the belief that it was published with p<0.05 so it must be true.

      • Kyle MacDonald says:

        That’s an interesting distinction. Outside of intuitionistic logic, though, what’s the practical difference between wanting certainty and eschewing doubt?

        • Kyle C says:

          Ask a military officer or a judge! In many professions, one adopts a course of action not out of certainty that it will achieve a goal, but because one trusts (I.e., chooses not to spend time or effort doubting) that belief in the rightness of the action well short of certainty is the best we can get as a practical matter.

          Arguably, this just pushes the “certainty” issue one level higher, to mean “confidence in the process that led to the satisficing decision,” but I’m suggesting that this kind of confidence in the professional process — which might be misplaced in any given instance — is qualitatively different from the kind of “certainty” that consumers of statistics are often accused of wanting.

  3. Keith O'Rourke says:

    Its opportunity cost/loss of a poorly managed enterprise. The phrase “practices were often unnecessary, unhelpful and/or potentially harmful” with harmful replaced by unprofitable surely has appeared in many management reviews of poorly performing firms.

    A former colleague spent an enormous amount of effort along with a panel to try to sort this out http://healthycanadians.gc.ca/publications/health-system-systeme-sante/report-healthcare-innovation-rapport-soins/alt/report-healthcare-innovation-rapport-soins-eng.pdf

    In the end suggesting a Collaboration for Healthcare Innovation: New Model, New Agency, New Money…

    Though perhaps academia needs to fixed first. “University administrators could take a more active role by randomly auditing faculty research (performing quality assurance checks) to assess best practices for reproducibility.” http://science.sciencemag.org/content/357/6353/759.full

    And I have heard some universities are actually starting to do that (e.g. random checks to see if the ethics protocol is being followed including looking at how data is handled and stored).

  4. Thanatos Savehn says:

    Petr Skrabanek was all over this issue 30 years ago. As with Meehl the wonder of it is why nobody listened despite the fact that his statistical arguments were damning and went uncontested. Maybe everybody deep down knew that the black box was empty, but people demanded that someone to do something and they wanted a new and improved version of that something PDQ. I suspect that yesterday’s acquisition of Kite by Gilead is a bet that going forward N = 1 and that the FDA will become increasingly skeptical of new treatments supported only by evidence of little harm and a small improvement hip checked across some arbitrary line (e.g. nudging median survival until p<0.05).

  5. Ed Hagen says:

    Don’t discount the placebo effect, e.g., it’s not that antidepressants don’t work, it’s that sugar pills work so well.

  6. paul alper says:

    More from Perry citing Epstein regarding unnecessary and harmful treatments:

    Stent implantation surgery for patients with stable heart disease (patients not in the throes of a heart attack)
    The use of atenolol and other beta-blockers as first-line treatment for high blood pressure
    Partial surgical removal of the meniscus for knee pain
    The use of the cancer vaccine Provenge to stop the progression of advanced prostate cancer
    Deep brain stimulation as an aid to improving cognitive function in individuals with memory disorders
    How published studies on new therapies tend to overestimate the treatment’s benefits and underestimate its potential harms
    How difficult it is for researchers to get medical journals to publish studies that found a common and popular drug or medical treatment doesn’t work
    How treatments that give patients better “numbers” (like lower blood pressure readings) don’t necessarily result in better outcomes (preventing heart attacks or extending life)
    How — and why — it can take 10 years or longer “for large swaths of the medical community to stop referencing popular practices after their efficacy was unequivocally vanquished by science.”

    And even more scary, Epstein writes:
    The 21st Century Cures Act — a rare bipartisan bill, pushed by more than 1,400 lobbyists and signed into law in December — lowers evidentiary standards for new uses of drugs and for marketing and approval of some medical devices. Furthermore, last month President Donald Trump scolded the FDA for what he characterized as withholding drugs from dying patients. He promised to slash regulations “big league. … It could even be up to 80 percent” of current FDA regulations, he said. To that end, one of the president’s top candidates to head the FDA, tech investor Jim O’Neill, has openly advocated for drugs to be approved before they’re shown to work. “Let people start using them at their own risk,” O’Neil has argued.

  7. Markk says:

    I Think there is a difference in Mainstream medicine in that these treatments which are ineffective and do harm will be announced, and doctors scolded for using them, by the mainstream literature. This may take some time but the article itself show it happens.
    In fact there is sometime a flip flopping as a consensus evolves.

    Not so much in the alternative.

  8. Tom Dietterich says:

    Is there a role here for a “best practices authority”? It seems unnecessary (and impossible) for every physician to do their own analysis of the literature to decide what treatments to make. Instead, could there be (or maybe there already exist) one or more organizations dedicated to maintaining a current summary of treatments believed to be appropriate? I can imagine many challenges in establishing such organizations and insulating them from pressures from drug and device manufacturers, alternative medicine, and patient groups.

    • Tim BP says:

      In the UK, there is the National Institute of Health and Care Excellence (NICE). They have an online database of all their published recommendations, and a robust method for reviewing evidence for and against treatments and forming their recommendations.

      Clinicians themselves are addressing the problem with initiatives such as Choosing Wisely (www.choosingwisely.org), where the professional societies produce lists of treatments that should be questioned. This is paired with information for patients. The aim is to increase awareness and empower both doctors and patients to discuss whether a treatment is actually necessary/appropriate.

  9. ZC says:

    The cost is that a *lot* of money (taxpayer and otherwise) is spent on ineffectual treatments. Moreover, Medicare, the largest actor in this space, largely has its hands tied from acting. The problem is not necessarily that ineffective treatments exist, it’s that they largely don’t get priced on value.

    For more, see, e.g., Chandra and Skinner: http://www.nber.org/papers/w16953 or Bagley, Chandra, and Frakt: https://www.brookings.edu/research/correcting-signals-for-innovation-in-health-care/

  10. Terry says:

    Does anybody know whether moderate drinking is actually beneficial to health?

    I keep seeing articles like this one that a drink or two a day reduce mortality from all sources by one-third: http://velvetgloveironfist.blogspot.co.uk/2017/08/teetotallers-still-dropping-like-flies.html

    If so, this seems huge: a one-third reduction in *all* mortality seems like a very big deal … roughly triple the benefits of statins or the treatment of mild hyper-tension with blood pressure medecine.

    Maybe the linked website is written by a goofball. But he keeps citing serious-looking studies. And he seems to have effective replies to objections.

    Does anybody know this area well?

    • KR says:

      There’s a lot of variation depending on the quality of your data on alcohol exposure, how you define a ‘abstainer’, how you deal with confounding, and your mortality outcome (all-cause vs. cause-specific). Unfortunately everyone has different ideas on these things which makes making sense of what’s published fairly difficult.

      I was involved in a study that included look at the correlation of alcohol consumption (ranging from high to none)
      with all-cause mortality over 8 years of follow-up in a large cohort (>200k). I warned my colleagues before we had data that with what we were getting (single measurement of self-reported alcohol consumption) I doubted that we’d see any change in mortality rate across the range of exposure and I was right.

    • Keith O’Rourke says:

      David Speigelhalter has looked into this a fair bit. I think his last take was light regular drinking likely does confer a mortality reduction (but that is uncertain) while more than light drinking confers an increase.

      • Kyle C says:

        Is there any reason to think this is not merely a screen for the drinking habits of the upper middle class and the rich? (One or two glasses of wine with dinner, no more, no less, no hard stuff.)

      • Terry says:

        Thanks.

        Here is a wiki page about Spiegelhalter’s concept of a “microlife” where he says one drink a day is good, but more than two or three is bad.

        https://en.wikipedia.org/wiki/Microlife

        • Keith O'Rourke says:

          Thanks (was not aware of that page).

          I like how it puts many lifestyle choices altogether – along with some description of the uncertainty necessities – “rough estimates, based on averages over population and lifetime. Effects of individual variability, short-term or changing habits, and causal factors are not taken into account”

    • Anoneuoid says:

      whether moderate drinking is actually beneficial to health?

      I think you’ll need to define “moderate”, “drinking”, “beneficial”, and “health”. After that it will still depend on non-stationary individual circumstances (alcohol dehydrogenase expression in various tissues, stress levels, exercise, cultural norms about drinking, the alcoholic drink production process being used, etc).

      So, inevitably the answer will be “sometimes”.

      • Terry says:

        What I have seen defines these terms this way:

        Moderate = 1 to 3 drinks a day (each drink about 10g of alcohol).

        Beneficial to health = a reduction in premature death from all causes.

        Alchohol = alcohol from any source, e.g., beer, wine, distilled spirits, etc.

        • Under those definitions it seems clear that this level of consumption probably does no real harm, and probably does help. But the degree of uncertainty is still large because we don’t have randomization of consumption, so things like socioeconomic class and soforth play into it a lot.

          What I will say is that people who become alcoholics are people who have very high tolerance to consumption. So a person who finds that they could drink 3 drinks in an hour and then feel totally “fine” should probably never drink another drop.

          I’ve been working on a model for consumption based on self-reported quantities, and it’s obvious from this data that self-reporting is very poor as a metric for reality. On the other hand, at the high end it’s probably a pretty accurate thing. around 1 percent of people seem to be severe alcoholics, and they consume typically around 10 to 12 drinks *every day*. Severe alcoholism isn’t something you could “have” and not know it. Of greater concern is how many people are heavy/binge drinkers, say 3-6 drinks more than 3 to 4 times per month, or the like. And the self-reporting data is pretty uninformative about that, too much measurement error.

          • Allan Cousins says:

            You are much braver then I. I think I would be in tears trying to approach measurement bias; from the questions down to the respondent’s self-delusions.

            But I do have a question. Is the population of people that consume drinks regularly (say at least once a month…yes, arbitrary) a readily available statistic? I would be curious to know what the average number of drinks per regular drinker is (which I think we could approximate if we had the regular drinking population and the distribution/sale of alcohol to the population…you’d probably eliminate holidays/ sporting events for draft sales, and assume some amount of the alcohol is stored and not consumed in the year purchased, etc.).

            I think it may be a useful frame of reference. Or it might not. Just curious.

            • Well, you have to deal with measurement bias even for that. perhaps people answer that they don’t drink, but really they’re severe alcoholics who just don’t want to admit to a government survey person…

              In my data from the CDC/Census the average weekly consumption of the bottom 60% is 0.4 drinks per week. And something like 30% of the US claims to be complete teatotalers.

              On the other hand, Alcohol sales tax data suggests that the US consumes 2.32 gallons of pure alcohol per person per year (8.78 liters). That’s the volume of just the ethanol as if you’d distilled it out of all the beverages sold.

              A US standard drink is 14g of pure ethanol, and density is 789 g/L so this means on average 9.5 drinks per person per week, even though 30% of people claim they don’t drink at all, and another 30% barely.

              It’s an interesting data set. I actually have several pages I want to put on my blog, but Stan has a hard time sampling this model and I got diverted by sampling divergences.

              • In my model I actually use the annual sales data. I sub-sample 2000 people who do admit to drinking representatively (weighted by state, sex) and then I try to infer for each one a pattern of drinking, and then I calculate average consumption across the pool and soft-constrain it to match sales data.

                Then, the thing is, you need to connect their underlying behavior to their answers. By disconnecting their answers from reality you can allow reality (the parameter) to bias towards fitting the sales data. It doesn’t work great, but it makes a LOT more sense than just believing their responses and doing frequentist statistics on the response data. Some people don’t understand the questions and this is very detectable in certain cases. For example about 2% of the survey answers that the max they ever drink is say less than 4 drinks on an occasion, but the average they drank over the last 30 days on days they did drink was 30 drinks per occasion. Yes, they answered as if it was the total across 30 days, not the average per day as asked.

                Just this 2% of people who answer this way can double the averages you calculate in standard sampling statistics methods.

              • Keith O'Rourke says:

                Years ago I was talking to a statistician who was working with researchers studying alcohol consumption. As part of the pilot he was (along with everyone else) asked to fill out the drinks questionnaire. He said he had no concerns at all about it being fully confidential but he still could not fill it out honestly. For instance, last weekend after football went to the pub with my mates – hey I exceeded guidelines for a month that afternoon. His answers were faked down to something reasonable. Even for the pilot and by a statistician.

Leave a Reply