I don’t often read the Iranian Journal of Cancer Prevention, but I like this quote:
I was thinking more about the PACE trial. God is in every leaf of every tree.
There’s been a lot of discussion about statistical problems with the PACE papers, and also about the research team’s depressing refusal to share their data. (As E. J. Graff, our editor at the sister blog put it, you funded these clinical trials, but you’ll never know what they found.)
But today I want to talk about something slightly different, which is the flawed way that research results get interpreted and translated into policy.
There are big problems here, which I attribute in part to one of our favorite villains, deterministic thinking.
The PACE study is a randomized controlled trial of alternatives to treating chronic fatigue syndrome, and the main finding of the study was that cognitive behaviour therapy or graded exercise therapy were more effective in reducing fatigue and improving physical function than adaptive pacing therapy or nothing at all. All treatments also included specialist medical care. See here for descriptions of the treatments. The basic idea is that CBT and exercise therapy are about getting the patient moving again, whereas adaptive pacing therapy and specialist medical care are about managing the condition.
The follow-up paper includes a bunch of mediation analyses that, as usual, I don’t understand, but that doesn’t really matter. The key point, to take their claims at face value for a moment, is that they found that talk therapy and exercise worked in a randomized controlled trial, and this led to three larger conclusions: First, that these are the therapies that should be recommended for the general population of chronic fatigue syndrome sufferers, and that these are the treatments that should be paid for by government and insurers; Second, that when people with these conditions complain about pain and exhaustion following exercise, the solution is for them to get rid of the negative thoughts and continue exercising; Third, that, as the investigators write, “the results support a treatment model in which both beliefs and behaviour play a part in perpetuating fatigue and disability in chronic fatigue syndrome.”
Or, as chronic fatigue syndrome sufferer and science writer Julie Rehmeyer put it, “according to the theory underlying this psychiatric research, my problem was that I was out of shape, afraid of exercise, and obsessed about my symptoms. The path to wellness was to drop the idea that I had a physical disease and steadily increase my exercise, no matter how bad it made me feel.”
So. My problem is in the conclusions being drawn from the study. I see the following chain of reasoning:
1. It’s a randomized controlled trial, perhaps the only large study of this sort on chronic fatigue syndrome. Results are (I assume) statistically significant.
2. This is the gold standard: a statistically significant effect in a randomized trial. I’ll assume for now that the design and analysis were clean. There’s some dispute about this, in particular regarding the appropriateness of the outcome measures, but for now let’s just take the empirical clinical finding as true. Just speaking generally, CBT and exercise therapy are both good things, so it is certainly plausible that they would help on average this group too.
3. Evidence-based medicine. If CBT and exercise therapy are the only proven treatments for chronic fatigue syndrome, then I see the argument for (a) recommending these treatments, and (b) being reluctant to pay for unproven alternatives.
4. Conclusions about how the disease works. If talk therapy cures it, the condition must be psychosomatic. If exercise therapy works, it must be that the people with this condition could get better, if they didn’t have these mental blocks.
What went wrong? Lack of understanding of variation. As noted above, it’s no surprise that CBT and exercise therapy can help people. Great. But lots of people with chronic fatigue syndrome will have big problems even if they do get CBT and exercise therapy. Even in the published study, the success rate was far below 100%. The success of these therapies for some percentage of people, does not at all contradict the idea that many others need a lot more, nor does it provide much support for the idea that “fear avoidance beliefs” are holding back people with chronic fatigue syndrome.
At this point, you might feel that I’m being too harsh on the PACE study. If a randomized controlled trial won’t satisfy me, what will? My response is that you have to take the study for what it is, you can’t assume that the positive features of the randomized trial will somehow lend credence to extrapolations, and you can’t assume that just cos a treatment does something for some people, that you’ve learned some general principle. Not for something like chronic fatigue syndrome which is, presumably, some mix of different conditions.
Don’t ask, “Which one is right?”
A commenter in yesterday’s thread picked up this revealing quote from Lancet editor Richard Horton:
[A]daptive pacing therapy essentially believes that chronic fatigue is an organic disease which is not reversible by changes in behaviour. Whereas cognitive behaviour therapy obviously believes that chronic fatigue is entirely reversible. And these two philosophies are kind of facing off against one another in the patient community and what these scientists were trying to do is to say, ‘Well, let’s see. Which one is right?’”
No no no no no. It’s just a radio quote so I’m not going to criticize Horton for saying that a therapy “believes” something; his meaning is clear. My problem is with his attitude that it’s one or the other.
“Which one is right?”, he asks, but this is not in general a good question to ask.
There are two reasons this is a bad question:
1. Chronic fatigue syndrome is a diverse condition, or set of conditions lumped under a common diagnosis. It is completely reasonable to think that different therapies could work for different people, and that the condition has different sources for different people. Indeed, even the PACE trial itself found many people did not improve under their treatments.
2. Even for any particular person, the condition could have a mix of causes and be amenable to a mix of therapies. So the attitude that it’s one or the other, can be a serious mistake even for a given patient, let alone when trying to characterize a broadly-diagnosed syndrome in the general population.
These points are relevant and important even if the published PACE trial had no flaws at all. That is, I believe Horton is seriously mistaken in his argument, even setting aside all the concerns about those published papers and even setting aside the refusal to share the data.