I was having an interesting discussion with Seth about his claim that “the overall benefits of health care are probably minor.” The basis of his claim is evidence cited by Aaron Swartz:
In the 1970s, the RAND Corporation picked out 7700 people in six cities and gave half of them free health care. Those lucky ones took advantage of it (spending 30-40% more on average) and they spent it on reasonable things (as judged by medical observers), but they didn’t seem to get any healthier. . . . The RAND study was by far the biggest study of this kind, but other studies find similar results. One analysis found that regions whose Medicare programs give out more money (when the underlying healthiness of the residents is held constant) see no increase in survival rates. A replication found the same results in VA hospitals. Cross-national comparisons find “the impact of public spending on health is … both numerically small and statistically insignificant”. Correlational studies find “Environmental variables are far more important than medical care.” And there are more where that came from.
Several discussants (including myself) at Seth’s blog were skeptical about his skeptism, citing various successful medical treatments (in my case, fixing a compound fracture of the wrist; others mentioned cancer treatment, etc.). Seth responded:
The RAND study, of course, is limited — but is there a better attempt to figure out the overall value of medicine? I don’t know of one. if you can point me to a study that shows the more-than-minor value of modern medicine I’d love to look at it. . . . when the overall effectiveness of medicine has come under scrutiny, it has not fared well — and the RAND study is a good example.
Total vs. marginal effects
I have not looked at the Rand study so can’t comment on the details, but my first thought is that the marginal benefits from additional health care will be less than the benefits from good existing care. So, even if more is not much better, that doesn’t mean that the overall benefits of existing care are “minor.”
From a policy standpoint, it is the marginal effects that are the most interesting, since nobody (even Seth?) is proposing to zero out medical treatment. Presumably there are diminishing returns, and the benefit/cost ratio for additional treatment is less than that for existing treatment. (And, indeed, some medical care can make things worse, even in expected value; for example, you can get catch the flu in the doctor’s waiting room.) But, unless I’m missing something, Seth and Aaron are confusing marginal with total effects.
P.S. Also see Robin Hanson’s discussion (with lots of links), which explicitly distinguishes between marginal and total effects. Here I’m not expressing any position on the marginal effects of health care (given my ignorance on the topic), just pointing out that Robin’s position seems to have become overstated by others.
P.P.S. See Jake Bowers’s comments below. Also more discussion here.
I was working with the basic principle that to measure the effect of something, the best way is to vary it. Another principle — of experimental design — is that it is better to compare a lower dose and a higher dose rather than some dose with no dose because the former is a "purer" comparison. From this point of view the RAND study (which compared two different "doses" of medical care) is better than the something vs. nothing comparison that you seem to favor.
Seth,
I agree about the dose-response principle. (For example, I have no problem with studying saccharin by giving rats a daily dose equivalent to 800 cans of diet soda.) But, if we're on to general principles, there's also the principle of declining marginal returns.
I wouldn't favor a something vs. nothing comparison, since, from a policy standpoint, "nothing" isn't one of the options on the table. (Or, maybe it is, but nobody considers it a desirable option.)
First, I have to say that I find the results of the study to be questionable. If RAND gave them access to 1970's healthcare, how much of the budget spent was preventative vs. reactive measures? My guess is that very few of those "healthcare" dollars went to improving eating or exercise habits – nor would it have any effect on preventing environmental factors that contribute to certain health hazards.
Second, this question, "Is there a better attempt to figure out the overall value of medicine?" Is easy to answer in theory (all those examples you and others presented), difficult in practice.
Assuming a purely economic focus (i.e. not including quality of life metrics), then the ability of healthcare to contribute to the national economy is certainly quantifiable. I'm willing to bet, however, that the conclusions we draw won't be a particularly pretty picture from an ethical standpoint.
Hi,
Since I read Free for All (the book reporting on the results of the RAND experiment), I've worried about the interpretation of the results now, more than 2 decades later.
Imagine that we had done this experiment in 1700. We would have expected more consumption of healthcare by the folks who got it free, and (I'd bet) a strong negative effect on their health as they took lots of mercury pills and were bled. Then, perhaps in 1940 or so if we repeated the experiment, the effect would be perhaps 0 — we had some basic antibiotics and knew about preventing infection, but really couldn't do anything about cancer or most other things, plus medical errors were not acknowledged widely and so probably lots of people died in the hospital because of errors. In 1978 or so, we find a slight positive effect, especially on people with low SES, especially for things like hypertension (and, if I recall correctly, dental and optical health. Is this right?). What if we did it now? We'd expect the positive effect to be stronger. And in 20 years? Yet stronger. So, when people use this single study to claim that medicine is NOW only providing a small benefit, I feel like we are extrapolating a bit far from the data. However, lots of arguments among health economists about adverse selection and moral hazard rest on the RAND study as an empirical foundation. If we discovered a different set of results now, then the arguments about pricing insurance plans would have to change (at least somewhat). And, of course our current health policies are supposed to apply to future relationships between healthcare consumption and healthcare outcomes — relationships which are changing extremely fast given technological growth.
Notice also that in that study the effects were estimated using long long lists of control variables in linear regressions. And the only effects estimated were (1) regressions of healthcare use on health insurance treatment status [positive coef] and (2) regressions of healthcare outcomes on health insurance treatment status [sometimes small positive coef, sometimes not distinguishable from zero]. The authors made clear that they realized that some of the small effects could be due to the negative effects of being in a hospital at all (some well people get sick merely by going to a hospital). But, they never tried to directly disentangle the effect of treatment on the treated (because the effects were so small). If someone reanalyzed that data now [about 800 files and 2gb available as ICPSR #06439] using more modern methods, I wonder how the results would change.
I know that my post doesn't bear directly on the question of marginal versus total effects. Although, the experiment itself gave different "doses" of health insurance (I think 5 different levels) in an attempt to look at the price elasticity of health care consumption. But, I thought that since the question had to do with what to make of the results of the RAND experiment now, I hoped that my thoughts here wouldn't be too off topic.
Jake
Do these studies just look at death as an outcome? That's pretty extreme…indeed, about as extreme as it comes.
Let's talk about me for the moment. I've had the following medical treatments in my life:
Major cost: broken leg set, knee arthroscopy (twice), braces, glasses/contact lenses (costly because I get new ones periodically for my whole life)
Minor: vaccinations, stitches once, antibiotics for eye infection twice, several fillings, wisdom teeth removed.
Suppose I had had no professional medical treatment at all, a ridiculous proposition. I suppose my parents would have set my leg, and perhaps it would have set badly, but it was a simple fracture and even if badly set it wouldn't have killed me. I would have lost a bunch of teeth and my smile would be all crooked too. I would have had some very painful eye problems, but at absolute worst I would have damaged my eyesight, not died. I suppose that without the vaccinations I could have died of tetanus or something. But the most likely case is that, even without any professional medical care at all, I would still be alive. And yet, there is no doubt whatsoever that professional medical care has greatly improved my life.
I also agree with Jake's suggestion that medical care now might be more effective than it was 40 years ago. My dad had open heart surgery a few years ago, and that probably wouldn't have been an option back then (or, rather, probably wouldn't have been recommended).
I do think the costs and benefits of medical care constitute an interesting area of research. I also think that some people expect too much from doctors; in my experience, they can't help very much with most ailments.
I’ve been lurking for a while and this post irked me enough to comment. I say that obviously health care has an overall positive effect. I cite the decline of infant mortality rates over the last century as proof. Looking at the comments to Seth’s entry, things like vaccinations are to be defined as “public health” rather than “medicine.” If by definition we eliminate main effects, there won’t be any main effects.
"But, they never tried to directly disentangle the effect of treatment on the treated (because the effects were so small)."
I haven't spent much time looking at the study, but this has been something that has been really bothering me. The study seems woefully underpowered to look at the effect of treatment on the treated. In fact, by enrolling a random sample of people in your study, you are including a lot of relatively healthy people in your study which could possibly mask the benefits of increased healthcare on those who are sick.
People who attempt to reconcile the results of the Rand study with clinical trials need to realize that clinical trials look specifically at people with a certain condition and compare usually one treatment…and they sometimes need over 5000 patients in order to have power to see a significant difference between the treatments. And remember, clinical trials focus on people who are sick, just as medicine is supposed to be given to people who are sick. How two sick people compare in either arm of the study would be an interesting comparison. Did the people who consumed more medicine recover faster? Did they receive different treatments for the same condition than those who had to pay more for healthcare.
I don't see any statistical reason why I would think that much an effect would be seen in the Rand study after taking a sample of relatively healthy patients after following them for only 5 years.
Seth,
The Rand study doesn't attempt to figure out the overall value of medicine. If this is a question that you truly want to answer then I believe you would be much better off, and have a much clearer comparison if you had a no-medicine versus some medicine comparison, or if you preset varying levels of medicine that people were allowed to receive rather than varying the costs.
The Rand study essentially looks at how varying levels of healthcare premiums and deductibles affected healthcare consumption and overall health. This is very different than if you were actually varying the amount of medicine people received who had a certain condition, or preset medicinal levels for each group for each condition. With the Rand study, you are still giving people a choice as to how much medicine they are going to receive; you leave them measuring the cost versus benefit of more medicine; they have to decide whether they are sick enough yet to warrant visiting the doctor or hospital, whereas those with free healthcare can go just in case. The point is, you are leaving a lot of choices in the hands of the people in your study which means that we have to be very careful when we make comparisons.
The Rand study is a dose-response of health insurance, not dose-response of medicine. Also, let's not forget that people were allowed to pick which level of health insurance they received, which opens up a huge door for potential confounding factors. We also can't forget that everyone had a fairl low cap on out-of-pocket spending ($1,000(1977)… $3,000( 2005 adjusted)) , which means that for anything serious, people probably received the same care.
Seth,
The Rand study doesn't attempt to figure out the overall value of medicine. If this is a question that you truly want to answer then I believe you would be much better off, and have a much clearer comparison if you had a no-medicine versus some medicine comparison, or if you preset varying levels of medicine that people were allowed to receive rather than varying the costs.
The Rand study essentially looks at how varying levels of healthcare premiums and deductibles affected healthcare consumption and overall health. This is very different than if you were actually varying the amount of medicine people received who had a certain condition, or preset medicinal levels for each group for each condition. With the Rand study, you are still giving people a choice as to how much medicine they are going to receive; you leave them measuring the cost versus benefit of more medicine; they have to decide whether they are sick enough yet to warrant visiting the doctor or hospital, whereas those with free healthcare can go just in case. The point is, you are leaving a lot of choices in the hands of the people in your study which means that we have to be very careful when we make comparisons and interpret the results. Namely, those who had to pay less for healthcare did not have noticeably higher health levels. Just because they used more services and were not noticeably healthier does not even come close to saying that the overall value of medicine is zero… we are more likely, as Andrew suggested, to be seeing diminishing marginal returns of more medicine. People in the cost sharing plans still had the chance to get the medicine they think they needed and were willing to pay for… whatever they didn't think they needed they didn't pay for. Perhaps the study instead says something about our ability to determine what healthcare we really do need.
The Rand study is a dose-response of health insurance, not dose-response of medicine. Also, let's not forget that people were allowed to pick which level of health insurance they received, which opens up a huge door for potential confounding factors. We also can't forget that everyone had a fairly low cap on out-of-pocket spending ($1,000(1977)… $3,000( 2005 adjusted)) , which means that for anything serious, people probably received the same care.
Maybe medicine is in a place that's similar to education, where variations in spending don't seem to have much effect. But that's marginal spending; no one is proposing to reduce education spending to zero.
And of course the argument in education is that the reason marginal spending doesn't help is that it often or usually gets spent on the wrong things — building a new gym or adding administrators or whatnot, rather than on the very few areas where schools can make a difference in achievement (i.e., having highly effective teachers; but that is very hard to do via spending anyway, b/c traditional qualifications such as certification don't seem to have much to do with it, and increasing teacher salaries just means that the ineffective teachers get paid more). In medicine too, maybe it's the case that marginal spending often gets spent on unnecessary tests/procedures.
I was under the impression that every surgery was supposed to be helpful. And, with less emphasis, that every other piece of health care, such as every drug you take, is supposed to be helpful. Not just the very first or most necessary. Because surgery and drugs are dangerous, it is reasonable to keep this as a standard. When this conspicuously fails to be true, as the RAND study suggests, we should wonder why.
Was the RAND study too small? The big effects in health care — giving a high-Vitamin-C food to someone who has scurvy, for example — require tiny experiments to show their effectiveness. I think the Lind experiment involved about 10 people. The first evidence that smoking causes lung cancer involved on the order of 40 patients.
When I get my computer fixed, I don't need a randomized experiment to be convinced of improvement. Of course we can hold doctors to a much lower standard — but should we?
Seth, smaller studies are fine for if you want to concentrate on a population of people who are sick and you are going to randomly give half the cure as in your scurvy example.
The point about the most necessary medicines is that those are where the biggest benefits are seen. Not every drug or every treatment will make a big impact in your overall health. For any treatments that do make a large impact on your health, I assure you that even people who had to pay received the necessary treatments.
Big effects in healthcare can be seen for those who are sick… it is much harder to see effects of medicine when you take a sample from the general population, thus you should need an even bigger sample if you want to see a benefit.
Unfortunately, as medicines become better and better, it becomes harder and harder to improve upon the standard of care. This is one of the reasons why clinical trials require so many patients to see the positive benefits of a new drug. In addition, even when the population in your study is sick, it can still be difficult to see the benefits when looking at events with low incidence rates…. and this is even when your population is sick. Can you imagine how woefully underpowered you must be when looking at a group of healthy people.
By the way, for those of you who are interested, the study looked at 2750 families, the mean age was ONLY 33, the youngest person was 14, and the maximum age at time of enrollment was 61.
In addition, please excuse me for misrepresenting the randomization in a previous post…. people were randomly assigned to a specific level of insurance and then were allowed to decide whether they wanted to continue with the study… acceptance rates varied from 92% for the free plan down to 75% for the catastrophic. Interestingly, as you might expect, those who agreed to be in the catastrophic, and would have to pay the most out of pocket for healthcare, had the best baseline measurements in several important categories:
physical functioning, role functioning, social contacts, risk of dying, smoking, vision, percent hospitalized in year before enrollment, and percent female. The implication is that those who rejected the plan with the least coverage may have not been as healthy as those who accepted.
Neat to see this conversation. I hope I haven't come too late to it. A few points.
As current grad student points out, RAND HIE looks at insurance, not medical care; questions of the marginal value of medicine should look at something like David Cutler's Your money or your life on the cost-effectiveness of expensive medical care.
That said, insurance is not entirely distinct from medical care. Insurance doesn't just spread risk, but also gives incentives for earlier and preventative care. RAND HIE and many subsequent studies suggest that cheap or free care of chronic conditions lead to better management, which can prevent amputations and blindness in diabetics, second heart attacks, and adverse effects from hypertension.
Relatedly, wrt Seth's scurvy model: It's very rare to be able to apply a treatment and have a disease simply go away. Most mortality is from chronic diseases where the etiology at the individual level is unclear, and the strategy is to minimize a couple dozen different risk factors.
Prevention vs. medical care is an arbitrary distinction: medical care is preventative. For instance, the major benefit of an annual physical for healthy young people comes from the social history because doctors have greater influence than other authority figures, and yet no one would go if doctors didn't do the whole stethescope routine.
Similarly, healthy adults could probably see the doctor every 5 years instead of every year with little effect, except that they might resist starting annual appointments at an older age when more frequent care brings benefits again.