Seth Roberts has had success with self-experimentation—among other things, he’s written a successful diet book on how to lose weight by eating unflavored oil or sugar water—and on his blog he reports his latest self-experiments and their effects on him. For example, recently he wrote about the beneficial effects of fermented food.
When Seth tries a new food, or a new lifestyle change, and finds positive effects, I’m always skeptical: maybe he’s hoping for such effects and then finding them. But often they work for others. For example, his correspondent Tucker Max writes:
Anyway, after a week of drinking two bottles a day, I have noticed these changes:
2. I have more energy. Aside from subjectively feeling it, I can see the difference in my workout logs, just in this past week I’ve gone up more weight on exercises than I normally do.
3. I am feeling overall better. This could very well be placebo effect/confirmation bias as it is a very subjective measurement, but I just feel better. . . .
Sure, but maybe this could all be a confirmation bias. The toilet stuff sounds objective, but who knows what else is happening when he’s doing this? And then of course there’s selection bias, that Seth is hearing about the successes.
Just to be clear: I’m not trying to criticize what Seth is doing, and I’m not trying to shoot it down. I’m trying to strengthen it by suggesting ways of thinking about it. As Seth says, criticism is easy, helping people is hard.
So here’s my thought. Maybe Seth could try a real placebo, as follows: he could make up some goofy food or behavior change (something like . . . eating fermented food! Or, I dunno, sleeping with the bed inclined at a 10 degree angle. Or, I dunno—Seth would be better than I at coming up with something. (Of course, it should be something he tries himself first and finds no adverse effects from.) He could then make up some fairly vague story about how it helped him, then post it on his blog and see what happens. Would people respond with stories about how helpful it was?
The great Linus Pauling conspiracy
I’m reminded of the idea I heard once that Linus Pauling knew all along that megadoses of Vitamin C have no effect, and that he altruistically sacrificed his reputation as a scientist to trumpet Vitamin C’s virtues, on the theory that it would reduce the suffering of millions via the placebo effect.
you forgot to close a bracket
Self-experimentation has little scientific validity, but it can be very persuasive. When you hear the story, for example, of a surgeon practicing an operation on himself, it grabs your attention. Such an anecdote can be more persuasive than an infinitesimal p-value.
Ethically, it may be better for him to say "One of these three things helped me: A, B, and C. The other two I made up. Please try them and let me know any results." That way, there would be no deception. What do you think?
Andrew,
I agree with you that Seth could be picking only the "winners" as he just did when he featured my testimony recently on his blog: http://www.blog.sethroberts.net/2009/03/01/nose-c…
However, his point, I believe is very clear. Many of the studies that involve human subjects are extremely expensive in terms of commitment, time and money on the part of the participants, researchers and funding agencies involved. It is therefore very clear that exploratory based research is likely very much dampened by the inertia of organizations dedicated to produce research results in this area.
Let me take another example, and I am sure they are plenty others:
Let's say your kid is autistic. Research currently barely makes a dent in the area of diagnostic as you recall ( http://www.stat.columbia.edu/~cook/MT/mt-search.c… ) and so real improvement in terms of systematic treatments for the different types of impairment is pretty much non existent. Let us take the example of the literature on vitamin B6. There is an on-going review that occurs every few years that summarizes the finding as to whether B6 can help alleviate some of the behavioral and other aspects
of the disease. Here is the latest one from PubMed:
http://tinyurl.com/be8q4j
please note how the ending goes:
"… This small study (n=8) only measured IQ and 'Social Quotient' and found a statistically significant benefit for IQ (5.2, 95% CI = [0.2 to 10.3]) when in the treated group, by using change scores. AUTHORS' CONCLUSIONS: Due to the small number of studies, the methodological quality of studies, and small sample sizes, no recommendation can be advanced regarding the use of B6-Mg as a treatment for autism…"
In other words, it may work, but most medical doctors reviewing this will tell you it doesn't. In the future, we will surely find out that autism is really a set of very different diseases and that only a subgroup can really respond to B6 in the same way that nose-clipping might work only for a subset of the population.
With this in mind, is Seth picking the winners or identifying a non-small subset of the population that clearly can benefit from his findings ? How do you go from an anecdotic occurence to a more general statement when testing is expensive ?
Igor.
How convenient for Linus Pauling!
Next time I do something stupid in my research, I'll stick to the claim that I did it on purpose.
From what I could read on Linus, his ego was gigantic. It is more likely that he got so invested into it that he could not back down.
When something succeeds in reducing a problem after a dozen other plausible treatments have failed, "placebo effect" is a very unlikely explanation. Likewise, unexpected effects — such as a mood boost on Tuesday from faces seen Monday morning — are also unlikely to be due to placebo effects. If there are any cases of such things being due to placebo effects, I'd like to hear about them.
Re Tucker Max. "The toilet stuff sounds objective, but who knows what else is happening when he's doing this?" He made a big change in diet, one that would plausibly affect digestion, and noticed "toilet stuff" that he did not anticipate. Sure, it's not a highly-controlled experiment but you seem to be ignoring what can be learned from it — that the kombucha probably made a big difference. Or, more precisely, that it should become more plausible that kombucha has big effects.
If placebo effects were strong and widespread, how many problems would remain? We'd just take a placebo for everything.
As the old adage goes, "the plural of anecdote is data." When you start with a sample size of 1, you are dealing with anecdotes, but if other people try it and achieve similar results, you can then have real data and real progress.
Different people also respond differently to various changes. I have self-experimented with different vitamin and mineral supplements for about 24 years. Somethings that I try are effective and I continue them; somethings that I try are ineffective, so I discontinue them. Usually I know within a few days what seems to be working. Before I try some vitamin, I cannot predict whether or not it will work, and much of the time it does not. Does this mean that the successes I've had are all a "placebo effect" or a "confirmation bias" ? I don't think so.
For example, I strongly believe in large doses of vitamin C. I used to have gut-wrenching abdominal pain for about 45 minutes every morning. Multiple doctors told me I had "a sensitive stomach" and would just have to live with it. When I take enough vitamin C, the pain disappears entirely.
In the early 1990's, I had very frequent sinus infections for about 2 years. After 2 surgeries and months of different antibiotics, I still had problems. The infections did not clear up until I upped the vitamin C dose very high (about 10,000 mg / day) for several weeks. Was this just a lucky coincidence that my sinuses finally cleared up?
I also discovered that different brands and formulations of vitamin C differ in their effectiveness. This difference appears to be unrelated to the cost of the supplement or its advertising. I just experience a stronger effect from some brands and formulations, and I cannot predict it ahead of time. Is this just a subconscious placebo effect or confirmation bias?
Sometimes, others have experienced similar effects. For about 10 years during and after the sinus problems, I would have a nose bleed for about 5 to 10 minutes just about every single day. I went to multiple ENT specialists who could not help me with the problem at all. After 10 years, I discovered that vitamin A reduced and ultimately ended the nose bleeds all together. I don't know why it works. I would never have guessed that it works. I had never read anywhere that it works. Surprisingly, beta carotene supplements have no similar effect. I just stumbled onto vitamin A. A few years ago, a colleague at work complained of a similar nose bleed problem during the winter months. I suggested taking vitamin A supplements, and it worked for him as well. Were both of us subject to a placebo effect? I don't think so.
Self-experimentation is perfectly valid. Just make sure what you are doing is fundamentally safe. What is really needed is some sort of clearinghouse so people can exchange different observations. We can then let individuals and the "marketplace" of observations decide on what works. Some might not consider that to be as rigorous as a "double-blind" study, but it is real data that can help better people's lives.