I want to explore the distinction between self-experimentation and formal experimentation in the context of a recent discussion on Seth’s blog.
The story begins with two people who found, via self-experimentation, how to make their acne go away:
A student . . . had gone on a camping trip and found that her acne went away. At first she thought it was the sunshine; but then, by self-experimentation, she discovered that the crucial change was that she had stopped using soap to wash her face.
A friend of Seth writes: “I started “washing” my face with water about a month ago, and [now] my face is acne free and soft as a pair of brand new UGG boots. [He had had acne for years.]”
In the comments section, someone writes:
While it would be nice to think that all we have to do to get rid of acne is stop using those expensive cleanser and just use water – this is just anecdotal evidence you present. It would require a large clinical trial to be conclusive.
Seth replies that informal experimentation is cheaper and faster than more formal clinical trials. Also, different things might work for different people, so whether or not a treatment has been evaluated a large study, it might make sense to test it yourself–especially for something such as acne or weight loss that is not an urgent concern.
This got me thinking . . . what are the benefits (if any) of a formal controlled trial? In statistics, we usually frame these benefits by comparing to observational studies. The big risk in an observational study is that the treatment and control groups will differ in important ways (as in the famous hormone replacement therapy story). Is this worth the cost? Maybe. Sometimes.
A related issue is bias, a word which I am using in the conversational rather than the statistical sense. For example, how would you want to evaluate the risks and effectiveness of a new drug that was developed by a pharmaceutical company at the cost of millions of dollars? I’d be suspicious of an observational study: even if conducted by professionals, there just seem to be too many ways for things to be biased.
In Seth’s acne example, there is no financial source of bias. And, as Seth points out, the test is free to apply on yourself. If I had a kid with acne, I’d give it a try and do an experiment–which means trying the soap and no-soap conditions on different days (or different weeks, or months) and measuring and recording acne levels. One thing I’ve gathered from Seth’s work is that there are big benefits to be gained by doing self-experimentation with careful measurement and record keeping, rather than simply trying different things and trying to remember what works.
On the other hand, yeah, I’m skeptical about Seth’s acne claims, and I think a larger study would be more likely to convince me. But I don’t think it would have to be expensive. All Seth (or somebody) needs is to set up a protocol for deciding when to wash with soap or water and a protocol for measuring acne, then he could get a bunch of volunteers to flip coins and try it. This blog has a few thousand readers, and Seth’s diet forum has thousands of participants, so it shouldn’t be so hard to find people to do this. I’m not so interested in acne myself, but according to Seth (and others, I assume), “acne really matters,” so maybe it’s worth giving this a try.
I wrote about this issue in the context of startup marketing, which for many companies is increasingly driven by rapid iteration and A/B testing.
The problem for most startups is that they don't have enough traffic to run an experience at the usual confidence level. That's why they're a startup!
So you wind up in this chicken and egg problem. I made that point that it's ok to run tests at a lower confidence level because the cost of making a wrong decision in this context is minimal. Most decisions can be undone and forward momentum is more important than accuracy, especially when you don't have a lot of customers to leverage your decisions.
To tie it back to what you wrote, one of the benefits of a controlled study in a medical context is the reduction of risk. When an unsafe drug slips through the clinical trials, for whatever reason, the cost is HUGE.
Bias is perhaps a part of this, e.g., a pharma company is incentivized to fake the results of their studies. But in the end it all comes down to the same thing: getting an accurate picture of the cost.
But in the acne cream case what is the cost? Do we really need a single/double/triple-blind study with hundreds of participants is a clinical context to be ok with saying, "Hey, try washing your face without soap!"
It's not like we're telling people to take an untested drug.
While i know I couldn't live with giving someone an untested drug that killed them, I think I could live with them getting rejected by their would-be prom date.
And that's about the worst that can happen.
This reminds me a bit of Jakob Nielsen's assertion that usability test should be done with only 5 users at a time. His point is that many iterative tests is more effective that fewer because problems can be corrected between tests. And of course tests with fewer subjects is less costly.
http://www.useit.com/alertbox/20000319.html
I'm not sure how this would apply to testing the efficacy of a product or procedure unless there were ways of changing the procedure after each test and then of course the old tests would no longer be valid.
You make a good point about bias. I call it the Pashler-Roberts Law: The more expensive the research, the less likely those who do it will be honest about it. Pashler is Hal Pashler who co-authored the "voodoo neuroscience" paper. That research was really expensive (for psychology).
I suppose I'm biassed because it is hard for me to see why you're "skeptical about [my] acne claims." There's only one claim: That in some cases, washing your face with soap causes acne. My friend had acne for years. It cleared up shortly after he intentionally made a certain change — stopped washing with soap — that he hoped (and had reason to believe) would clear it up. You can be sure it isn't the first thing he's tried so you can rule out expectations. That you're "skeptical" makes it sound like you can think of plausible alternative explanation not just for this result but for the earlier observations (camping trip, self-experimentation). I can't imagine a plausible alternative.
Interesting. Reminds me of the discussion in the current issue of Wired magazine about "living by the numbers," where people measure aspects of their life constantly and use that self-feedback to make positive changes.
I should note one issue with both this acne anecdote and an anecdote in the Wired magazine, which is that the result is an improvement over time. Many things improve over time without treatment, including acne. In the Wired note (Experimental Man, p. 90), the anecdote is about a psych professor (who should know better!!) who tested his cognitive function when taking Omega-3 capsules. The raw data looks good, with an improvement over time and a jump when he increased the dosage. But there are two confounds — practice effects on the arithmetic problems, and a placebo effect. A better experiment would have three sets of pills (no dosage, low dosage, high dosage) taken in blocks of a week or two each, for six months or so. A random daily dose is a bad idea, because the effects of the Omega-3s may take several days or longer to take effect, or to lose effect if stopped. Blocks that are too long are also a bad idea, since you lose power if you only compare two or three transitions between dosages/placebo. But you do want to be able to subtract out the overall practice effect (the shape of which has been studied extensively in the literature).
Same with the acne study. It's not enough to have half of 1000 people stop washing their faces for a month. They need to stop washing for two weeks, start washing for two weeks, stop washing for two weeks, etc.
In any case, I think this acne result is fairly well known. If you stop washing your hair with a detergent, your hair will be greasy for a few days as your scalp rebounds, but will actually become quite pleasant after a while, as long as you rinse the actual dirt and dust out…
I remember reading about putting toothpaste on bee stings.
I was very skeptical, but read an account of one fellow allowing himself to be stung repeatedly in order to try different treatments (google toothpaste bee sting to find several relevant links) and then rated them for effectiveness, and ice and toothpaste both got high ratings (ice was obvious and used for all manner of painful injuries, and I had used it before, so that was already on the list). I felt the account of that experiment was enough of a start to have me at least try it next time we had a bee sting – it seemed harmless at least.
My son was stung a few months later. I tried the toothpaste immediately and I was astonished at how effective it was (dramatically reducing redness, swelling and pain in minutes) compared to the typical course of untreated bee stings.
Doubtless the response of individuals will show a good bit of variation, but simple remedies like that which are likely to utterly be harmless can at least be tried individually – with inexpensive harmless treatments the costs of the two error types are very different; even if it was entirely placebo (which I doubt, having seen it), it would hardly matter as long as the placebo effect was noticeable to the patient.
Hmmm interesting. I performed a similar (self-)experiment with acne, and also came to the conclusion that it was soap. Now my more refined analysis, after some more self-experimentation, is that increased stress, too much soap (e.g., washing and washing, to try to keep pores clean), and importantly worrying about acne, tends to increase acne.
I think the psychological bit of this is terribly important.
I would have collected data on myself if I weren't scared that by so doing I would bring more acne on…:-)
One of the problems is hard water. If you have unusually high hardness it can allow the soap to stay on your skin where it can get into your pores and cause it to become acne. A good water softner can help a lot or even a good shower water filter.
@Furnace: actually I think it's soft water that allows soap to stay on skin: http://en.wikipedia.org/wiki/Hard_water#Effects_o…
Personally I haven't had bad acne, but I have had excessive oil on my face. About 6 months ago I stopped washing my forehead with soap (just use hot water now) and things are much improved. I figured my skin was trying to produce so much oil because I kept stripping it away.
Anyway, about the clinical trial vs. personal experimentation – if you want to understand your life better, do experiments. It's just like anything else. =) But don't expect a skeptic to believe your results with a sample size of 1.
This is really the difference between alternative medicine and the more structured "normal" Western medicine. Alternative medicine is empirical -its often based on folk remedies that have been used successfully for centuries- but the effectiveness of the methods haven't shown up in double blind trials lasting for years etc.
But its quicker and cheaper to walk into a health food store and ask which product to try than to make an appointment with a doctor. You should do both, but in the meantime bathing with oatmeal or whatever will at least have a placebo effect.
Andrew:
Doesn't a lot depend on the "signal-to-noise" of the treatment impact vs. natural variation?
If the treatment causes me to go from 100 pimples to 0 within a week, and but for the treatment my weekly pimple count would be 100, 98, 104, 101,…., then I can draw a common-sense conclusion that the cure worked. If, however, it causes a net reduction of 10 pimples and natural variation has a STDEV of 50, it might be a lot harder to draw a conclusion. This is why I need carefully matched test and control groups to create a valid "but for" baseline.
I suspect this is why clinical trials developed originally for therapeutics as opposed to surgery.
Jim:
Yes, definitely. And this suggests another intermediate step between informal self-experimentation and full clinical trials. The intermediate step is formal self-experimentation, of the sort Seth does on his blog, where he gives a pretty complete description of his treatment schedule and measurement protocols and then graphs his numerical measurements.
One reason I'm somewhat skeptical of his correspondents' reports is that I don't know if they really did what they said they did, and saw what they said they saw. Even if you remember doing X and observing Y, it didn't necessarily happen that way at the time. I'm not saying anyone is lying, more that they could be fooling themselves. Journalism and science are littered with examples of observations of impossible events, that maybe didn't happen quite the way they were remembered. Formal self-experimentation wouldn't fix this problem but it might mitigate it.
HI,
I also suffered a lot from Acne – even in my 40's I would get enormous pimples just before presentations. —
But soap?
Yes, there is a not just an experiment, but a mechanism. Washing with soap changes the pH of your skin, this affects the suitability of your skin for growing different bacteria.
When I saw a doctor in the 70's (when I was in my 20's) he gave a pH balanced washing solution to use, and this definitely helped. I was studying stats then & I didn't to do a test to know that 30-40 spots a week going to 1-2 was significant. It was significant by the 'disco test' and that was what was practical ;-)
Later I had a course of low dose antibiotics and that finished off the little red obfuscations, almost forever. I try not to think about upcoming presentations :-)
Dave
I also had acne for years and I tried everything. I saw several different skin specialists and endured salasylic acid baths on my face, and tried Proactive as well as several of their competitors, and nothing worked. I even lived in a house where there was a $6000 water filter installed and all the shower water was filtered and soft water.
When I stopped using soap on my face, my acne went away, and it never returned.
I don't understand why this information is not more available, I searched the internet for years. If anyone had told me that all I needed to do was stop using soap on my face it could have saved me suffering through an acne covered childhood and all the awkwardness that brings with it.
I wish I had the ability to tell more people about this, I just think of how many teenagers today are suffering needlessly.
Hot water is all you need to wash your face. Any kind of soap at all only strips the oils from your face, and your face reacts by overcompensating and overproducing skin oil which in turn causes acne.
The only reason nobody knows about it is because this cure is free. Your doctor doesn't make any money if he tells you that household water and nothing more is the perfect cure for acne. The proactive people won't make money by telling people that. Even an alternative health store won't make money from you if they tell you you don't need to buy anything to cure your acne.
Even today, if I search the internet for cures for acne its really hard to find anyone saying "No Soap, just Water". Even Google does not know the answer on this one!
This blog was the ONLY place that I found anyone else telling the truth – that all anyone needs to do to cure acne is stop using soap on your face.
You know how I'm not lying? Because I'm not selling anything.
Gabriel
The natural methods are always good methods for getting rid of acne.
For example, we can use tea tree oil, because it is a well known natural antiseptic and has a very good effect on skin, no matter pure tear tree oil, or products made from tear tree oil.
We can select foods which are high in zinc, because zinc can control hormonal level, and hormonal change is a major reason of causing acne.
Also we can try apple cider vinegar, which is proven to be successful for many people in acne treatment. You can use the apple cider vinegar from the market, mixed with water in 50/50 ratio. Then just apply it with cotton to the skin.
These are all natural and effective acne treatments.