Paul Alper points us to this horrifying op-ed by M. Gregg Bloche about scientific study of data from U.S. military torture programs.
I’ll leave the torture stuff to the experts or this guy who you’ve probably heard of.
Instead, I have a technical point to make. In the op-ed, Bloche writes:
In a true experimental study, the C.I.A. would have had to test its interrogation strategy against one or more standard interrogation methods, using experimental and control groups of captives. There’s no evidence the agency did this.
They [the psychologists, Mitchell and Jessen] argued that interrogation strategies can’t be standardized and therefore can’t be compared, like medical treatments, in randomized, prospective fashion.
No one, though, is claiming that C.I.A. review efforts involved experimental and control groups and so were “experimentation” as science defines it.
This statement, that a true experiment requires a control group, is wrong. In a controlled experiment there needs to be a comparison, but an uncontrolled experiment is still a form of experiment. To put it another way, we use the term “controlled experiment” because control is not a necessary part of an experiment. In many cases control is good practice, and control makes it easier to perform certain inferences, but you can do experimentation with out a control group.
I have never done an experiment without a control in my entire life, and I think that day will never come.
A reasonable experiment to do without a control is something you do on yourself, like changing your habits in some way, or changing your business practices.
It’s an experiment – there ought to be a time afterwards where you consider whether it worked as expected, or whether you should go back to the old – but you have to live with a lot of uncertainty, and you can pretty much forget about quantifying it.
Maybe Mitchell and Jessen should torture themselves for a bit and see if it makes them more inclined to help their prosecutors.
This experiment requires no control cola:
https://www.youtube.com/watch?v=GnmN9L5eP5s
Furthermore, I am of the opinion that John Yoo be fired from the University of California.
jrc, Yes, it does require a control. The pre-Mentos cola is the control. If you had never seen cola absent Mentos you would have no idea that the candy could cause such a reaction. As Shravan suggests, all experiments have a control (comparison) condition. Control groups are obvious; units serving as their own control is less obvious but still very much a control. This applies to self-experimentation as well. In fact, from the potential outcomes perspective, the unit absent the treatment immediately before receiving the treatment is the closest you can get to an ideal control under real-world conditions. There is no such thing as an ‘uncontrolled’ experiment. To suggest otherwise is a fundamental misunderstanding of what an experiment is and how it works. Now, whether a control is good is a different question.
Sentinel:
I agree that all causal inferences are comparisons; where I disagree is the claim in that op-ed that all experiments must have control groups. You can have an experiment with only a treatment group; in that case the estimated treatment effect is a comparison to an implicit control. Such an inference can have robustness problems (it is inherently non-robust to large changes in the assumed outcome under the implicit control condition); nonetheless, you can have an experiment without a control group.
Andrew: The interesting question is whether there is a situation in which one should run an uncontrolled experiment even when it is possible to find controls.
Givewell wants to put lithium in your drinking water: http://statmodeling.stat.columbia.edu/2016/01/07/28459/
All we need now is Peter Sellers and Sterling Hayden to write the IRB….
https://www.youtube.com/watch?v=Qr2bSL5VQgM
We perform a lot of studies comparing one treatment to another established treatment, as for instance to establish that two drug formulations have similar bioavailability. A control treatment isn’t necessarily required.
I believe there are also ethical issues in some cases. This isn’t my field but if 30 patients are dying of a rare disease that nobody has ever survived, and you have an experimental treatment available, it might be unethical to give 15 of them a placebo. The resulting data is probably a little less valuable but far from worthless.
Many clinical trials lack control groups. In phase 1 trials you administer the drug to healthy volunteers to check if there are side-effects and what’s the tolerable dose, without a control group. Phase 2 trials (exploratory) are often single-arm (i.e. without a control group).
Several drugs have been approved based on single-arm trials. See for example https://www.focr.org/sites/default/files/Non-Randomized%2BTrials.pdf and http://www.ema.europa.eu/ema/index.jsp?curl=pages/news_and_events/events/2016/05/event_detail_001285.jsp
Of course there is always a “baseline”, either implicit or explicit. The use of the “natural history” of rare diseases as a benchmark is sometimes controversial (maybe the data is incomplete, the trial population is not comparable or there are reasons to suspect that this “prior” is stale). But if the effect of the treatment is strong enough there are no issues.
If memory serves, we hanged German and Japanese “researchers” for this type of thing right after WWII.
Still, if you are going to do it do it right so that after we hang you, you at least get an acknowledgement in the subsequent paper.
From the article by Bloche
“Our real concern about what Dr. Mitchell and Dr. Jessen did or didn’t do isn’t human experimentation; it’s torture. Collection of data from victims is morally incidental.”
Recall how the Nazis did terrible “experiments” such as seeing how long prisoners could survive in cold water. The moral issue after the war involved whether or not this data, taken under inhuman circumstances, could or should be used for the better understanding of hypothermia. From
https://en.wikipedia.org/wiki/Nazi_human_experimentation#Modern_ethical_issues
“This, together with the recent use of data from Nazi research into the effects of phosgene gas, has proven controversial and presents an ethical dilemma for modern physicians who do not agree with the methods used to obtain this data. Some object on an ethical basis, and others have rejected Nazi research purely on scientific grounds, pointing out methodological inconsistencies.”