David Hogg points me to this post by Thomas Lumley regarding a social experiment that was performed by randomly manipulating the content in the news feed of Facebook customers. The shiny bit about the experiment is that it involved 700,000 participants (or, as the research article, by Adam Kramera, Jamie Guillory, and Jeffrey Hancock, quaintly puts it, “689,003”), but, as Tal Yarkoni points out, that kind of sample size is just there to allow people to estimate tiny effects and also maybe to get the paper published in a top journal and get lots of publicity (there it is, “massive-scale” right in the title).
Before getting to Lumley’s post, which has to do with the ethics of the study, I want to echo the point made by Yarkoni:
In the experimental conditions, where negative or positive emotional posts are censored, users produce correspondingly more positive or negative emotional words in their own status updates. . . . [But] these effects, while highly statistically significant, are tiny. The largest effect size reported had a Cohen’s d of 0.02–meaning that eliminating a substantial proportion of emotional content from a user’s feed had the monumental effect of shifting that user’s own emotional word use by two hundredths of a standard deviation. In other words, the manipulation had a negligible real-world impact on users’ behavior. . . .
The attitude in much of science, of course, is that if you can conclusively demonstrate an effect, that its size doesn’t really matter. But I don’t agree with this. For one reason, if we happen to see an effect of +0.02 in one particular place at one particular time, it could well be -0.02 somewhere else. Don’t get me wrong—I’m not saying that this finding is empty, just that we have to be careful about out-of-sample generalization.
Now on to the ethics question. Lumley writes:
The problem is consent. There is a clear ethical principle that experiments on humans require consent, except in a few specific situations, and that the consent has to be specific and informed. . . . The need for consent is especially clear in cases where the research is expected to cause harm. In this example, the Facebook researchers expected in advance that their intervention would have real effects on people’s emotions; that it would do actual harm, even if the harm was (hopefully) minor and transient.
I pretty much disagree with this, for reasons that I’ll explain in a moment. Lumley continues:
The psychologist who edited the study for PNAS said
“I was concerned,” Fiske told The Atlantic, “until I queried the authors and they said their local institutional review board had approved it—and apparently on the grounds that Facebook apparently manipulates people’s News Feeds all the time.”
Fiske added that she didn’t want the “the originality of the research” to be lost, but called the experiment “an open ethical question.”
To me [Lumley], the only open ethical question is whether people believed their agreement to the Facebook Terms of Service allowed this sort of thing. This could be settled empirically, by a suitably-designed survey. I’m betting the answer is “No.” Or, quite likely, “Hell, no!”.
Amusingly enough, this is the same Susan Fiske who was earlier quoted in support of the himmicanes study, but that doesn’t seem to be particularly relevant here.
I don’t feel strongly about the ethical issues here. On one hand, I’d be a bit annoyed if I found that my internet provider was messing with me just to get a flashy paper in a journal (for example, what if someone told me that some researcher was sending spam to the blog, wasting my time (yes, I delete these manually every day) in the hope of getting a paper published in a tabloid journal using a phrase such as “massively online experiment”). Indeed, a couple of years ago I was annoyed that some researchers sent me a time-wasting email ostensibly coming from a student who wanted to meet with me. My schedule is a mess and it doesn’t help me to get fake appointment requests. On the other hand, as Fiske notes, corporations manipulate what they send us all the time, and any manipulation can possibly affect our mood. It seems a bit ridiculous to say that a researcher needs special permission to do some small alteration of an internet feed, when advertisers and TV networks can broadcast all sorts of emotionally affecting images whenever they want. The other thing that’s bugging me is the whole IRB thing, the whole ridiculous idea that if you’re doing research you need to do permission for noninvasive things like asking someone a survey question.
So, do I consider this Facebook experiment unethical? No, but I could see how it could be considered thus, in which case you’d also have to consider all sorts of non-research experiments (the famous A/B testing that’s so popular now in industry) to be unethical as well. In all these cases, you have researchers, of one sort or another, experimenting on people to see their reactions. And I don’t see the goal of getting published in PNAS to be so much worse than the goal of making money by selling more ads. But, in any case, I don’t really see the point of involving institutional review boards for this sort of thing. I’m with Tal Yarkoni on this one; as he puts it:
It’s not clear what the notion that Facebook users’ experience is being “manipulated” really even means, because the Facebook news feed is, and has always been, a completely contrived environment. . . . Facebook—and virtually every other large company with a major web presence—is constantly conducting large controlled experiments on user behavior.
Again, I can respect if you take a Stallman-like position here (or, at least, what I imagine rms would say) and argue that all of these manipulations are unethical, that the code should be open and we should all be able to know, at least in principle, how our messages are being filtered. So I agree that there is an ethical issue here and I respect those who have a different take on it than I do—but I don’t see the advantage of involving institutional review boards here. All sorts of things are unethical but still legal, and I don’t see why doing something and publishing it in a scientific journal should be considered more unethical or held to a more stringent standard than doing the same thing and publishing it in an internal business report.
P.S. This minor, minor story got to me what seems like a hugely disproportionate amount of attention—I’m guessing it’s because lots of people feel vaguely threatened by the Big Brother nature of Google, Facebook, etc., and this is a story that gives people an excuse to grab onto these concerns—and so when posting on it I’m glad of our 1-to-2-month lag, which means that you’re seeing this post with fresh eyes, after you’ve almost forgotten what the furore was all about.