There’s been some discussion recently about an experiment done in Montana, New Hampshire, and California, conducted by three young political science professors, in which letters were sent to 300,000 people, in order to (possibly) affect their voting behavior. It appears that the plan was to follow up after the elections and track voter turnout. (Some details are in this news report from Dylan Scott.)
The Montana experiment was particularly controversial, with disputes about the legality and ethicality of sending people an official-looking document with the Montana state seal, the official-looking title of “2014 Montana General Election Voter Information Guide,” and the instructions, “Take this to the polls!”
The researchers did everything short of sending people the message, “I am writing you because I am a prospective doctoral student with considerable interest in your research,” or, my all-time favorite, “Our special romantic evening became reduced to my wife watching me curl up in a fetal position on the tiled floor of our bathroom between rounds of throwing up.”
There’s been a bunch of discussion of this on the internet. From a statistical standpoint, the most interesting reaction was this, from Burt Monroe commenting on Chris Blattman’s blog. Burt writes:
In my “big data guy” role, I have to ask why 100,000? As far as I can tell there’s exactly one binary treatment, or is there some complicated factorial interaction that leads to a number that big? For that to be the right power number, the treatment effect has to be microscopic and the instrument has to be incredibly weak — that is, it has to be a poorly designed experiment of a substantively unimportant question. Conversely, if the team believes the treatment will work, 100000 is surely big enough to actually change outcomes. . . .
Related to this is the following quote from the Dylan Scott news article, from a spokesman for one of the universities involved, that the now-controversial study “was not intended to favor any particular candidate, party, or agenda, nor is it designed to influence the outcome of any race.”
But a treatment that affects voter turnout will, in general, influence the outcome of the race (the intervention in Montana involved giving voters an estimate of the ideological leanings of the candidates in a nonpartisan (or, according to Chris Blattman, a “technically nonpartisan”) judicial election, along with the prominently-displayed recommendation to “Take this to the polls!”).
So I’m not sure how they could say the study was not designed to influence the outcome—unless what was done was to purposely pick non-close elections where a swing of a few hundred votes wouldn’t make a difference in who won. Even then, though, the election could be affected if people are motivated by the flyer to come out to vote in the first place, assuming there are any close elections in the states or districts in question.
I will set aside any ethical questions about the extent to which academic researchers should be influencing elections or policy, via experiments or other activities. Chris Blattman makes a good point that ultimately in our research we typically have some larger political goal, whether it’s Robert Putnam wanting to increase Americans’ sense of local community, or Steven Levitt wanting more rational public policy, or various economists wanting lower tariff barriers, and research is one way to attain a political end. The election example is a bit different in that the spokesman for one of the universities involved is flat-out denying the intention of any agenda, but Chris might say that, whether or not that spokesman knows what he’s talking about, having an agenda is basically ok.
To put it another way: Suppose the researchers in question had done this project, not using foundation funding, but under the auspices of an explicitly political organization such as the Democratic or Republican party, and with an avowed goal of increasing voter turnout for one candidate in particular. (One of the researchers involved in this study apparently has political consulting company, so perhaps he is already doing such a study.) Lots of social scientists work with political or advocacy organizations in this way, and people don’t generally express any problems with that. To the extent that the study in question is, ummm, questionable, it’s the idea that it violates an implicit wall separating research with a particular political or commercial purpose, and research with no such purpose. I have mixed feelings here. On one hand, Chris is right that no such sharp boundary exists. On the other hand, I understand how voters in Montana might feel a bit like “lab rats” after hearing about this experiment—even while a comparable intervention done for one party or another would be considered to be just part of modern political advertising.
But I don’t want to get into that here. What I do want to discuss is the statistical question raised by Burt Monroe in the above quote: why a sample size of 100,000 for that Montana experiment? As Monroe points out, 100,000 mailers really could be enough to swing an election. But if you need 100,000, you must really be studying small effects, or maybe localized effects?
When we study public opinion it can be convenient to have a national sample size of 50,000 or more, in order to get enough information on small states interacted with demographic slices (see, for example, this paper with Yair). And in our Xbox survey we had hundreds of thousands of respondents, which was helpful in allowing us to poststratify and also to get good estimates day by day during the pre-election period.
In this election-mailers study it’s not clear why such a large sample was needed, but I’m guessing there was some good reason. The study was expensive and I assume the researchers had to justify the expense and the sample size. A bad reason would be that they expected a very small and noisy effect—as Burt said, in that case it’s not clear why it would be worth studying in the first place. A good reason might be that they’re interested in studying effects that vary by locality. Another possibility is that they are studying spillover effects, the idea that sending a mailer to household X might have some effect on neighboring households. Such a spillover effect is probably pretty small so you’d need a huge sample size to estimate it, but then again if it’s that small it’s not clear it’s worth studying, at least not in that way.
P.S. Full disclosure: I do commercial research (Stan is partly funded by Novartis) and my collaborator Yair Ghitza works at Catalist, a political analysis firm that does work for Democrats. And my office is down the hall from that of Don Green, who I think has done work for Democrats and Republicans. So not only do I accept partisan or commercial work in principle, I support it in practice as well. I’m not close enough to the details of such interventions to know if it’s common practice to send out fake-official mailers, but I guess it’s probably done all the time.
P.P.S. No worry about the prof who was last seen lying curled up in the fetal position on his bathroom floor. He landed on his feet and is a professor at the Stanford Business School. Amusingly enough, he is offering “breakfast briefings” (no throwing up involved, I assume) and studies “how employees can develop healthy patterns of cooperation.”