Anthony Fowler and Andy Hall write:
We reassess Achen and Bartels’ (2002, 2016) prominent claim that shark attacks influence presidential elections, and we find that the evidence is, at best, inconclusive. First, we assemble data on every fatal shark attack in U.S. history and county-level returns from every presidential election between 1872 and 2012, and we find little systematic evidence that shark attacks hurt incumbent presidents or their party. Second, we show that Achen and Bartels’ finding of fatal shark attacks hurting Woodrow Wilson’s vote share in the beach counties of New Jersey in 1916 becomes substantively smaller and statistically weaker under alternative specifications. Third, we find that their town-level result for beach townships in Ocean County significantly shrinks when we correct errors associated with changes in town borders and does not hold for the other beach counties in New Jersey. Lastly, implementing placebo tests in state-elections where there were no shark attacks, we demonstrate that Achen and Bartels’ result was likely to arise even if shark attacks do not influence elections. Overall, there is little compelling evidence that shark attacks influence presidential elections, and any such effect—if one exists—appears to be substantively negligible.
The starting point here is a paper presented by political scientists Chris Achen and Larry Bartels in 2002, “Blind Retrospection – Electoral Responses to Drought, Flu and Shark Attacks.” here’s a 2012 version in which the authors trace “the electoral impact of a clearly random event—a dramatic series of shark attacks in New Jersey in 1916” and claim to “show that voters in the affected communities significantly punished the incumbent president, Woodrow Wilson, at the polls,” a finding that has been widely discussed in political science over the past several years and was featured in Achen and Bartels’s recent book, Democracy for Realists.
Fowler and Hall make a convincing case that Achen and Bartels’s analysis was statistically flawed, that the statistically significance obtained in that observational analysis is misleading. The problems were correlation (making p-values not follow their nominal distribution) and forking paths (making it that much easier for the original researchers to find a statistically significant pattern and fit a story around it).
See more at the sister blog—for some reason, they don’t like cross-posting so I’ll send you there for the full story. Also, as discussed at the end of that post, I think Achen and Bartels have some good points in their book; their larger arguments do not rely on the validity of that shark-attack study.
Graphs can mislead and graphs can inform
There’s one thing I did want to talk about here which did not come up in that other post, and that’s the role of statistical graphics. Here’s the graph from Achen and Bartels showing the dramatic effect of shark attacks in the New Jersey 2016 election:
It looks pretty impressive. Actually, though, as Fowler and Hall discuss, it’s not such a stunning pattern:
1. Election results are correlated, so those 4 beach counties are not really 4 independent data points.
2. This graph is not actually displaying the raw data! Check out the y-axis label: “adjusted for machine counties.” If you look at this scatterplot Fowler and Hall made of the raw data, those 4 beach counties don’t stand out at all:
This is not to say that one shouldn’t plot adjusted data—I do it all the time, it’s a useful statistical technique—but the point is that the adjustment matters, and that’s where forking paths come in. There are many different adjustments one might make.
So, yes, that graph looks pretty convincing but it’s hiding some serious statistical problems, which is why Fowler and Hall convincingly argue that it’s all too easy to find statistical significance in this setting, even in the absence of any underlying effect.
The graph of towns shows some interesting stuff as well. In the left panel, we highlight Sea Side Park, the “treated” town that was accidentally omitted from their analysis, and which had a positive swing for Wilson after the attacks. In the right panel, we show how the analysis changes once you (a) bring in the towns in the other beach counties and (b) include the two Matawans (borough and township, respectively), where two of the four attacks occurred (up river from the beach, which is why A+B choose to omit them)—these two towns also saw positive swings towards Wilson in 1916.
P.S. Best comment comes from Raghu, who writes:
When I started reading, I assumed that “shark attack” was some sort of metaphor for something political — a vicious ad, maybe. But no, it’s really shark attacks!
Indeed. Shark attacks, himmicanes, power pose, the contagion of obesity . . . it’s been a wacky decade or so in social science. Let’s hope we’re getting over it.
P.P.S. I just happened to notice this uncritical review of Achen and Bartels book. No mention of shark attacks, though!