Someone pointed me to a recent post by Nate Silver, “Polling averages shouldn’t be political litmus tests, and they need consistent standards, not make-it-up-as-you-go,” where Nate wrote:
The new Editorial Director of Data Analytics at ABC News, G. Elliott Morris, who was brought in to work with the remaining FiveThirtyEight team, sent a letter to the polling firm Rasmussen Reports demanding that they answer a series of questions about their political views and polling methodology or be banned from FiveThirtyEight’s polling averages, election forecasts and news coverage. I found several things about the letter to be misguided. . . .
First, I strongly oppose subjecting pollsters to an ideological or political litmus test. . . . Why, unless you’re a dyed-in-the-wool left-leaning partisan, would having a “relationship with several right-leaning blogs and online media outlets” lead one to “doubt the ethical operation of the polling firm”? . . .
Rasmussen has indeed had strongly Republican-leaning results relative to the consensus for many years. Despite that strong Republican house effect, however, they’ve had roughly average accuracy overall because polls have considerably understated Republican performance in several recent elections (2014, 2016, 2020). . . . Is that a case of two wrongs making a right — Rasmussen has had a Republican bias, but other polls have had a Democratic bias, so they come out of the wash looking OK? Yeah, probably. Still, there are ways to adjust for that — statistical ways like a house effects adjustment . . .
Second, even if you’re going to remove Rasmussen from the averages going forward, it’s inappropriate to write them out of the past . . . It’s bad practice to revise data that’s already been published, based on decisions you made long after that data was published. For one thing, it makes your numbers less reliable as a historical record. For another, it can lead to overconfidence when using that data to train or build models. . . .
Third, I think it’s clear that the letter is an ad hoc exercise to exclude Rasmussen, not an effort to develop a consistent set of standards. . . . The thing about running a polling average is that you need a consistent and legible set of rules that be applied to hundreds of pollsters you’ll encounter over the course of an election campaign. Going on a case-by-case basis is a) extremely time-consuming . . . and b) highly likely to result in introducing your own biases . . . Perhaps Morris’s questions were getting at some larger theme or more acute problem. But if so, he have should stated it more explicitly in his letter. . . .
Nate raises several interesting questions here:
1. Is there any good reason for a relationship with “right-leaning” outlets such as Fox News and Steve Bannon to cause one to “doubt the ethical operation of the polling firm”?
2. Does it ever make sense to remove a biased poll, rather than including in your analysis with a statistical correction?
3. If you are changing your procedure going forward, is it a mistake to make those changes retroactively on past work?
4. Is it appropriate to send a letter to one polling organization without going through the equivalent process with all the other pollsters whose data you’re using?
Any followups?
I’ll go through the above questions one at a time, but first I was curious if Nate or Elliott had said anything more on the topic.
I found these two items on twitter:
– This from Elliott: “asking pollsters detailed methodological questions is not (or shouldn’t be!) controversial. it’s standard practice in most media organizations, and aggregators should probably even be publishing responses for the public and using them as a way to gauge potential measurement error,” linking to a list of questions that CNN asks of all pollsters.
– This from Nate, referring to Elliott’s letter to Rasmussen as a “Spanish Inquisition” and linking to this article from the Washington Examiner which, among other things, reported this from a Rasmussen poll:
Whaaaaa? As a check, I googled *abortion roe wade polling* and found some recent items:
Gallup: “As you may know, the Supreme Court overturned its 1973 Roe versus Wade decision concerning abortion, meaning there is no Constitutional protection for abortion rights and each state could set its own laws to allow, restrict or ban abortions. Do you think overturning Roe versus Wade was a good thing or a bad thing?”: 38% “good thing,” 61% “bad thing,” 1% no opinion.
CBS/YouGov: “Last year, the U.S. Supreme Court ended the constitutional right to abortion by overturning Roe v. Wade. Do you approve or disapprove of the Court overturning Roe v. Wade?”: 44% “approve,” 56% “disapprove.”
USA Today (details here): “It’s been a year since the Supreme Court overturned the Roe v. Wade decision, eliminating a
constitutional right to an abortion at some stages of pregnancy. Do you support or oppose the court decision to overturn Roe v. Wade?”: 30% “support,” 58% “oppose,” 12% undecided.
There’s other polling out there, all pretty much consistent with the above. An then there’s Rasmussen, which stands out. Would I want to include Rasmussen’s “Majority Now Approve SCOTUS Abortion Ruling” in a polling average? I’m not sure.
Some of it must could be their question wording: “Last year, the Supreme Court overturned the 1973 Roe v. Wade decision, so that each state can now determine its own laws regarding abortion. Do you approve or disapprove of the court overturning Roe v. Wade?” This isn’t far from the Gallup question, but they does remove the “Constitutional protection” phrase, and I guess that could make a difference. Also, they’re just counting “likely voters,” and much could depend on where those respondents come from.
Whether or not it makes sense to take the Rasmussen organization seriously (I remain concerned about their numbers that added up to 108%), I think it’s kinda journalistic malpractice for the Washington Examiner to report their claim of “Support for overturning Roe v. Wade is up since last year. 52% to 44%, US likely voters approve,” without even noting how much that disagrees with all other polling out there. My first thought was that, yeah, the Washington Examiner is a partisan outlet, but even partisans benefit from accurate news, right? I guess the point is that the role of an operation such as the Washington Examiner is not so much to inform readers as to circulate talking points and get them out into the general discussion—indeed, thanks to Nate and then me, it happened here!
1. Is there any good reason for a relationship with “right-leaning” outlets such as Fox News and Steve Bannon to cause one to “doubt the ethical operation of the polling firm”?
OK, now on to Nate’s questions. First, should we doubt the ethics of a pollster who hangs out with Fox News and Steve Bannon? My answer here is . . . it depends!
On one hand, . . . Should we discredit my statistical work because I teach at Columbia University, an institution whose most famous professor was Dr. Oz and which notoriously promulgated false statistics for its college rankings? Lots of people teach at Columbia, similarly lots of people go on Fox News: there’s an appeal to reaching an audience of millions. Going on Fox might be a bad idea, but does it cast doubt on a pollster’s ethics?
As I said, it depends. If a pollster or quantitative social scientist is consistently using crap statistics to push election denial, then, yes, I do doubt their ethics. The relevant point here is not that Fox and Bannon are “right-leaning” but rather that they’ve been fueling election denial misinformation, and distorted election statistics are part of the process.
So, yeah, I agree with Nate that Elliott’s phrase, “several right-leaning blogs and online media outlets,” doesn’t tell the whole story—as Nate put it, “Perhaps Morris’s questions were getting at some larger theme or more acute problem.” There is a larger theme and more acute problem, and that’s refuted claims about the election that have been endorsed by major political and media figures. Given what Rasmussen’s been doing in this area, I think Nate’s been a bit too quick to take their side of the story on this, to refer to Elliott’s inquiries as an “inquisition,” etc. You don’t have to be a “dyed-in-the-wool left-leaning partisan” to doubt the ethical operation of a polling firm that is promoting lies about the election.
How close does a pollster need to be to election deniers so that I don’t trust it at all? I don’t know. I guess it depends on context, which is a good reason for Elliott to ask specific questions to Rasmussen about their polling methodology. If they’re open about what they’re doing, that’s a good sign; if they give no details, that’s gonna make it harder to trust them. Rasmussen has no duty to respond to those questions, Fivethirtyeight has no duty to include its polls in their analyses, etc etc all down the line.
2. Does it ever make sense to remove a biased poll, rather than including in your analysis with a statistical correction?
Discarding a data point is equivalent to including it but giving it a weight of zero or, from a Bayesian point of view, allowing it to be biased with an infinite-variance prior on the bias. So we can transform Nate’s very reasonable implied question (why discard Rasmussen polls? Why not just include your skepticism in your model?) as the question: Why not just give the Rasmussen polls a very small weight or, from a Bayesian point of view, allow them to have a bias that has a very large uncertainty?
There are two answers here. The first is that if the weight is very small or the bias has a huge uncertainty, then it’s pretty much equivalent to not including the survey at all. Remember 13. The second answer is that if these surveys are really being manipulated, then there’s no reason to think the bias is consistent. To put it another way: if you don’t think the Rasmussen polls are providing useful information, then you might not want to include them for the same reason that you wouldn’t include a rotten onion in your stew. Sure, one bad onion won’t destroy the taste—it’ll be diluted amid all the other flavors (including those of all the non-rotten onions you’ve thrown in)—but what’s the point?
This second answer is as much procedural as substantive: by excluding a pollster entirely, Fivethirtyeight is saying they don’t want to be using numbers that they can’t, on some level, trust. They’re making the procedural point that they have some rules for the polls they include, some red lines that cannot be crossed.
From the other direction, Nate’s plea for Fivethirtyeight to continue including Rasmussen’s polls in its analyses is also a procedural and perception-based argument: he’s making the procedural point that “you need a consistent and legible set of rules” and can’t be making case-by-case decisions.
The funny thing is . . . Nate and Elliott are kind of saying the same thing! Elliott’s saying they’ll be removing Rasmussen unless they follow the rules and Nate’s saying that too. I looked up Fivethirtyeight’s rules for pollsters from when Nate was running the organization and it says “Pollsters must also be able to answer basic questions about their methodology, including but not limited to the polling medium used (e.g., landline calls, text, etc.), the source of their voter files, their weighting criteria, and the source of the poll’s funding.” And they don’t include “‘Nonscientific’ polls that don’t attempt to survey a representative sample of the population or electorate.” So I guess a lot depends on the details; see item 4 below.
3. If you are changing your procedure going forward, is it a mistake to make those changes retroactively on past work?
I have a lot of sympathy for Nate’s argument here. He created the Fivethirtyeight polling averages, then combined this with his interest in sports analytics, worked his butt off for over a decade . . . and now the new team is talking about changing things. It would be kind of like if CRC Press hired someone to create a fourth edition of Bayesian Data Analysis, and the new author decided to remove chapter 6 because it didn’t match his philosophy. I’d be furious! OK, that’s not a perfect analogy because my coauthors and I have copyright on BDA, but the point is that Nate was Fivethirtyeight for awhile, so it’s frustrating to think of the historical record being changed.
That said, it’s not clear to me that Elliott is planning to change the historical record. From his quoted letter: “If banned, Rasmussen Reports would also be removed from our historical averages of polls and from our pollster ratings. Your surveys would no longer appear in reporting and we would write an article explaining our reasons for the ban.” It could be that the polls would still be in the database, just flagged and not included in the averages. I think that would be OK.
To put it another way, I think it’s ok to go back and clean up old data, as long as you’re transparent about it.
From a slightly different angle, Nate writes, “There’s also an implicit conflict here about the degree to which journalists should gatekeep or shield the public from potential sources of ‘misinformation.'” I’m not exactly sure of Elliott’s motivations here, but my guess is that his goal is not so much to “shield the public” but rather to come up with more accurate forecasts. Nate argues that including a Republican-biased poll should lead to more accurate forecasts by balancing other polls with systematic polling errors favoring the Democrats. I guess that if Fivethirtyeight going forward is not going to include Rasmussen polls, they’ll have to adjust for possible systematic errors in some other way. That would make sense to me, actually. If you do want to adjust for the possibility of errors on the scale of 2016 or 2020 (polls that showed the Democrats getting approximately 2.5 percentage points more support than they actually received in the vote), then it would make sense to make that adjustment straight up, without relying on Rasmussen to do it for you.
4. Is it appropriate to send a letter to one polling organization without going through the equivalent process with all the other pollsters whose data you’re using?
I have no idea what’s been going on between Fivethirtyeight and Rasmussen and between Fivethirtyeight and other polling organizations. The quoted letter from Elliott to Rasmussen begins, “I am emailing you to send a final notice . . .”, so it seems safe to assume this is just one in a series of communications, and we haven’t seen the others that came before.
Nate writes, “I think it’s clear that the letter is an ad hoc exercise to exclude Rasmussen, not an effort to develop a consistent set of standards.” My guess is that it’s neither an ad hoc exercise to exclude Rasmussen, nor an effort to develop a consistent set of standards, but rather that it’s an effort to apply an imperfect set of standards. Rules such as “Pollsters must also be able to answer basic questions about their methodology, including but not limited to . . .” and “‘Nonscientific’ polls that don’t attempt to survey a representative sample” are imperfect—but that’s the nature of rules.
I guess what I’m saying is that it’s hard to compare Fivethirtyeight’s interactions with Rasmussen with their interactions with other pollsters, given that (a) we don’t know what their interactions with Rasmussen are, and (b) we don’t what their interactions with other pollsters are.
Let me just say that this sort of thing is always challenging, as there’s no way to have completely consistent rules. For example, we have good reasons to be suspicious that Brian Wansink ever used his famous bottomless soup bowl in any actual experiment. Do we apply this level of scrutiny to the apparatus described in every peer-reviewed research article? No, first because this would require an immense amount of effort, and second because “this level of scrutiny” is not even defined. It’s judgment calls all the way down. Fivethirtyeight has a necessarily ambiguous policy on what polls they will include in their analyses—there’s no way for such a policy to not have some ambiguity—and Nate and Elliott are making different judgment calls on whether Rasmussen violates the policy.
Having this discussion
Unfortunately there hasn’t been much of a conversation on this poll-inclusion issue, which I guess is no surprise given that Nate (indirectly) called Elliott a bullshitter and explicitly writes, “I don’t intend this a back-and-forth.” Which is too bad, given that we’ve had good conversations on forecasting before.
It’s easier for me to have this discussion because I know both Nate and Elliott. I don’t know either of them well on a personal level, but I’ve collaborated with both of them (for example, here and here) and I think they both do great work. I’ve criticized Nate’s forecasting procedure; then again, I’ve also criticized Elliott’s, even though (or especially because) it was done in collaboration with me.
To say I like both of them is not an attempt to put myself above the fray or to characterize their disagreements as minor. People often get themselves into positions where they are legitimately angry at each other—it’s happened to me plenty of times! The main point of the present post is that the decisions Elliott is making regarding which polls to include in his analysis, and the questions that Nate is asking, are challenging, with no easy answers.
P.S. Here’s a brief summary of statistical concerns with the 2020 presidential election forecasts from Economist and Fivethirtyeight forecasts. tl;dr: both had problems, in different ways.