Pollster Doug Schoen illustrates a general point: There’s more than one way to present survey results

Everybody knows how you can lie with statistics by manipulating numbers, making inappropriate comparisons, misleading graphs, etc. But, as I like to remind students, the simplest way to lie with statistics is to just lie! there’s more than one way to interpret data, even from your own poll. You see this all the time, Sometimes you see advocates who make up numbers or present numbers with such little justification that they might as well be made up (as in this purported survey of the “super-rich”).

Here I’m not talking about the innumeracy of a Samantha Power or a David Runciman, or Michael Barone-style confusion or Gregg Easterbrook-style cluelessness or even Tucker Carlson-style asininity. No, I’m talking about flat-out lying what I would consider misleading representation by a professional who has the numbers and deliberately chooses to misrepresent them.

The culprit is pollster in question is Doug Schoen, and the catch was made by Jay Livingston. Schoen wrote the following based on a survey he took of Occupy Wall Street participants:

On Oct. 10 and 11, Arielle Alter Confino, a senior researcher at my polling firm, interviewed nearly 200 protesters in New York’s Zuccotti Park. Our findings probably represent the first systematic random sample of Occupy Wall Street opinion.

Our research shows clearly that the movement doesn’t represent unemployed America and is not ideologically diverse. Rather, it comprises an unrepresentative segment of the electorate that believes in radical redistribution of wealth, civil disobedience and, in some instances, violence. . . .

Unfortunately for Schoen, he made the mistake of sharing his data with Azi Paybarah, who posted the some of the raw numbers:

What would you like to see the Occupy Wall Street movement achieve? {Open Ended}

35% Influence the Democratic Party the way the Tea Party has influenced the GOP
4% Radical redistribution of wealth
5% Overhaul of tax system: replace income tax with flat tax
7% Direct Democracy
9% Engage & mobilize Progressives
9% Promote a national conversation
11% Break the two-party duopoly
4% Dissolution of our representative democracy/capitalist system
4% Single payer health care
4% Pull out of Afghanistan immediately
8% Not sure

Here’s Livingston catching a couple more errors by Schoen:

There are other ways to misinterpret survey results. Here is Schoen in the WSJ:

Sixty-five percent say that government has a moral responsibility to guarantee all citizens access to affordable health care, a college education, and a secure retirement—no matter the cost.

Here is the actual question:

Do you agree or disagree with the following statement: Government has a moral responsibility to guarantee healthcare, college education, and a secure retirement for all.

“No matter the cost” is not in the question. As careful survey researchers know, even slight changes in wording can affect responses. And including or omitting “no matter the cost” is hardly a slight change.

As evidence for the extreme radicalism of the protestors, Schoen says,

By a large margin (77%-22%), they support raising taxes on the wealthiest Americans,

Schoen doesn’t bother to mention that this isn’t much different from what you’d find outside Zucotti Park. Recent polls by Pew and Gallup find support for increased taxes on the wealthy ($250,000 or more) at 67%. (Given the small sample size of the Zucotti poll, 67% may be within the margin of error.) Gallup also finds the majorities of two-thirds or more think that banks, large corporations, and lobbyists have too much power.

I could be charitable and label these last two errors as accidental: perhaps Schoen is so out of it that he did not actually realize that most Americans support higher taxes on the rich, and perhaps he did not bother to read the wording of his own survey item on health care attitudes.

But the first mistake—labeling the group as supporting “radical redistribution of wealth” when his own survey reported only 4% with that opinion—that’s just flat-out unethical. I can’t see how it could’ve happened by accident.

Spin is bad enough—I’d think that the job of a pollster is to get the numbers and then leave the misleading manipulations to others—but to me this seems worse than mere spin. It crosses the line into unprofessional behavior.

P.S. No, there’s no reason to be surprised to see unethical shady characters in politics. But I think it’s important for statisticians and political scientists to call out these people who abuse the trust that we have built up, based on decades of research on sampling and opinion polling.

P.P.S. (10 Mar 2016) I’ve edited the title and some of this post because it was over the top to say that Schoen was lying. See also this post by Chris Wilson for further context.

tl;dr: I don’t think Schoen accurately summarized his poll, but ultimately this comes down to interpretation, so I shouldn’t have said he was lying.

20 thoughts on “Pollster Doug Schoen illustrates a general point: There’s more than one way to present survey results

    • I didn’t contact him directly but in any case I doubt I’ll hear from him. It’s hard for me to see that he’d get anything out of responding. It would just drag him in deeper. I never heard from Easterbrook either. I assume these people have long ago decided that it’s better for them to ignore all criticism.

      The only time I recall getting a response from someone I was mean to was Jonathan Chait. I didn’t agree with him on the merits of the argument but I admired his willingness to argue rather than to simply act as if the criticism had never happened.

  1. I can’t see how it could’ve happened on purpose.

    I assume you mean that it had to have happened on purpose, which is to say, you can’t see how it couldn’t have happened on purpose.

  2. From high school onward, I’ve figured statistics as a key tool in extracting better approximations to reality, knowing that people can easily disagree legitimately. Of course having read “How to Lie with Statistics” then, too, I certainly knew of the reverse.

    But actually, I observe there are at least 2 flavors of the reverse:

    1) Attempt to lie with statistics to convince people of a model that radically differs from reality.

    2) Attempt to create doubt, in effect to convince people of much larger uncertainty than there really is, or that experts are arguing It is usually easier to create confusion than clarity, which has made this a favored tactic, I.e., try Oreskes and Conway’s Merchants of Doubt (2010). The tobacco guys were the best.

    If this framework makes sense, where would Schoen fit?

  3. My Introduction to Statistics Class at Cornell was taught by Prof Lionel Wiess. He started the first class by saying he was going to teach us to make numbers lie. It got me to pay attention and go to class for a whole term. A rare accomplishment for me. Great Guy!

  4. I originally shared this story with my friends but then pulled it back. I had first assume the pollster was actually tweaking the numbers. That doesn’t seem to be the case. Let’s see what the grievances are:

    “More to the point, that “large majority” opposed to free-market capitalism is 4% – eight of the people interviewed. Another eight said they wanted “radical redistribution of wealth.” So at most, 16 people, 8%, mentioned these goals. ”

    Livingston seems thinks that only these two statements constitute opposition to free-market capitalism. Sixty-five percent of the protesters agree that the government has a moral responsibility to provide healthcare. That is at least 65% who reject free-market capitalism, a large majority.

    ““No matter the cost” is not in the question. As careful survey researchers know, even slight changes in wording can affect responses. And including or omitting “no matter the cost” is hardly a slight change.”

    No matter the cost is a comment of the writer on the response to a question. I agree it’s ambiguous and good be construed as being part of the question. That said, if costs are a factor, then there is not “guarantee”.

    “As evidence for the extreme radicalism of the protestors, Schoen says,”

    I read this as support that the protestors don’t support free market capitalism, not as support for extreme radicalism.

    “That means that half the protesters were never politically active until Occupy Wall Street inspired them. ”

    And what is the baseline? Suddenly this isn’t important anymore?


    Overall I agree that the article claims more than is supported by the data, but that isn’t lying. The data was not manipulated, it is presented candidly in the article.

    • I think that the phrase “an unrepresentative segment of the electorate that believes in radical redistribution of wealth, civil disobedience and, in some instances, violence” is an unethical distortion of a poll in which only 4% of respondents mentioned “radical redistribution of wealth.”

      And I think it’s a lie to add “no matter the cost” to a survey question that does not ask this.

      Schoen did the poll himself. If he wanted to ask “no matter the cost,” he could’ve done so. But he didn’t. Maybe because he had a feeling that if he added that phrase, his numbers would be change. It’s so much easier just to lie to the readers of the Wall Street Journal. After all, the vast majority of them don’t read Azi Paybarah, Jay Livingston, or me. That’s the beauty of the mass media: you can make something up, and if the refutation doesn’t hit the front page somewhere, you’ve done your job. And somehow I don’t see the Wall Street Journal running a correction notice on this one any time soon.

      • Still, even unethical pollsters create polls that are useful. Knowing, for instance, that 5% of Occupy protesters support a Flat Tax presents useful knowledge, even if that knowledge is “don’t generalize these Occupiers as being leftist”.

  5. I can’t see how it’s possible for Arthur B. to have read the article the way he did. Gelman is clearly right… Is it really necessary to explain his points further? I don’t think so…

    Anyway, nice piece. It’s important that the people can trust in the work and opinion of top academics…

  6. Pingback: The easy way out. | Stats Chat

  7. I really like this blog, but you really sound like a huge dick in posts like this, where you feel the need to list out a bunch of names and an antagonistic adjective for each one, when its not even relevant to the post at hand. Of course there are a lot of intellectuals and public figures who do not know about statistics as much as you do, but its unnecessary and not at all productive to keep a list of these individuals and call them names. Its just immature and makes me want to stop reading.

    • I’m glad you like the blog.

      To clarify, and setting aside the “huge dick” thing, I referred to Power, Runciman, Barone, etc., to contrast Schoen’s intentional misrepresentations to their unintentional mistakes.

      I’ll also say this. Nobody has ever asked me to write for the New Yorker or the London Review of Books or even Reuters. When I write for a large-circulation journal (or even a small-circulation journal), I take care to get my facts right. As both a producer of news and a consumer of research, I do not appreciate people with a wide audience who don’t bother to check their numbers or their concepts. It’s not about these people knowing less about statistics than I do, it’s about them not seeming to care if they are telling the truth. This angers me, and I don’t mind using my blog to call these people out. I get especially annoyed by someone like Easterbrook who makes big mistakes, then fixes his column based on my corrections but without citing me in any way, then comes back next week with another crappy column.

      Nobody is holding a gun to these people’s heads and making them publish their writings, and I think they could show the decency to at least run their ideas past an expert before broadcasting them.

  8. Pingback: Doug Schoen has 2 poll reports « Statistical Modeling, Causal Inference, and Social Science

  9. bit late to the party here…

    one thing I noticed was that the % in the question add up to 100%, which implies that the respondent could only choose one answer. It could well be that, say, 50%+ actually believe in ‘radical redistribution of wealth’ or many of the other responses, but they just don’t see that as the primary aim of OWS

  10. Pingback: Occupy Wall Street Opinion Poll - Page 2 - Christian Forums

  11. Andrew: “I could be charitable and label these last two errors as accidental: perhaps Schoen is so out of it that he did not actually realize that most Americans support higher taxes on the rich, and perhaps he did not bother to read the wording of his own survey item on health care attitudes.”

    If he’s a professional pollster, then he’d have to be rather ignorant and incompetant. Considering that his ‘errors’ and ‘ignorance’ are rather politically aligned, ……….

Comments are closed.