In which I compare “POLITICO’s chief political columnist” unfavorably to a cranky old dead guy and one of the funniest writers who’s ever lived

Neil Malhotra writes:

I just wanted to alert to this completely misinformed Politico article by Roger Simon, equating sampling theory with “magic.” Normally, I wouldn’t send you this, but I sent him a helpful email and he was a complete jerk about it.

Wow—this is really bad. It’s so bad I refuse to link to it. I don’t know who this dude is, but it’s pitiful. Andy Rooney could do better. And I don’t mean Andy Rooney in his prime, I mean Andy Rooney right now. The piece appears to be an attempt at jocularity, but it’s about 10 million times worse than whatever the worst thing is that Dave Barry has ever written.

My question to Neil Malhotra is . . . what made you click on this in the first place?

P.S. John Sides piles on with some Gallup quotes.

15 thoughts on “In which I compare “POLITICO’s chief political columnist” unfavorably to a cranky old dead guy and one of the funniest writers who’s ever lived

  1. Can you at least give the title of the article? It seems a bit unfair to publicly berate him without giving others the chance to see why. (I looked on Politico, and didn’t immediately see anything that fit the description.

  2. I confess that I don’t see the evil in the Politico piece. Maybe the author has a history of such things that makes it clearer, but I’m not able to spot the felony in this one.

    The author seems well aware of the statistical justification for making inferences from small samples, but to paraphrase the author’s point: there are more things in heaven and earth, Andrew, then are dreamt of in your sampling models.

    Polls in fact, are two levels removed from reality. First the pollster has to ask his victim about their state of mind. Any response is a tiny and stale model of their actual mind. I’ve been telling people my state of mind for decades for example, yet people still can’t predict accurately what I’m going to next.

    Then the pollster has to infer the big population from the small one, which if the last Survey Sampling text I looked at was any indication, is far more problematic than the Random Sampling folklore we tell Political Science freshmen would suggest.

    Certainly it wouldn’t be hard to find professional polls reported over the same days, about the same question, who’s confidence intervals fail to overlap more than the stated confidence would suggest.

    So even if a poll is done right, it isn’t all of reality. It’s not even all of the relevant bits for that specific question. Reminding oneself of that hardly means you’ve declared war against Rationality.

  3. That Roger Simon column doesn’t seem all that outrageous.

    Simon makes two basic points– one sadly correct and the other amusingly false.

    Point one is that American political polling is sloppy, non-scientific, and way over-emphasized in the media. (TRUE)

    Point two is that the statistical sampling basis of such public opinion polling is logically invalid due to the small sample sizes versus the huge American population of 313 Million people. (FALSE)

    Point One is rarely addressed in the dominant media, but it is absolutely true & very significant.
    Arianna Huffington was one of the rare media-types who did a good addressing this huge problem, in her columns back in 1998.

    Point Two is a very common and naive viewpoint on polling & sampling — one hears it frequently from Joe-Sixpack on call-in shows and blogs. Simon says: “I have never been called by a political pollster and don’t know anybody who has…”. Andy Rooney likely agrees/d.

    Supposed intolerable affront to the absolute sanctity of statistical-sampling … is insufficient reason to condemn Roger Simon’s entire message. Real-world political polling is indeed a hotbed of deception & error. Even some academic masters of statistics seem oblivious to this problem.

    • Andy Rooney did write about polls over the years. Here’s a (non-random) sample:

      1980: http://goo.gl/HVlKX

      1984: http://goo.gl/Tx6P4

      1992: http://goo.gl/O9TTn

      2010: http://www.cbsnews.com/stories/2010/11/12/60minutes/main7048950.shtml

      In the latter, he wrote: “Despite all the surveys I read, I’ve never been asked what I think about anything. My answers would be a lot different than the answers they say people give. They should survey people about what they think of surveys.” This seems to be on par with the Simon passage Caldwell quotes above.

      And in 1984: “The poll-takers question as few as 1,500 people to determine what it is 230 million Americans think.” He merely states this as a fact, however, and doesn’t repeatedly mention the sampling fraction like Simon does.

      Although Simon raises some important points in the rest of his column, the problem with his article as a whole is twofold: a) he propagates that fallacy about the relevance of the sampling fraction to the general public; and b) those aware of its irrelevance will likely question the accuracy of his other writings — if he he doesn’t care enough to exercise due diligence before publishing this, what else has he failed to fact-check before using in his columns?

  4. It’s not so dumb. He’s a writer who confesses he doesn’t understand the CLT, or how surveyors can get a good answer (including uncertainty) despite circumstances which one would expect to create differential non-response, or how they can really identify likely voters. That’s fair enough, especially the latter two are complex topics which some companies legitimately do a bad job at. I read the opening and second page as describing how most people interpret polls, which he rightfully criticizes for not taking into account the uncertainty of future movements.

  5. So this article seems to make 2 cases:
    1. ‘Polls are not a good instrument for exploring public opinion’
    and
    2. ‘The public is influenced by the media, who base reporting on polls because it is easy, without really understanding how they should be interpreted’.

    The case for 1. is made very poorly, largely by appealing to ‘common sense’ like ‘well they didn’t ask anyone I know so maybe they make the results up’. This is a dumb argument, so let’s pretend it’s a straw-man to set up 2.

    The article argues that journalists like the author don’t really understand polling very well, and does so with great wit. Demonstrating a perverted pride in his lack of curiosity about basic statistics, the author makes striking generalisations, presents what ever evidence confirms his bias, and generally implies more than he’s willing to state.
    So if he’s arguing that journalists need to do a stats evening class, or see it as their roll to manipulate the reader and just use polls as a tool for doing that, I think he won.

  6. To all:

    The problem to me with Simon’s article was that he (a) made a terrible mistake about sampling, and (b) was coming at it from a position of authority. If Politico’s chief political columnist, it’s not so much to ask that he at least have some vague idea of how a sample can represent the population. Politico reports on surveys all the time. Or, if he really doesn’t understand and doesn’t want to understand, he could ask someone to help him write the column.

    • “made a terrible mistake about sampling”

      I don’t think he did. He seems to concede that the calculation used to choose the sample size (N~1500) is done correctly and is meaningful within the sample model.

      However, in the real world there are problems with the sampling model. So the low sampling size (only a few thousands) possibly makes the poll’s implications sensitive to imperfections in the sampling model. If I were to paraphrase his point it would be:

      “The small sample size may be correct as far as it goes, but it may also make the final results sensitive to dirty real world imperfections. I’m in no position to evaluate this possibility, so I dunno whether it’s a problem or not”

      • Joseph:

        I think Simon’s repeated invocation of the sampling fraction (0.0003 percent) represents a serious error. Nonresponse is an issue. The total sampling fraction is not. Any knowledgeable person could’ve told him that.

        • Agreed – the truly idiotic aspect of the .0003 statement is that if you applied the same idea to other settings, it would be absurd. “The phlebotomist only drew .04 percent of my blood – how can she draw conclusions from such a small amount?”

        • Oh come on Andrew, it’s nothing of the sort. He’s not writing an anti-poll article, he writing an article about the overreliance on polls, especially by people like him for whom the details are a Black Box (or poetically “magic”).

          The first mention of .0003 illustrates that perfectly. He’s talking about the details of a reliable poll and the main focus is on how few details are given. Then he states most journalists don’t know or even care about those details and wouldn’t be able to evaluate them anyway. And then finally, he states that he believes the details are correct because he trusts the experts not because he’s verified them himself.

          There is no incorrect technical claim in any of that, nor is any of it unreasonable or irrational.

          The second mention was “Does Obama really lead Gingrich by 8 percentage points in a (currently) imaginary matchup? I dunno. Sounds right to me. But I’m an even smaller sample than .0003 percent”

          This is actually an extremely good point. What exactly does a poll pitting Obama and Gingrich actually measure? No one is going to claim it is a prediction of what’s going to happen next November. They’d more likely say “if there were an election held today between the two Obama would win by 8 points”. But this, if taken literally, is a fantasy. We don’t have an alternate Universe handy where such an election is held today (but everything else stays the same) in order to test this proposition.

          Then he makes the point that his belief in the correctness of this has at least as much to do with his internal qualitative estimate as it does the results of the poll. There’s almost a Bayesian quality to it.

          The whole point of the article, including the mentions of “.0003”, is that there is a lot of reality not captured by polls, not that polls are bad.

          The author’s point touches a nerve with me frankly. Most of my applied statistics work has been Military/tactical in nature. In all that work, I’ve never once seen a Combat Commander put troop’s lives at risk because he irrationally believed statistics was bunk. On the other hand, I see professional statisticians every week, put troops at risk because they fail to appreciate how much reality isn’t captured in the numbers and models.

          So if some pundit states that he believes reputable polls, but he doesn’t really believe it’s the whole story, then I’m rooting for him. The rest of you pedants can send him primers explaining how “combinatorial factors grow with sample size” if you want.

        • Joseph:

          I’ve never written the phrase “combinatorial factors grow with sample size” in my life, so no worry there.

          I know nothing about military tactics, and I am sorry to hear that professional statisticians are putting troops at risk. If Simon wants to write about such problems, I’d prefer that he write about them directly rather than misleadingly parading numbers such as .0003 which are irrelevant to the real concerns about representativeness of a survey.

        • Andrew: That was the point I was struggling to get across here http://statmodeling.stat.columbia.edu/2010/04/when_engineers/

          As professional statisticians, we often _cause_ more un-necessary deaths than any (single) reckless Combat Commander ever could.

          (e.g. If PSA screening does do more good than harm then many people who would have gotten screened given adequate versus inadequate evidence as it now stands, will _needlessly_ die.)

          Fortunately for me, before this sank in, I had numerous conversations with how clinicians come to terms with this aspect in their work – you win some and you lose some.

Comments are closed.