A downside of following good graphical practice

I love stories and for a long time have wanted to put together a little book of my favorite statistics stories. I know this is not something that would ever reach David Sedaris levels of popularity (to say the least) but at least it would give me some good material to use at the beginning of class or for other times when I want to engage students in a way that’s not too taxing for them. (In the meantime, I recommend that all of you who teach statistics or methods classes begin each of your classes, while the students are walking in, with a 5-minute discussion of whatever the latest items are on this blog.)

Anyway, I have a new story right here for ya.

A couple of days ago, Matt Yglesias wrote in his blog that, “In the US and in Europe, income level is fairly predictive of voting behavior . . . It reflects the fact that politics is, among other things, a concrete contest over concrete economic interests.” Reading this reminded me of John Huber and Piero Stanig’s research on voting for left and right parties around the world, where, among other things, they found that in European countries there was not typically a big difference between left and right groupings, either in economic policy or in differential voting by the rich and poor. I sent Yglesias a message to this effect and attached a graph from our recent book, Red State, Blue State, showing the differences between how rich and poor have voted in recent legislative elections in a bunch of countries.

Here’s where the statistical story comes in. When Yglesias posted my graph, there were a bunch of comments, some expected (we hadn’t clearly defined what we meant by “conservative parties” when making the graph) and one that we hadn’t thought of, which was that Russia’s GDP per capita was $14,000, not the $2,600 that we showed in our graph. Further discussion with my collaborators revealed that there are many different GDP numbers out there. In some ways, this makes our life more difficult, but from another perspective it’s not so bad: given that there is no universally accepted number, we can feel more free to use a single standard source (for example, from the World Bank) and simply cite it.

Our graph was about economic voting within countries, so why did we use GDP per capita at all? Because I was following sound graphical principles, which in this case was to display the data graphically and use an informative x-axis, instead of, for example, displaying a table with the countries listed alphabetically. In this case, though, we were really just using GDP per capita on the x-axis as a convenience; the actual values didn’t really matter so much.

The funny story is that, by putting in the effort to apply sound graphical principles, I brought these difficulties upon myself–thus, among other things, spending several hours that I could otherwise have spent doing research, for example. Usually when discussing the costs of good graphical display, I think only of the costs of preparing the graph itself, not the hazards entailed by putting more numbers out there that have to be defended.

In retrospect, I wish we’d made a graph more directly from Huber and Stanig’s numbers, plotting rich-poor voting gap vs. their measure of economic policy differences between left and right parties. When putting together the book, I made the decision to crunch our own numbers, partly so that we’d have some control over the definitions of left and right parties, and partly so that we could use additional countries that Huber and Stanig hadn’t included in their paper. Our decision was a mistake, though, first because we underestimated the amount of work it would take to put the data together, and second because had we purely relied on Huber and Stanig, we could’ve disposed of many questions with a simple reference to their research, keeping ourselves out of the loop.

P.S. Yeah, yeah, Sedaris it ain’t, I know. In context, though, it’s interesting, I think. In class I’d begin it with, “Once upon a time . . .”

9 thoughts on “A downside of following good graphical practice

  1. I'd read that book. What else would be in it?

    I love the story of the lady tasting tea as motivation for randomization in experiments. How the student-t distribution came about and got its name is another great one. There seem to be a lot of crazy stories about Gaussian distributions, like skulls filled with metal beads to measure volumes.

    The story of how the Bertillon measurement system came to be replaced by Galton's fingerprinting system as a way of identifying criminals is a great tale of how a well designed research protocol can generate errors in measurement when introduced into new contexts.

  2. The big lesson I get from this is that there's no such thing as "sound graphical principles". There are certainly guidelines, like how one should prefer a meaningful ordering over an entirely uninformative ordering like alphabetical order.

    But in this case, you were indifferent, and were not trying to tell a story about the relationship between the voting differential and any other variable, like GDP/cap, but just about the voting differential itself. Using an uninformative ordering might have been better, because any significant ordering would imply a relation that you're not trying to imply.

  3. Once upon a time – a senior colleague remarked that rather than finding a reference for power in conditional logistic regression that I should derive the likelihoods myself and do some simulations.

    I retorted – but I am working with a clinical research group on a grant application and that would be unfair as a reference would be safer/more credible (as well as faster).

    Although that likely was a good retort, when I ended up doing those derivations a couple years later I realized I had missed a real opportunity to learn and "grow" as a statistician.

    Then many years later, when I was working on a resubmission of a grant (that was almost funded on the first submission) I did such a simulation and in the spirit of "reproducible research" I included many comments in the R program, the random number seed I had used and the R web site.

    Now the "person" who did the statistical review acquired the R software but thought they had discovered a bug in my code to create treament and control groups and changed the code to a real bug that haphazardly mixed the treatment and control groups … and reported to the review group that they had corrected simmulations and the power was not 83% as claimed but actually only 4% [ this being lower than the nominal type one error rate apparently not suggesting a problem with the new coding ]

    The result was the grant was not even reviewed by the rest of the group but returned as unsuitable to be reviewed along with a lecture on the well known lack of power that arises with the use of conditional logistic regression.

  4. I'm glad I am not the only one hear who enjoyed The Lady Tasting Tea. The story of the woman who measured beer cask volumes is one I frequently cite to illustrate how process influences measure.

    FWIW, I would read a book of statistical anecdotes as well. And I would not confuse them with antidotes… that's how my buddy Billy died. Not cool.

  5. Let's make a distinction here.

    The story of the lady tasting tea was first in R.A. Fisher's book on design of experiments, first published in 1935. A fuller version is given in Joan Fisher Box's biography of her father, published in 1978. That's certainly a good story for this project.

    "The Lady Tasting Tea" is the title of a book by David Salsburg. His version of the story, which is his lead anecdote, gets both place and people wrong, as I pointed out in my review in Biometrics. Although many people have evidently found the book entertaining, it's full of similar historical mistakes and confusions. For several other examples (and far from a complete list), see my review. (The precise volume and page numbers are not to hand as I write this.)

  6. For more stories to motivate statistics and data analysis, see Howard Wainer's book Graphic Discovery. I especially like the anecdote about Wald trying to determine where to put extra armor on planes during WW II (pg. 148).
    Tufte's books may be another source of motivating stories (e.g. discovering the source of the cholera epidemic in London), although they may be more about graphics than statistics.

Comments are closed.