Skip to content

On deck this week

Mon: How literature is like statistical reasoning: Kosara on stories. Gelman and Basbøll on stories.

Tues: Understanding Simpson’s paradox using a graph

Wed: Advice: positive-sum, zero-sum, or negative-sum

Thurs: Small multiples of lineplots > maps (ok, not always, but yes in this case)

Fri: “More research from the lunatic fringe”

Sat: “Schools of statistical thoughts are sometimes jokingly likened to religions. This analogy is not perfect—unlike religions, statistical methods have no supernatural content and make essentially no demands on our personal lives. Looking at the comparison from the other direction, it is possible to be agnostic, atheistic, or simply live one’s life without religion, but it is not really possible to do statistics without some philosophy.”

Sun: I was wrong . . .

An old discussion of food deserts

I happened to be reading an old comment thread from 2012 (follow the link from here) and came across this amusing exchange:

Screen Shot 2014-02-06 at 11.18.41 AM

Perhaps this is the paper Jonathan was talking about?

Here’s more from the thread:

Screen Shot 2014-02-06 at 11.24.01 AM

Screen Shot 2014-02-06 at 11.24.33 AM

Anyway, I don’t have anything to add right now, I just thought it was an interesting discussion.

Bizarre academic spam

I’ve been getting these sorts of emails every couple days lately:

Respected Professor Gelman

I am a senior undergraduate at Indian Institute of Technology Kanpur (IIT Kanpur). I am currently in the 8th Semester of my Master of Science (Integrated) in Mathematics and Scientific Computing program. I went through some of your previous work and found it to be very interesting, especially ‘Discussion of the article “website morphing”‘. I am interested in working under your guidance in a full time research during this summer (May 2014 – July 2014)

I have a deep interest in Economics (especially Game Theory), Applied Mathematics and Statistics and I have consistently performed well in many courses. My past research experience convinced me of my potential for research and I am in search of an opportunity under your guidance to hone my analytic and research skills

As evident from my resume, most of my work till now hovers around analysis and application of abstract ideas, where in most cases I have had taken up famous research papers (like “On computable numbers with an application to the Entscheidungsproblem, A.M.Turing, 1936”) and build upon them to solve a particular problem, often applying my coding skills and knowledge of statistics. As a result of these experiences, I am confident of my solid problem solving skills

I strongly believe that this opportunity to work under your guidance in a research project would provide me with an invaluable experience in real life research. I would seek this opportunity as a long term commitment to continue working under you in future

Thank You for your time and cooperation. Attached is a copy of my resume for your reference

Yours faithfully

OK, I understand the basic economics here. I live in a rich country, this person lives in a poor country so he wants to come here. The success rate of any pitch is approximately N*p, so I assume he’s going for the traditional spam plan and maximizing N. He has access to a long list of emails of math, stat, econ, and engineering professors in the First World and he’s sending this message to all of us. Finally, he is demonstrating his access to computing skills by stripping out an article with my name on it. But I don’t think this particular student wrote the software to do this. I get so much of this sort of spam that I’m pretty sure there’s a free or pirated program do do this strip-cut-and-paste action.

What amazes me is that these spammers seem uniformly to pick the most inappropriate of my articles for these pitches. Always, it seems, they’ll pick a discussion or a comment or an article on the history of statistics or something else that’s not really so close to my most active research. Maybe it’s something about the program they use to grab an article title? Maybe it purposely takes the title of an article with very few citations on the theory that I’ll be impressed that the student “went through” something obscure?

The whole situation just makes me feel sad. I hate to see people lie. I mean, sure, I liked American Bluff as much as the next guy, but actual lying in real life—especially this sort of thing, a poor person lying to a rich person in a hopeless attempt to climb the ladder of economic opportunity—it’s just a sad, sad thing.

I was going to criticize this on blog but I’m just too tired of things like this. What’s really horrible is the news article which takes all this so seriously. My problem is not with people who run regressions and post them on the web—the more the merrier, I say—but with reputable news outlets whose editors should know better

A friend pointed me to this monstrosity. As an MIT grad, I’d like to think that Technology Review could do better.

To elaborate a bit: A one-paragraph blurb would be fine to me, you can report that someone ran some regressions on the GSS and came up with an amusing hypothesis. That’s enough, then move on to the real technology news of robots playing ping-pong or whatever. I’m not saying to suppress this sort of thing, just place it in the appropriate context. It is what it is. If you want to write a full article on it, fine, but then talk to someone who studies the subject area (you’re Technology Review, you can get these people on the phone!) and move the ball forward a bit. Otherwise why bother at all?

The Notorious N.H.S.T. presents: Mo P-values Mo Problems

A recent discussion between commenters Question and Fernando captured one of the recurrent themes here from the past year.

Question: The problem is simple, the researchers are disproving always false null hypotheses and taking this disproof as near proof that their theory is correct.

Fernando: Whereas it is probably true that researchers misuse NHT, the problem with tabloid science is broader and deeper. It is systemic.

Question: I do not see how anything can be deeper than replacing careful description, prediction, falsification, and independent replication with dynamite plots, p-values, affirming the consequent, and peer review. From my own experience I am confident in saying that confusion caused by NHST is at the root of this problem.

Fernando: Incentives? Impact factors? Publish or die? “Interesting” and “new” above quality and reliability, or actually answering a research question, and a silly and unbecoming obsession with being quoted in NYT, etc. . . . Given the incentives something silly is bound to happen. At issue is cause or effect.

At this point I was going to respond in the comments, but I decided to make this a separate post (at the cost of pre-empting yet another scheduled item on the queue), for two reasons:

1. I’m pretty sure that a lot fewer people read the comments than read the posts; and

2. I thought of this great title (see above) and I wanted to use it.

First let’s get Bayes out of the way

Just to start with, none of this is a Bayes vs. non-Bayes battle. I hate those battles, partly because we sometimes end up with the sort of the-enemy-of-my-enemy-is-my-friend sort of reasoning that leads smart, skeptical people who should know better to defend all sorts of bad practices with p-values, just because they (the smart skeptics) are wary of overarching Bayesian arguments. I think Bayesian methods are great, don’t get me wrong, but the discussion here has little to do with Bayes. Null hypothesis significance testing can be done in a non-Bayesian way (of course, just see all sorts of theoretical-statistics textbooks) but some Bayesians like to do it too, using Bayes factors and all the rest of that crap to decide whether to accept models of the theta=0 variety. Do it using p-values or Bayes factors, either way it’s significance testing with the goal of rejecting models.

The Notorious N.H.S.T. as an enabler

I agree with the now-conventional wisdom expressed by the original commenter, that null hypothesis significance testing is generally inappropriate. But I also agree with Fernando’s comment that the pressures of publication would be leading to the aggressive dissemination of noise, in any case. What I think is that the notorious N.H.S.T. is part of the problem. It’s a mechanism by which noise can be spread. This relates to my recent discussion with Steven Pinker (not published on blog yet, it’s on the queue, you’ll see it in a month or so).

To say it another way, the reason why I go on and on about multiple comparisons is not that I think it’s so important to get correct p-values, but rather that these p-values are being used as the statistical justification for otherwise laughable claims.

I agree with Fernando that, if it wasn’t N.H.S.T., some other tool would be used to give the stamp of approval on data-based speculations. But null hypothesis testing is what’s being used now, so I think it’s important to continue to point out the confusion between research hypotheses and statistical hypotheses, and the fallacy of, as the commenter put it, “disproving always false null hypotheses and taking this disproof as near proof that their theory is correct.”

P.S. “The aggressive dissemination of noise” . . . I like that.

As the boldest experiment in journalism history, you admit you made a mistake

Screen Shot 2014-02-24 at 11.33.22 PM

The pre-NYT David Brooks liked to make fun of the NYT. Here’s one from 1997:

I’m not sure I’d like to be one of the people featured on the New York Times wedding page, but I know I’d like to be the father of one of them. Imagine how happy Stanley J. Kogan must have been, for example, when his daughter Jamie got into Yale. Then imagine his pride when Jamie made Phi Beta Kappa and graduated summa cum laude. . . . he must have enjoyed a gloat or two when his daughter put on that cap and gown.

And things only got better. Jamie breezed through Stanford Law School. And then she met a man—Thomas Arena—who appears to be exactly the sort of son-in-law that pediatric urologists dream about. . . .

These two awesome resumes collided at a wedding ceremony . . . It must have been one of the happiest days in Stanley J. Kogan’s life. The rest of us got to read about it on the New York Times wedding page.

Brooks is reputed to be Jewish himself so I think it’s ok for him to mock Jewish people in print. The urologist bit . . . well, hey, I’m not above a bit of bathroom humor myself—and nor, for that matter, is the great Dave Barry—so I can hardly fault a columnist for finding a laugh where he can.

The interesting part, though, comes near the end of the column:

The members of the cognitive elite will work their way up into law partnerships or top jobs at the New York Times, but they probably won’t enter the billionaire ranks. The real wealth will go to the risk-taking entrepreneurs who grew up in middle- or lower-middle-class homes and got no help from their non-professional parents when they went off to college.

One of the fun things about revisiting old journalism is that we can check how the predictions come out. So let’s examine the two claims above, 17 years later:

1. “The members of the cognitive elite . . . probably won’t enter the billionaire ranks.” Check. No problem there. Almost nobody is a billionaire, so, indeed, most people with graduate degrees who are featured in the NYT wedding section do not become billionaires.

2. “The real wealth will go to the risk-taking entrepreneurs who grew up in middle- or lower-middle-class homes and got no help from their non-professional parents when they went off to college.” Hmmm . . . I googled rich people and found this convenient wikipedia list of members of the Forbes 400. Let’s go through them in order:

Bill Gates
Warren Buffett
Larry Ellison
Charles Koch
David H. Koch
Christy Walton
Jim Walton
Alice Walton
S. Robson Walton
Michael Bloomberg
Sheldon Adelson
Jeff Bezos
Larry Page
Sergey Brin
Forrest Mars, Jr.

Most of these had backgrounds far above the middle class. For example, of Gates, “His father was a prominent lawyer, and his mother served on the board of directors for First Interstate BancSystem and the United Way.” Here’s Buffett: “Buffett’s interest in the stock market and investing also dated to his childhood, to the days he spent in the customers’ lounge of a regional stock brokerage near the office of his father’s own brokerage company.” Koch: “After college, Koch started work at Arthur D. Little, Inc. In 1961 he moved back to Wichita to join his father’s business, Rock Island Oil & Refining Company.” And I don’t think I have to tell you about the backgrounds of the Waltons or Forrest Mars, Jr. Larry Page had more of a middle class background but not the kind that David Brooks was looking for: “His father, Carl Page, earned a Ph.D. in computer science in . . . and is considered a pioneer in computer science and artificial intelligence. Both he and Page’s mother, Gloria, were computer science professors at Michigan State University.” And here’s Sergei Brin: “His father is a mathematics professor at the University of Maryland, and his mother a researcher at NASA’s Goddard Space Flight Center.” Damn! Foiled again. They might have even really violated Brooks’s rule and paid for Brin’s college education.

That leaves us with Larry Ellison, Sheldon Adelson, Michael Bloomberg, and Jeff Bezos: 4 out of the Forbes 15. So, no, I think Brooks would’ve been more prescient had he written:

The real wealth will go to the heirs of rich people or to risk-taking entrepreneurs who grew up in rich or upper-class homes or who grew up middle class but got lots of help from their well-educated professional parents when they went off to college and graduate school.

But that wouldn’t have sounded as good. It would’ve been like admitting that the surf-and-turf at Red Lobster actually cost more than $20. As Sasha Issenberg reported back in 2006:

I went through some of the other instances where he [Brooks] made declarations that appeared insupportable. He accused me of being “too pedantic,” of “taking all of this too literally,” of “taking a joke and distorting it.” “That’s totally unethical,” he said.

This time, let me make it clear that I’m not saying that Brooks did any false reporting. He just made a prediction in 1997 that was way way off. I do think Brooks showed poor statistical or sociological judgment, though. To think that “the real wealth” will go to the children of the “middle- or lower-middle-class” who don’t even pay for their college education . . . that’s just naiveté or wishful thinking at best or political propaganda at worst.

Brooks follows up his claim with this bizarre (to me) bit of opinionizing:

The people on the New York Times wedding page won’t make $4 million a year like the guy who started a chain of erotic car washes. They’ll have to make do with, say, $1.2 million if they make partner of their law firms. Maybe even less. The cognitive elite have more status but less money than the millionaire entrepreneurs, and their choices as consumers reflect their unceasing desire to demonstrate their social superiority to people richer than themselves.

I honestly can’t figure out what he’s getting at here except that I think it’s a bit of “mood affiliation,” as Tyler Cowen might say. According to Brooks’s ideology (which he seems to have borrowed from Tom Wolfe), “the guy who started a chain of erotic car washes” is a good guy, and “the cognitive elite” are bad guys. One way you can see this is that the erotic car wash guy is delightfully unpretentious (he might, for example have season tickets to the local football team and probably has a really big house and and a bunch of cars and boats, and he probably eats a lot of fat steaks too), while the cognitive elite have an “unceasing desire to demonstrate their social superiority.” They’re probably Jewish, too, just like that unfortunate urologist from the first paragraph of Brooks’s article.

But the thing that puzzles me is . . . isn’t 1.2 million a year enough? I mean, sure, if this car wash guy really wants more more more, then he can go for it, why not. But it seems a bit rich to characterize a bigshot lawyer as being some sort of envious hater because he was satisfied to max out at only a million a year. I mean, that’s just sad. Really sad, if there are people out there who think they’re failures unless they make 4 million dollars a year. There just aren’t that many slots in the world for people like that. If you have that attitude, you’re doomed to failure, statistically speaking.

Why bother?

The question always comes up when I write about these political journalists: why spend the time? Wouldn’t the world be better off if I were to put the equivalent effort into Stan, or EP, or Waic, or APC, or MRP, or various other statistical ideas that can really help people out?

Even if you agree with me that David Brooks is misguided, does it really help for me to dredge up a 17-year-old column? Better perhaps to let these things just sit, forgotten for another 17 years, perhaps.

My short and lazy answer is that I blog in part to let off steam. Better for me to just express my annoyance (even if, as in this case, it took me an hour to look up all those Wiki pages and write the post) than have it fester in my mind, distracting me from more important tasks.

My longer answer is: Red State Blue State. I do think that statistical misunderstandings can lead to political confusion. After all, if you really think that a good ticket for massive wealth is having lower-middle-class parents who won’t pay for college . . . well, that has some potential policy implications. But if you go with the facts and look at who the richest Americans really are and where they came from, that’s a different story.

Also, more generally, I wish people would revisit their pasts and correct their mistakes. I did it with lasso and I wish Brooks would do it here. What a great topic for his next NYT column: he could revisit this old article of his and explain where he went wrong, and how this could be a great learning experience. A lesson in humility, as it were.

I’ll make a deal with David Brooks: if you devote a column to this, I’ll devote a column to my false theorem—the paper my colleague and I published in 1993 that we had to retract because our so-called theorem was just wrong. I mean wrong wrong wrong, as in someone sent us a counterexample.

But I doubt Brooks will take me up on his offer, as I don’t think he ever ran a column on his mistake regarding the prices at Red Lobster, nor did he ever retract the “potentially ground-shifting” but false claims he publicized awhile ago in his column.

So, even though I would think it would be excellent form, and in Brooks’s best interests, to correct his past errors, he doesn’t seem to think so himself. I find myself in the position of Albert Brooks in that famous scene in Lost in America in which he tries in vain to persuade the casino manager to give back all the money his wife just gambled away: “As the boldest experiment in advertising history, you give us our money back.”

Am I too negative?

For background, you can start by reading my recent article, Is It Possible to Be an Ethicist Without Being Mean to People? and then a blog post, Quality over Quantity, by John Cook, who writes:

At one point [Ed] Tufte spoke more generally and more personally about pursuing quality over quantity. He said most papers are not worth reading and that he learned early on to concentrate on the great papers, maybe one in 500, that are worth reading and rereading rather than trying to “keep up with the literature.” He also explained how over time he has concentrated more on showcasing excellent work than on criticizing bad work. You can see this in the progression from his first book to his latest. (Criticizing bad work is important too, but you’ll have to read his early books to find more of that. He won’t spend as much time talking about it in his course.) That reminded me of Jesse Robbins’ line: “Don’t fight stupid. You are better than that. Make more awesome.”

This made me stop and think, given how much time I spend criticizing things. Indeed, like Tufte I’ve spent a lot of time criticizing chartjunk! I do think, though, that I and others have learned a lot from my criticisms. There’s some way in which good examples, as well as bad examples, can be helpful in developing and understanding general principles.

For example, consider graphics. As a former physics major, I’ve always used graphs as a matter of course (originally using pencil on graph paper and then moving to computers), and eventually I published several papers on graphics that had constructive, positive messages:

Let’s practice what we preach: turning tables into graphs (with Cristian Pasarica and Rahul Dodhia)

A Bayesian formulation of exploratory data analysis and goodness-of-fit testing

Exploratory data analysis for complex models

as well as many many applied papers in which graphical analysis was central to the process of scientific discovery (in particular, see this paper (with Gary King) on why preelection polls are so variable and this paper (with Gary King) on the effects of redistricting.

The next phase of my writing on graphics accentuated the negative, with a series of blog posts over several years criticizing various published graphs. I do think this criticism was generally constructive (a typical post might point to a recent research article and make some suggestions of how to display the data or inferences more clearly) but it certainly had a negative feel—to the extent that complete strangers started sending me bad graphs to mock on the blog.

This phase peaked with a post of mine from 2009 (with followup here), slamming some popular infographics. These and subsequent posts sparked lots of discussion, and I was motivated to work with Antony Unwin and write the article that eventually became Infovis and statistical graphics: Different goals, different looks and was published with discussion in the Journal of Computational and Graphical Statistics. Between the initial post and the final appearance of the paper, my thinking changed, and I became much more clear on the idea that graphical displays have different sorts of goals. And I don’t think I could’ve got there without starting with criticism.

(Here’s a blog post from 2011 where I explain where I’m coming from on the graphics criticism. See also here for a slightly broader discussion of the difficulties of communication across different research perspectives.)

A similar pattern seems to be occurring in my recent series of criticisms of “Psychological Science”-style research papers. In this case, I’m part of an informal “club” of critics (Simonsohn, Francis, Ioannidis, Nosek, etc etc), but, again, it seems that criticism of bad work can be a helpful way of moving forward and thinking harder about how to do good work.

It’s funny, though. In my blog and in my talks, I talk about stuff I like and stuff I don’t like. But in my books, just about all my examples are positive. We have very few negative examples, really none at all that I can think of (except for some of the examples in the “lying with statistics” chapter in the Teaching Statistics book). This suggests that I’m doing something different in my books than in my blogs and lectures.

Association for Psychological Science announces a new journal


The Association for Psychological Science, the leading organization of research psychologists, announced a long-awaited new journal, Speculations on Psychological Science. From the official APS press release:

Speculations on Psychological Science, the flagship journal of the Association for Psychological Science, will publish cutting-edge research articles, short reports, and research reports spanning the entire spectrum of the science of psychology. We anticipate that Speculations on Psychological Science will be the highest ranked empirical journal in psychology. We recognize that many of the most noteworthy published claims in psychology and related fields are not well supported by data, hence the need for a journal for the publication of such exciting speculations without misleading claims of certainty.

- Sigmund Watson, Prof. (Ret.) Miskatonic University, and editor-in-chief, Speculations on Psychological Science

I applaud this development. Indeed, I’ve been talking about such a new journal for awhile now.

The most-cited statistics papers ever

Robert Grant has a list. I’ll just give the ones with more than 10,000 Google Scholar cites:

Cox (1972) Regression and life tables: 35,512 citations.

Dempster, Laird, Rubin (1977) Maximum likelihood from incomplete data via the EM algorithm: 34,988

Bland & Altman (1986) Statistical methods for assessing agreement between two methods of clinical measurement: 27,181

Geman & Geman (1984) Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images: 15,106

We can find some more via searching Google scholar for familiar names and topics; thus:

Metropolis et al. (1953) Equation of state calculations by fast computing machines: 26,000

Benjamini and Hochberg (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing: 21,000

White (1980) A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity: 18,000

Heckman (1977) Sample selection bias as a specification error: 17,000

Dickey and Fuller (1979) Distribution of the estimators for autoregressive time series with a unit root: 14,000

Cortes and Vapnik (1995) Support-vector networks: 13,000

Akaike (1973) Information theory and an extension of the maximum likelihood principle: 13,000

Liang and Zeger (1986) Longitudinal data analysis using generalized linear models: 11,000

Breiman (2001) Random forests: 11,000

Breiman (1996) Bagging predictors: 11,000

Newey and West (1986) A simple, positive semi-definite, heteroskedasticity and autocorrelationconsistent covariance matrix: 11,000

Rosenbaum and Rubin (2004) The central role of the propensity score in observational studies for causal effects: 10,000

Granger (1969) Investigating causal relations by econometric models and cross-spectral methods: 10,000

Hausman (1978) Specification tests in econometrics: 10,000

And, the two winners, I’m sorry to say:

Baron and Kenny (1986) The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations: 42,000

Zadeh (1965) Fuzzy sets: 45,000


But I’m guessing there are some biggies I’m missing. I say this because Grant’s original list included one paper, by Bland and Altman, with over 27,000 cites, that I’d never heard of!

P.S. I agree with Grant that using Google Scholar favors newer papers. For example, Cooley and Tukey (1965), “An algorithm for the machine calculation of complex Fourier series,” does not make the list, amazingly enough, with only 9300 cites. And the hugely influential book by Snedecor and Cochran has very few cites, I guess cos nobody cites it anymore. And, of course, the most influential researchers such as Laplace, Gauss, Fisher, Neyman, Pearson, etc., don’t make the cut. If Pearson got a cite for every chi-squared test, Neyman for every rejection region, Fisher for every maximum-likelihood estimate, etc., their citations would run into the mid to high zillions each.

P.P.S. I wrote this post a few months ago so all the citations have gone up. For example, the fuzzy sets paper is now listed at 49,000, and Zadeh has a second paper, “Outline of a new approach to the analysis of complex systems and decision processes,” with 16,000 cites. He puts us all to shame. On the upside, Efron’s 1979 paper, “Bootstrap methods: another look at the jackknife,” has just pulled itself over the 10,000 cites mark. That’s good. Also, I just checked and Tibshirani’s paper on lasso is at 9873, so in the not too distant future it will make the list too.

On deck this week

Mon: The most-cited statistics papers ever

Tues: American Psychological Society announces a new journal

Wed: Am I too negative?

Thurs: As the boldest experiment in journalism history, you admit you made a mistake

Fri: The Notorious N.H.S.T. presents: Mo P-values Mo Problems

Sat: Bizarre academic spam

Sun: An old discussion of food deserts