Skip to content

The past, present, and future of statistics

This is going to be a letdown after this grand title . . . . Lingzhou Xue writes,

I just read recently the talk titled “The Future of Statistics” from Bradley Efron. Actually, I see some enlightening ideas but also fall a little puzzled. In this talk, Efron first gave a simple review of the rapid development of statistics last century. He is humorous to comment that “The history of statistics in the Twentieth Century is the surprising and wonderful story of a ragtag collection of numerical methods coalescing into a central vehicle for scientific discovery”.

After this humor is just what puzzles me and what I really hope your instructions and ideas. Efron cited a simple example to illustrate the limitations of classical statistics in the model selection problems and also exploit a figurative comment that “History seems to be repeating itself: we’ve returned to an era of ragtag heuristics propelled with energy but with no guiding direction.”

Finally, he presented an helpful instructions that “During such time it pays to concentrate on basics and not tie oneself too closely to any one technology.”

My reply: Efron is an interesting example of a leading statistical researcher who has developed and used a diverse set of tools, most notably model-based empirical Bayes and nonparametric boostrap and permutation tests. So he, more than most, is justified in seeing statistics as being extremely successful without needing a guiding direction. In the hedgehog/fox distinction, he’s a fox.

It’s hard for me to make generalizations about the field of statistics since there are so many different strands. I guess some sort of analysis based on papers and citation counts would give a clue. I guess it is true that statistics in the 1950s, like politics in the 1950s, had a unity that we didn’t see before and don’t see today. 1950s-style statistics was limited but it was all people had and so they used it well. It broke down when it got overwhelmed with data.

One Comment

  1. ZBicyclist says:

    This sort of agreeing on a model, then a period of disorder, then agreeing on a different model — isn't that what knowledge expansion is all about?

    The development of a new model that explains a bunch of old data (earth moving around the sun, not vice versa; Newton's model; Darwin and Wallace on evolution) seems to bring great clarity for a time, and it does.

    But providing a place to stand (on the shoulders of giants) enables us to see farther, which enables us to see areas of ignorance on the fringes we couldn't see before.

    The statement "1950s-style statistics … broke down when it got overwhelmed with data" is true enough, but it also broke down as we tried to do more things with existing data. In particular, as we tried to extend the usage of statistical methods farther into areas which aren't easily subjected to experimental control.