Skip to content
 

“Is machine learning a subset of statistics?”

Following up on our previous post, Andrew Wilson writes:

I agree we are in a really exciting time for statistics and machine learning. There has been a lot of talk lately comparing machine learning with statistics. I am curious whether you think there are many fundamental differences between the fields, or just superficial differences — different popular approximate inference methods, slightly different popular application areas, etc. Is machine learning a subset of statistics?

In the paper we discuss how we think machine learning is fundamentally about pattern discovery, and ultimately, fully automating the learning and decision making process. In other words, whatever a human does when he or she uses tools to analyze data, can be written down algorithmically and automated on a computer. I am not sure if the ambitions are similar in statistics — and I don’t have any conventional statistics background, which makes it harder to tell. I think it’s an interesting discussion.

My reply:

I don’t know enough about machine learning to know what differences there are between the fields. One of my sayings is that theoretical statistics is another name for the theory of applied statistics. That is, statistics is all about modeling what we do, and modeling what we should be doing. As always in the social sciences, normative modeling has a descriptive flavor and descriptive modeling has a normative flavor: to the extent that we’re not doing what we say we should be doing, this suggests potential changes in our theory or in our practice. And much of my work over the years has been to give theoretical foundations for various areas of statistical practice that have typically been treated informally.

Thus, compared to other academic statisticians, I think I spend more time monitoring convergence of my iterative simulations, checking the fit of my models, and graphing data and fitted curves—but at the same time I do these things more formally than many statisticians have been trained to do. I think that some of the research we’ve been discussing lately on automatic model construction (done by people other than me, let me emphasize!) is important in that is moving toward a better description—and thus also a better normative theory—of model building. To me, it’s a big step forward from that thing where “learning a model” is associated with taking a big multivariate dataset and trying to identify conditional independence structures. To me, all that stuff is static, and I’m much happier with a framework in which models are built out recursively in a language-like fashion.

That said, for now this is all a sideshow. We still have a ways to go in fitting models that we’ve already specified. Hence, Stan.

Are we at the stage of “fully automating the learning and decision making process”? I don’t think so. But the only way forward is to try, not getting too stuck in our current understanding at any time.

4 Comments

  1. Thanks for your reply. I suppose the more formal we are about what we do as “data analysts”, the more we are able to naturally automate learning and decision making. I agree we as a community are not at the stage of fully automating learning and decision making, and I too am excited for the future.

  2. Erin Jonaitis says:

    Truth be told, as someone who makes a living crunching numbers, I find this idea a bit discomfiting! I mean, I figured the robots might come for my job someday, but I didn’t think it would be anytime soon…

    It’s still an interesting problem, of course. In my former life I was a graduate student in cognitive psychology, studying grammar learning. This bit of psychology has loose ties to the machine learning world, with the difference that cognitive scientists work under the constraint of trying to model a particular kind of learner. Also, the exactitude that programming requires is great for deepening your own understanding of what you’re doing, so even if nobody is successful in replacing me with a Statistics Roomba, it is likely to be a useful exercise that could teach me a thing or two about my job!

  3. Machine learning researchers think about problems in terms of algorithms; statisticians think about problems in terms of models.

    Of course, there are many people (like myself) that fall somewhere in between these two poles.

  4. MD says:

    I’m wondering, where (and how) does statistical learning fit in the big picture?

    http://www-stat.stanford.edu/~tibs/ElemStatLearn/
    http://en.wikipedia.org/wiki/Statistical_learning_theory