Steve Cohen writes:
As someone who has been working with Bayesian statistical models for the past several years, I [Cohen] have been challenged recently to describe the difference between Bayesian Networks (as implemented in BayesiaLab software) and modeling and inference using MCMC methods.
I hope you have the time to give me (or to write on your blog) and relatively simple explanation that an advanced layman could understand.
My reply:
I skimmed the above website but I couldn’t quite see what they do. My guess is that they use MCMC and also various parametric approximations such as variational Bayes. They also seem to have something set up for decision analysis.
My guess is that, compared to a general-purpose tool such as Stan, this Bayesia software is more accessible to non-academics in particular application areas (in this case, it looks like business marketing). But I can’t be sure.
I’ve also heard about another company that looks to be doing something similar: https://www.priorknowledge.com/
Stan and Bayesian do completely different things.
I see the word “Bayesian” used to describe three very different applications of Bayes Theorem. I’ve found it useful to identify them according to *who* is doing the learning.
* Bayesian Statistics: The researcher is doing the learning. She has data and she wants to update her knowledge about some quantities of interest. Stan, BUGS, MCMCpack, and other tools are used for this.
* Bayesian Networks: The computer is doing the learning. The researcher has a lot of data and wants the computer to be able to make decisions based on it (machine learning). Bayesia and the product by Prior Knowledge are used for this.
* Bayesian Games: The subjects of study are doing the learning. The researcher makes a behavioral assumption about the people she is studying that those people use Bayes Theorem to update *their* beliefs about the world.
Technically the first two categories overlap because statistical methods sometimes are used as components of machine leaning algorithms. Still, if the computer is going to make a prediction, I think of it as machine learning; if the researcher is making inferences, I think of it as statistics.
Hi Andrew! Thanks for the shout-out! At P(K), we’re doing nonparametric bayes to learn the joint distribution over arbitrary tables of data, so that people without as much statistics training as us can do amazing things with their data. Since the real world is messy, uncertain, heterogeneous, and sparse (all areas where Bayes does a great job), this lets us make prediction as easy as SQL.
This is related to our new postdoc (and my new officemate) Yajuan (Sophie) Si’s Ph.D. work, which used non-parametric Bayesian models for multiple imputation:
https://stat.duke.edu/sites/default/files/people/theses/thesis-YAJUANSI(ys54).pdf
Generic implementations of multiple imputation look a whole lot like modeling database tables. Sophie also did some work comparing full Bayes to various alternatives for multiple imputation.
It’s nice to see commercial MCMC and approximate bayes; I think that has been an area ripe for development for years. I expect that they’ll learn a lesson from stata and SAS: the leaders for commercial software have exquisitely thorough documentation for both how to use them and what they are doing. There’s certainly a market for black box machine learning, but the ability to have somebody at your company / lab know the strenghts and limitations of the tool is important.
I am also interested in hearing more about differences between Bayesian Nets and MCMC; In addition, apparently BNs, MRF, Kalman Filters, HMM can all be reduced to Factor Graphs, and its inventor Dr. Frey shows i.e. even HMM Viterbi algorithm is a special case of sum-product algorithm that is used on FGs.
Affinity Progation (also an application of FGs) works really well for unsupervised clustering, and since mixture models + MCMC can also be used there, I was wondering if there is a clear cut answer for using one or the other on a problem. Or in certain cases they are the same thing (?).
You’re confusing general representations of models (MRF, factor graph), particular models (HMM, Kalman filter), and estimation algorithms at various levels (MCMC, sum-product, forward-backward, Viterbi).
To run MCMC efficiently over structured models like HMMs, you need good ways to compute normalized probability functions, such as the forward-backward algorithm.
For some more insight about factor graphs vs. MRFs, see:
http://metaoptimize.com/qa/questions/810/why-are-markov-random-fields-usually-preferred-over-factor-graphs
You can also get some nice generalizations of structured models like HMMs using Bayesian non-parametrics. Frank Wood has done some great work in this area:
http://www.stat.columbia.edu/~fwood/
Frank just moved within Columbia from stats to CCLS, bridging the mental and administrative gap between machine learning and stats.