There’s been a lot of discussion, here and elsewhere, of the problems with null hypothesis significance testing, p-values, deterministic decisions, type 1 error rates, and all the rest. And I’ve recommended that people switch to a Bayesian approach, “embracing variation and accepting uncertainty,” as demonstrated (I hope) in my published applied work.
But we recently had a blog discussion that made me realize there was some confusion on this point.
Emmanuel Charpentier wrote:
It seems that, if we want, following the conclusion of Andrew’s paper, to abandon binary conclusions, we are bound to give :
* a discussion of possible models of the data at hand (including prior probabilities and priors for their parameters),
* a posterior distribution of parameters of the relevant model(s), and
* a discussion of the posterior probabilities of these models
as the sole logically defensible result of a statistical analysis.
It seems also that there is no way to take a decision (pursue or not a given line of research, embark or not in a given planned action, etc…) short of a real decision analysis.
We have hard times before us selling *that* to our “clients” : after > 60 years hard-selling them the NHST theory, we have to tell them that this particular theory was (more or less) snake-oil aimed at *avoiding* decision analysis…
We also have hard work to do in order to learn how to build the necessary discussions, that can hardly avoid involving specialists of the subject matter : I can easily imagine myself discussing a clinical subject ; possibly a biological one ; I won’t touch an economic or political problem with a ten-foot pole…
Wow—that sounds like a lot of work! It might seem that a Bayesian approach is fine in theory but is too impractical for real work.
But I don’t think so. Here’s my reply to Charpentier:
I think you’re making things sound a bit too hard: I’ve worked on dozens of problems in social science and public health, and the statistical analysis that I’ve done doesn’t look so different from classical analyses. The main difference is that I don’t set up the problem in terms of discrete “hypotheses”; instead, I just model things directly.
And Stephen Martin followed up with more thoughts.
In my experience, a Bayesian approach is typically less effort and easier to explain, compared to a classical approach which involves all these weird hypotheses.
It’s harder to do the wrong thing right than to do the right thing right.