Christian Bartels has a new paper, “Efficient generic integration algorithm to determine confidence intervals and p-values for hypothesis testing,” of which he writes:
The paper proposes to do an analysis of observed data which may be characterized as doing a judicious Bayesian analysis of the data resulting in the determination of exact frequentist p-values and confidence intervals. The judicious Bayesian analysis comprises the steps which one would or should do anyway:
Bayesian sampling of parameters given the data, e.g., using Stan
Simulation of new data given the sampled parameters
Comparison of the simulations with actually observed data
Using frequentist concepts to do the comparison of simulations with observations, one obtains frequentist p-values and confidence intervals. The frequentist p-values and confidence intervals are exact in the limit of investing sufficient computational time. This holds true independent of the probability model used, and independent of whether the observed data consists of a few or many observations. As such the algorithm is a valid if not superior alternative to bootstrap sampling of frequentis parameter estimates.
In the evaluation of the proposed algorithm, it has also been investigated in how far Bayesian estimates may be used as a frequentist test procedure. It has been shown that this is feasible, simple and results are comparable to those obtained with likelihood-ratio tests.
I haven’t looked at it in detail but, as described above, the approach reminds me of my 1996 paper with Meng and Stern in that we were originally thinking our largest audience would be users of classical statistics who would appreciate a stable and general approach for getting reasonable p-values in the presence of nuisance parameters. As it happened, the Bayesian-ness of our method was pretty much toxic to outsiders, but our paper did have some influence within Bayesian statistics (even as my own attitudes have changed, so that I still think model checking is important but I’m less likely to be using p-values, Bayesian or otherwise, in my applied work).
Bartels also includes some R and Stan code! See here for all the files.