Can’t Stop Won’t Stop Mister P Beatdown

Ben Highton and Matt Buttice point us to this response addressing some of the issues Jeff Lax raised in his most recent MRP post.

P.S. Jeff replies in comments:

It sounds like we’ve converged. They acknowledge MRP performance is significantly better on average than reported in their new paper in PA and yet performance variation in terms of correlation to “truth” remains higher than some might have thought.

Cool. I hope this sort of blog exchange can be a model of scientific discussion. Instead of a paper just sitting there by itself, it can be openly explored. Ideally, the published paper would include a link to these discussions of Highton, Buttice, Lax, Phillips, and Ghitza, so that readers would automatically get all this information.

3 thoughts on “Can’t Stop Won’t Stop Mister P Beatdown

  1. It sounds like we’ve converged. They acknowledge MRP performance is significantly better on average than reported in their new paper in PA and yet performance variation in terms of correlation to “truth” remains higher than some might have thought. I appreciate their attention to these issues. They think the variation too high to feel fully comfortable with MRP. I do not, given that (a) mean absolute errors (the other metric for success) are still often small and (b) in practice one can inspect the estimates and model and bring additional information to the table. I’m working on benchmarks and diagnostics for such variation. -Jeff

Comments are closed.