If I made a separate post for each interesting blog discussion, we’d get overwhelmed. That’s why I often leave detailed responses in the comments section, even though I’m pretty sure that most readers don’t look in the comments at all.
Sometimes, though, I think it’s good to bring such discussions to light. Here’s a recent example.
Poor predictive performance usually indicates that the model isn’t sufficiently flexible to explain the data, and my understanding of the proper Bayesian strategy is to feed that back into your original model and try again until you achieve better performance.
It was my impression that — in ML at least — poor predictive performance is more often due to the model being too flexible and fitting noise.
And Rahul agreed:
Good point. A very flexible model will describe your training data perfectly and then go bonkers when unleashed on wild data.
But I wrote:
Overfitting comes from a model being flexible and unregularized. Making a model inflexible is a very crude form of regularization. Often we can do better.
This is consistent with Michael’s original comment and also with my favorite Radford Neal quote:
Sometimes a simple model will outperform a more complex model . . . Nevertheless, I believe that deliberately limiting the complexity of the model is not fruitful when the problem is evidently complex. Instead, if a simple model is found that outperforms some particular complex model, the appropriate response is to define a different complex model that captures whatever aspect of the problem led to the simple model performing well.
I’ll give Radford the last word for now (until anyone responds in the comments).