The discussants are all talking about the connection between adaptive data analysis and the garden of forking paths; for example, this from one commenter:
The idea of adaptive data analysis is that you alter your plan for analyzing the data as you learn more about it. . . . adaptive data analysis is typically how many researchers actually conduct their analyses, much to the dismay of statisticians. As such, if one could do this in a statistical valid manner, it would revolutionize statistical practice.
Just about every data analysis I’ve ever had is adaptive, and I do think most of what I do is “statistically valid,” so whassup with that? A clue is provided by my 2001 paper with Jennifer and Masanao, “Why we (usually) don’t have to worry about multiple comparisons.” If you fit a multilevel model (or a Bayesian model with informative prior distributions), then it’s perfectly “statistically valid” to look at many comparisons. The key is aim to do all the analyses you might do, avoiding selection bias by performing all relevant comparisons, and avoiding the problems with p-values by partially pooling all your comparisons rather than just reporting a selected subset.