This post is by Keith.
(Image from deviantart.com)
There are a couple posts I have been struggling to put together, one is on what science is or should be (drawing on Charles Peirce). The other is on why a posterior is not a posterior is not a posterior: even if mathematically equivalent – they are (or could be) all very different (drawing on Susan Haack). However, Sander Greenland just sent me one of his new papers “For and Against Methodologies” which amongst other things brings out the motivations I have for trying to do those posts. It is well worth reading and I’ll try here to briefly provide some motivation for reading the paper primarily using excerpts.
Sander’s title pays homage to Against Method by Paul Feyerabend and my post title pays homage to Sigmund Freud commenting that a scientific psychology was not likely in his life time but he had no intention of changing careers (from memory, unfortunately without a reference). By applying statistics in science, I mean enabling researchers to be less misled by the observations they encounter and also to have a good sense of how much they unavoidably may have been misled. Realistically the observations they encounter will not be always randomized nor even practically _as if randomized_ and researchers will almost always want to be targeting something about reality. Descriptively when just targeting one reality, causally when there are two or more realities being compared and transportably when more than one reality is being targeted with something taken as being common. Now, the being unreasonably difficult arises from having to have much more that adequate knowledge of statistical methods and modelling but also a rather advanced grasp of what science is or should be as well as what to make of the output of statistical modelling (e.g. posteriors) beyond the mathematical summaries of the modelling. Or so I think Sander argues.
One definition of scientific enquiry is just everyday enquiry with helps. Statistical methods and inference approaches are then seen as one of the helps, often a critically important one but not a method of enquiry. My focus in this blog post will primarily be on Sander’s take on “the overconfidence that plague[s] statistical methodology” specifically when “formal statistical inferences are treated as sound judgments about causality” when they are instead just “an aid to inference” being “a useful technology emerging from formal methodology, rather than a theory” of enquiry or inference. More generally, “Overconfident inferences … touted as unconditionally sound scientific inferences instead of the tentative suggestions that they are.” Though Sander focuses primarily on targeting something about reality causally, I believe the arguments are important descriptively even if somewhat less critical (there is always a price to be paid for getting reality wrong at some point in time).
He further clarifies his arguments with “Every methodology has its limits, so do not fall into the trap of believing that a given methodology will be necessary or appropriate for every application; conversely, do not reject out of hand any methodology because it is flawed or limited, for a methodology may perform adequately for some purposes despite its flaws. … [this] raises the problem of how to choose from among our ever-expanding methodologic toolkit, how to synthesize the methods and viewpoints we do choose, and how to get beyond automated methods and authoritative judgments in our final synthesis.” The later being a much under discussed topic in the literature.
“Much beneﬁt can accrue from thinking a problem through within these models, as long as the formal logic is recognized as an allegory for a largely unknown reality. A tragedy of statistical theory is that it pretends as if mathematical solutions are not only sufﬁcient but ‘‘optimal’’ for dealing with analysis problems when the claimed optimality is itself deduced from dubious assumptions. … in the end an exercise in hypothetical reasoning to aid our actual inference process, and should not be identiﬁed with the process itself … Analysis interpretation depends on contextual judgments about how reality is to be mapped onto the model, and how the formal analysis results are to be mapped back into reality.”
Here the unreasonably difficult part is brought out as “the potential for greater realism in these approaches [which unfortunately he argues the sophistication demanded for proper use and review of – is lacking amongst many (most?) practising statisticians] suggests they should be part of the skill set for everyone who will direct or carry out statistical analyses of nonexperimental data, to help them gauge the distance between the model they are using and more realistic models, and to help them critically evaluate the sensitivity and bias analyses they encounter.” It perhaps needs to be added here that almost all data is not quite fully experimental as perfect experiments are just a threat – you can’t rule out someone doing one some day but it would be naive to assume someone had.
Now part of the problem can be blamed on the content in many (most, all?) statistical programs “degrees in statistics and medicine do not require substantial training in or understanding of scientiﬁc research or reasoning, but nonetheless serve as credentials licensing expressions of scientiﬁc certainty” but one does wonder how and who would be able to address this? A possible example of this oversight being Karl Pearson’s Grammar of Science. It also underlines a serious challenge to be overcome for John Ioannidis’ idea “Teams which publish scientific literature need a ‘licence to analyse’ and this licence should be kept active through continuing methodological education.” : what is the training that will be needed for understanding scientiﬁc research or reasoning and who will provide it?
Now I am not quite sure on what is meant “I depart from the mainstream Bayesian revival in regarding simulation methods as insufﬁcient for Bayesian education and sensible application” but I certainly agree with ” because the complexity of actual context prohibits anything approaching complete modeling, the models actually used are never entirely coherent with that context, and formal analyses can only serve as thought experiments within informal guidelines.”
OK, what are some ways to make things less unreasonably difficult?
This post is by Keith.