Robin Gong writes:
While we’re on the topic of visualization, I’ve been puzzled by a more general question and I’m unsure where it fits in actually.
There seem to be two parts to a good visualization practice, and in our class we’ve been focusing more on one of them, that is “how to get my point across?” To me that’s a psychology question for which a recipe-type solution could exist, e.g. what choices of graph types, layouts and details are better at alleviating the cognitive burden in the audience, making the visual cue more salient thus a more effective graph. But this “how” question begins with the premise that I know “what point is to be made”, and what if I don’t? When doing EDA in high-dimensional data, how does one visualize the potentially multi-way joint dependency among variables? Or when checking the performance of a high-dim model, can I see anything (estimation, model fit, prediction) beyond uni-/bi-variately? And to throw in something inspired by immediate research, suppose one wishes to compare the posterior inference between MCMC and approximate inference methods (e.g. VB or EP), what can be done to display the loss of authenticity of the approximation, beyond the dry metrics such as KL divergence or marginal likelihood? I call them dry since they aren’t revealing of the specific problem, such what areas of the support or which dimensions are missing out the most. The challenge is that we’re exactly relying on visualization to teach ourselves about the behavior of this object (data or model) that would otherwise been impossible to learn, but if we are ignorant of the object, how to visualize it to begin with? To this end I think this is perhaps fundamentally a statistical methodology question, whose solution calls for a different kind of ingenuity; after all if I knew how to display a higher dimensional object in lower dimension with all essence captured, I would’ve found a good inferential method. People say that if an MCMC sampler doesn’t converge, chances are the model is problematic to begin with, and I wonder if it’s the same with statistical visualization: if I can’t display it properly, am I just asking the wrong questions?
My reply: There are a few interesting ideas here:
First, we do spend lots of time on the details of how to graph a particular idea or pattern of information, but not so much time on what to graph.
Second, there’s the challenge of trying to discover the unexpected in high dimensions. It’s my impression that there was a lot of research on this in the 1970s and 1980s: The statistics world was pretty small back then, and after Tukey started writing about exploratory data analysis, various people started working on ideas such as rotating point clouds, or automatically searching for interesting dimensions for graphical comparisons. My guess is that a lot of this work is still valuable and worth looking into further. The idea seems very powerful, to treat the human and the computer as a pattern-recognition team, and to have an algorithm to find projections that are worth further exploration.
Third, how to you compare in high dimensions, for example if you have multiple chains of HMC and you want to check that they’re pretty much in the same place? Steve Brooks and I did some work on this awhile ago with the multivariate potential scale reduction factor but it didn’t work so well in practice, perhaps because our goals weren’t so clear.