The Difficulty with Difficult Questions
Andrew’s commented during our Stan meetings that he’s observed that when a user sends an easy question to a mailing list, it gets answered right away, whereas difficult questions often languish with no answers.
These difficult questions usually come from power users with real issues, whereas the simple questions are often ill-formulated or already answered in the top-level doc. So we’re arguably devoting our energy to the wrong users by adopting this strategy.
Of course, this is related to Andrew’s suggestion that this whole blog be called “tl;dr” (i.e., too long, didn’t read).
On the Stan Users Group, we often get very complex models with a simple accompanying question such as “How can I make my model faster or mix better?” An example is this recent query, which involves a difficult multivariate multilevel model. Such questions require a lot of work on our part to answer. Model fitting is hard and often very problem specific. And it varies by platform — the tweaks you need to do for BUGS/JAGS are Gibbs sampling specific whereas those required for Stan are HMC specific.
Mitigating the Problem
The degree-of-difficulty difficulty can be mitigated somewhat by breaking questions down into simpler, digestible bits. Everyone likes a short question they can understand and answer. Everyone feels good after the exchange.
But breaking a problem down is often impossible for a user —- if a user could isolate the problem, they could probably solve it. I often find myself struggling to express a problem to mailing lists such as the Boost Spirit or Rcpp list in terms other than “I tried this and didn’t work.”
Rubber Ducky to the Rescule?
Sometimes the mere act of breaking a question down into digestible bits leads me to the answer and I can spare the mailing list. This is closely related to the “rubber ducky” phenomenon in debugging, namely that the mere act of explaining a problem clearly often leads to a solution.
Maybe Andrew can come up with a better name for this phenomenon than “degree-of-diffuclty difficulty” and drop it into his Handy Statistical Lexicon for posterity.