This plot is perhaps an interesting start to pinning down experts (extracting their views and their self assessed uncertainties) – contrasting and comparing them and then providing some kind off overall view.
Essentially get experts to express their best estimate and its uncertainty as an interval and then pool these intervals _weighting_ by a pre-test performance score on how good they are at being experts (getting correct answers to a bank of questions with known answers).
For those who are not familiar with consensus group work, a very good facilitator is needed so that experts actually share their knowledge instead of just personalities and stances.
The experts’ intervals could easily be plotted by their performance score, and weighting schemes of %correct, versus (%correct)^2, or ( (%correct)^2 or 0 if %correct< 50% ) considered. Better still - some data mining and clustering of experts' pre-test answers and best estimates. Note these could be viewed as univariate priors and points towards the much more challenging area of extracting, contrasting and combining multivariate priors. K
Keith: See Section 4 (in particular, Figure 3) of this article for a review of a famous example of this type.
Sounds like what the R package 'expert' does.
http://cran.r-project.org/web/packages/expert/ind…
The complete paper is available at:
http://www.rff.org/Documents/Features/294-295%20O…
Nice article, nice graphic, nice idea.
I'd like to point out that I tried to do more or less what is suggested in this article — get people to describe their personal probability distribution for an important parameter — in a post on climate change a few months ago. The post drew 36 comments from about 25 different commenters, but only 3 of them were willing to put a distribution on the parameter.
Andrew: Is the link missing?
Wayne: neat that R has a package and some work has been done on more interesting weighting for the pooling
Phil: recall some of those posts – but I don't think you brought them together for two days and had a good facilitor take them through a consensus building process.
One I was previously involved in had a neat trick – you offer a nice meal on the final day but pre-arrange with the chef and thier staff that the food won't be ready until the facilitator says it is – food for thought or more like – thoughful work or no food.
K
Keith: I added the link. It's fun going back and reading my old articles. 1998 was a good year.