Two people pointed me to an article by Emre Soyer and Robin Hogarth that was linked to by Felix Salmon.
Here are my reactions:
1. Soyer and Hogarth’s paper seems very strong to me, and Salmon’s presentation is an impressive condensation of it. I’d say good job on the science and the reporting.
2. I don’t see the point of focusing on economists. This seems just like a gimmick to me. But, then again, I’m not an economist. So of course I’d be more interested in a similar paper studying political scientists or statisticians. This should be easy enough for someone to do, of course.
3. To elaborate on this last point: I’m not surprised that people, even expert practitioners, screw up with statistics. Kahneman and Tversky found this with psychology researchers back in the 1970s. I’m not knocking the current paper by Soyer and Hogarth but I don’t see it as surprising. Perhaps the focus on economists is what allowed it to get all this attention. If you want people to read your newspaper article, write it about celebrities. If you want people to read your academic article, write it about economists?
4. Soyer and Hogarth’s paper is all about how difficult it is to understand statistical results presented as tables. I was disappointed (but, unfortunately, not surprised) to see them present many of their findings in tables rather than graphs, and the graphs they do use are uninspired—they’re not the worst graphs in the world, but they’re a bunch of poorly-organized bar charts. A collaboration with someone such as Kaiser Fung would’ve made the presentation a lot better (if they were to follow the minimal strategy of letting Kaiser improve, compress, and combine their graphs) or a lot lot lot lot better (if they were to think hard about presenting all their key results graphically). I would prefer to see all the numbers in Tables 1-3 and Appendix C presented graphically. As I’ve discussed on numerous occasions, such plots can table up less space as well as displaying relevant comparisons more effectively than the corresponding tables.
5. Going forward, I see the real question as how to better understand and communicate statistical results. To me the recommendation from the present paper is not so much to display regressions as graphs (although I agree with this advice) but rather to use the statistical model to answer any questions of interest (what we call qoi’s) directly. For example, if you want people to know the value of x for which Pr(y>0)=.95, just calculate it directly from the model. All the time I see talks where people present regressions and start interpreting the coefficients and making various indirect claims that could be answered from the model directly.
I think Soyer, Hogarth, and Salmon for giving us the opportunity to think about all this and to bang home again once more the message we’ve learned from so much of the work of Kahneman, Tversky, Gigerenzer, and others that uncertainty and variation are hard to understand, that people are overconfident about their guesses, and that how a problem is framed can make a difference in how people perform on a psychological task.