Jeff Heer and Mike Bostock provided Mechanical Turk workers with a problem they had to answer using different types of charts. The lower error the workers got, the better the visualization. Here are some results from their paper Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design:
They also looked at various settings, like density, aspect ratio, spacing, etc.
Visualization has become empirical science, no longer just art.
Visualisation has been an empirical science for at least 25 years!
1. I echo Hadley on this one. See Cleveland's classic book from 1985.
2. I hate hate hate those little vertical segments at the end of the error bars. They draw attention to the endpoints of the intervals, which are not particularly informative.
They also force you to make the central point a big fuzzy blob to draw attention back to the center of the interval. If the ends weren't vertical segments, the center could be a vertical segment.
I assume the practice started as a sort of serif, to cope with bad printing, copying, and faxing, which could wash out the ends.
There were papers on how bar and pie charts play with users in the Journal of the American Statistical Association in the 1920s, as far as I can recall. Cleveland and colleagues did not overlook that literature.
The question of what works with graph consumers is good and important, but I note puzzles on different levels:
1. I've fed a variety of displays informally to moderately numerate students and have usually found that they invert Cleveland-Tufte order in preferring pie charts to bar charts and evincing no enthusiasm for dot charts, even after a preparatory "brainwashing" lecture explaining the logic and illogic of various designs and being scathing about chartjunk. The usual tone is of the order of "I hear what you say, but pie charts are very familiar to me and I feel comfortable with them".
2. I suspect that many of us resist studies of popular attitudes if the results clash with our principles. The recent thread on chartjunk is a case in point. I agree with commenters that the study in question looks poor but (a) it got a best paper award (b) what would make us change our minds? (c) are we condemned to go round the same debates perpetually?
3. What makes basic graphics fair game for studies of popular opinion but not, it seems, other areas of statistical science? If a study reported that users of statistics greatly preferred means to medians, would that change your mind on their relative merits? Are there studies of whether non-specialists who just read the journal literature prefer certain kinds of models to others? My guess is that most people who develop such models or routinely fit them would regard that as quite secondary, but where does the difference lie?
Hadley and Andrew: The study does list Cleveland's results, and I've included their chart comparing their own results with his. Blame me for short provocative sentences, not them.
Nick – it would actually be interesting to check people's accuracy of judgments using models communicated to them by means vs medians. An example of a crowd-sourced way to test for performance, see Yes, You Are (Maybe) Overconfident on the Messy Matters blog.
Nick:
I don't think we're going around the same debates. The researchers that I disagreed with in that other blog entry did something new, then Kaiser Fung and I (separately) looked at the details and weren't so impressed. In particular, the comparison is all about how to display 5 data points. But Tufte-ism isn't about how best to display 5 data points, it's about using graphical techniques to display lots of information in a useful way. Finally, good for them for getting a best paper award, but that doesn't really affect my evaluation of what they're doing.