My article with the above title is appearing in the journal Teaching Statistics. Here’s the introduction:
We can improve our teaching of statistical examples from books by collecting further data, reading cited articles and performing further data analysis. This should not come as a surprise, but what might be new is the realization of how close to the surface these research opportunities are: even influential and celebrated books can have examples where more can be learned with a small amount of additional effort.
We discuss three examples that have arisen in our own teaching: an introductory textbook that motivated us to think more carefully about categorical and continuous variables; a book for the lay reader that misreported a study of menstruation and accidents; and a monograph on the foundations of probability that over interpreted statistically insignificant fluctuations in sex ratios.
And here’s the conclusion:
Individually, these examples are of little importance. After all, one does not go to a statistics textbook to learn about handedness, menstruation, and sex ratios. It is striking, however, that the very first examples I looked at in the Zeisel and von Mises books – the examples with interesting data patterns – collapsed upon further inspection. In the Zeisel example, we went to the secondary source and found that his sketch was not actually a graph of any data, and that he in fact misinterpreted the results of the study. In the von Mises example, we reanalysed the data and found his result to be not statistically significant, thus casting doubt on his already doubtful story about ethnic differences in sex ratios. In the Utts and Heckard example, we were inspired to collect data on handedness and look at survey questions on religious attendance to find underlying continuous structures.
You can do it yourself!
These are examples that I’ve encountered during the past twenty years of teaching. The real message I want to send, though, is that you can do it yourself. Anything you read, you can check, for example this implausible (and, indeed, false) claim by a public health expert that “Consumption [of chicken] in the US has increased . . . a hundredfold between 1934 and 1994.” (It actually increased by a factor of six.)
Textbooks are commonly written in an authoritative style, but that doesn’t mean everything in them is correct. You can learn a lot by going back to the original source of the data, and even running the occasional chi-squared test of your own!