Pascal Biber writes:
I am a science journalist for Swiss public television and have previously regularly covered the “crisis in science” on Swiss public radio, including things like p-hacking, relative risks, confidence intervals, reproducibility etc.
I have been giving courses in basic statistics and how to read scientific studies for Swiss journalists without science backgrounds. As such, I am increasingly wondering what to teach (knowing they will not dive into Bayesian statistics). How should a non-science journalist handle p values? Should he at all? What about 95 percent confidence intervals? Should relative risks be reported when absolute numbers aren’t available? What about small studies? Should they even report about single studies? And how about meta-analysis?
I wondered if you have some basic advise for journalists that you could share with me? Or are you aware of good existing checklists?
To start with, here’s my article from 2013, “Science Journalism and the Art of Expressing Uncertainty.”
Also relevant is this post on Taking Data Journalism Seriously. And this on what to avoid.
There are some teaching materials out there, which you can find by googling *statistics for journalists* or similar terms, but there is a problem that once you try to get in deeper, there’s little agreement in how to proceed.
I suppose that the starting point is understanding the statistical terms that often arise in science: sample and population, treatment and control, randomized experiment, observational study, regression, probability.
Should you teach p-values and 95% intervals? I’m not sure. OK, yeah, you have to say something about these, as they appear in so many published papers. And then you have to explain how these terms are defined. And then you have to explain the problems with these ideas. I think there’s no way around it.
Bayesian methods? Sure, you have to say something about these, because, again, they’re used in published papers. You don’t have to “dive” into Bayesian methods but you can and should explain the idea of predictive probabilities such as arise in election forecasts.
For the big picture, I recommend this bit from the first page of Regression and Other Stories:
I appreciate that 3rd challenge especially.
Exactly once we get deeper the slog gets deeper.
First, you need to understand two XKCD comics. Once you’ve done that you’re ahead of the curve.
Significant
https://xkcd.com/882/
Frequentists vs. Bayesians
https://xkcd.com/1132/
Man this is like riding bareback. LOL
David:
On that second xkvd with the vulgar frequentist falling victim to the base-rate Fallacy: https://www.researchgate.net/profile/Aris_Spanos/publication/255669979_Is_Frequentist_Testing_Vulnerable_to_the_Base-Rate_Fallacy/links/544664f40cf22b3c14de2022.pdf
I don’t personally buy this argument, p-thresholding for publication seems to nenexavtly this.
I assume “nenexavtly” is a typo — but I haven’t been able to figure out what was intended.
Sorry, “to be exactly this”
They both wanted to test for the technical “has the sun gone nova” but actually draw conclusions about the different (yet similar sounding) “has the sun exploded” question, something that happens all the time: https://www.youtube.com/watch?v=8i1pi0O_iNk
I think that is the bigger (most likely unintentional) lesson here.
Also, the detector in the end is measuring neutrino flux, not sun explosions/novas. It is all quite convoluted since that would actually be closer to measuring sun explosions rather than novas… Just the usual NHST-induced mass confusion I guess.
But also see http://www.explainxkcd.com/wiki/index.php/1132
Yeah but he doesn’t adress whether this is analogous to frequentist. Certainly it is analogous to NHST!
Also, he neglects to mention that the odds in favor of nova are 35 times higher than they were initially.
David:
I had some problems with that second cartoon you cite; see discussion here (including a response from Randall Munroe, the cartoonist).
Your explication is also interesting. But it’s a cartoon. It wouldn’t be funny if it were written with every clause necessary so that so no one could have a problem with it. (Other than as a parody of scientific writing, or like the German in a Patton Oswalt skit.)
I’d start with Gerd Gigerenzer’s book on Risk. Any journalist who absorbs its teachings can at least be confident of being more expert in stats than the average doctor. Or the very-much-above average doctor, probably.
I’d also consider using some anecdotes from Malcolm Kendrick’s Doctoring Data.
Gigerenzer has written some good stuff, but also some not-so-well-thought-out stuff. See http://www.ma.utexas.edu/blogs/mks/2015/02/03/another-mixed-bag-gigerenzers-mindless-statistics/