Enrico Schaar points me to two news articles: What psychology’s crisis means for the future of science by Brian Resnick and These doctors want to fix a huge problem with drug trials. Why isn’t anyone listening? by Julia Belluz.
I don’t really have anything to add here beyond what I’ve blogged on these topics before. (I mean, sure, I could laugh at this quote, “The average person cannot evaluate a scientific finding for themselves any more easily than they can represent themselves in court or perform surgery on their own appendix,” which came from a psychology professor who is notorious for claiming that the replication rate in psychology is “statistically indistinguishable from 100%”—but I won’t go there.)
No, I just wanted to express pleasure that journalists are seeing the big picture here. At this point there’s a large cohort of science writers who’ve moved beyond the “Malcolm Gladwell” or “Freakonomics” model of scientist-as-hero, or the “David Brooks” model of believing anything that confirms your political views, or even the controversy-in-the-lab model, to a clearer view of science as a collective enterprise. We really do seem to be moving forward, even in the past five or ten years. Science reporters are no longer stenographers; they are active citizens of the scientific community.
Re: performing surgery on ones own appendix:
Off topic, but I remember reading a fascinating report in a Journal by a doctor who was part of an old Antartic expedition circa 1950’s who successfully performed a self-appendictiomy.
Amazing report it was.
An amazing story indeed: http://www.southpolestation.com/trivia/igy1/appendix.html
Relatedly, there was this story in the NY Times last week:
http://www.nytimes.com/2016/08/11/upshot/were-so-confused-the-problems-with-food-and-exercise-studies.html?_r=0
Nice quote from the story:
The problem is one of signal to noise. You can’t discern the signal — a lower risk of dementia, or a longer life, or less obesity, or less cancer — because the noise, the enormous uncertainty in the measurement of such things as how much you exercise or what exactly you eat, is overwhelming. The signal is often weak, meaning if there is an effect of lifestyle it is minuscule, nothing like the link between smoking and lung cancer, for example.
And there is no gold standard of measurement, nothing that everyone agrees on and uses to measure aspects of lifestyle.
The result is a large body of studies whose conclusions are not reproducible. “We don’t know how to measure diet or exercise,” said Dr. Barnett Kramer, director of the National Cancer Institute’s division of disease prevention.
Bob
Oops. Got HTML wrong twice. Should have been
The problem is one of signal to noise. You can’t discern the signal — a lower risk of dementia, or a longer life, or less obesity, or less cancer — because the noise, the enormous uncertainty in the measurement of such things as how much you exercise or what exactly you eat, is overwhelming. The signal is often weak, meaning if there is an effect of lifestyle it is minuscule, nothing like the link between smoking and lung cancer, for example.
And there is no gold standard of measurement, nothing that everyone agrees on and uses to measure aspects of lifestyle.
The result is a large body of studies whose conclusions are not reproducible. “We don’t know how to measure diet or exercise,” said Dr. Barnett Kramer, director of the National Cancer Institute’s division of disease prevention.
We can be happy. Or sad that the journalists had to figure it out instead of Journal Editors.
We can only hope so. (And pray. ;-) )
||
“Science reporters are getting the picture”
….well those vox.com references don’t support that sea-change viewpoint– unconvincing stuff.
For decades observers have bemoaned the lack of scientific rigor and accuracy in scientific journalism– not much change is apparent now.
General trust in journalists on any topic is usually misplaced. And of course U.S. opinion polls place journalists down in the basement with lawyers and politicians.
Suppose a Congressional committee “preregisters” a “study” of the Benghazi attack, but the witnesses are asked many unrelated questions and the data is carefully sifted for any possible areas of further investigation. Emails are subpoenaed and suddenly the existence of a private email server is discovered, which becomes the next area of focus. But then, with thousands of emails to review, the focus of the “study” turns to the other thousands of emails that remain missing. On the publication side, journalists feel their own pressures to publish new and surprising results and follow the “studies” wherever they lead, without regard to the “outcome switching”. OK, that’s a little overstated and the parallels are imperfect, but fixing one type of problem may help the other.
Interesting and shocking to me:
Ben Goldacre et al.: How did NEJM [New England Journal of Medicine] respond when we tried to correct 20 misreported trials? (misreported because of unreported outcome switching)
http://www.cebm.net/how-did-nejm-respond-when-we-tried-to-correct-20-misreported-trials/
NEJM is the highest impact factor journal in medicine.
It also aligns with the annoying practice: Never admit that you have made a mistake.
If you someday teach this class, please let me audit: https://hardsci.wordpress.com/2016/08/11/everything-is-fucked-the-syllabus/
“Wear black” to the final, well played.
It would make for a very purposeful course, but I recall that Sanjay mentioned he was not teaching for the next year or so.
Belluz also had a nice article in December on outcome switching:
http://www.vox.com/2015/12/29/10654056/ben-goldacre-compare-trials
Gilbert’s quote about evaluating scientific findings is such a joke. The point isn’t that regular people or even journalists are evaluating scientific findings, it’s that experts in statistical methods are looking at the way those methods are being used in these studies and pointing out that a big chunk of “evidence” for various theories isn’t really evidence for the theories at all.
I guess I can understand the defensiveness, it can’t feel good when the credibility of your field is disintegrating.
Hgfalling:
It’s even worse with Gilbert, as on one hand he’s hearing from actual experts that the credibility of his field is disintegrating, while on the other hand he’s getting feedback from TV producers, Ted talk audiences, book publishers, etc., that everything he and his friends are doing is awesome. He’s standing with one foot in an ice bath and the other in boiling water.