Skip to content
Archive of posts filed under the Zombies category.

“Using numbers to replace judgment”

Julian Marewski and Lutz Bornmann write: In science and beyond, numbers are omnipresent when it comes to justifying different kinds of judgments. Which scientific author, hiring committee-member, or advisory board panelist has not been confronted with page-long “publication manuals”, “assessment reports”, “evaluation guidelines”, calling for p-values, citation rates, h-indices, or other statistics in order to […]

The State of the Art

Christie Aschwanden writes: Not sure you will remember, but last fall at our panel at the World Conference of Science Journalists I talked with you and Kristin Sainani about some unconventional statistical methods being used in sports science. I’d been collecting material for a story, and after the meeting I sent the papers to Kristin. […]

Robustness checks are a joke

Someone pointed to this post from a couple years ago by Uri Simonsohn, who correctly wrote: Robustness checks involve reporting alternative specifications that test the same hypothesis. Because the problem is with the hypothesis, the problem is not addressed with robustness checks. Simonsohn followed up with an amusing story: To demonstrate the problem I [Simonsohn] […]

Chocolate milk! Another stunning discovery from an experiment on 24 people!

Mike Hull writes: I was reading over this JAMA Brief Report and could not figure out what they were doing with the composite score. Here are the cliff notes: Study tested milk vs dark chocolate consumption on three eyesight performance parameters: (1) High-contrast visual acuity (2) Small-letter contrast sensitivity (3) Large-letter contrast sensitivity Only small-letter […]

“Recapping the recent plagiarism scandal”

Benjamin Carlisle writes: A year ago, I received a message from Anna Powell-Smith about a research paper written by two doctors from Cambridge University that was a mirror image of a post I wrote on my personal blog roughly two years prior. The structure of the document was the same, as was the rationale, the […]

The purported CSI effect and the retroactive precision fallacy

Regarding our recent post on the syllogism that ate science, someone points us to this article, “The CSI Effect: Popular Fiction About Forensic Science Affects Public Expectations About Real Forensic Science,” by N. J. Schweitzer and Michael J. Saks. We’ll get to the CSI Effect in a bit, but first I want to share the […]

Cornell prof (but not the pizzagate guy!) has one quick trick to getting 1700 peer reviewed publications on your CV

From the university webpage: Robert J. Sternberg is Professor of Human Development in the College of Human Ecology at Cornell University. . . . Sternberg is the author of over 1700 refereed publications. . . . How did he compile over 1700 refereed publications? Nick Brown tells the story: I [Brown] was recently contacted by […]

An actual quote from a paper published in a medical journal: “The data, analytic methods, and study materials will not be made available to other researchers for purposes of reproducing the results or replicating the procedure.”

Someone writes: So the NYT yesterday has a story about this study I am directed to it and am immediately concerned about all the things that make this study somewhat dubious. Forking paths in the definition of the independent variable, sample selection in who wore the accelerometers, ignorance of the undoubtedly huge importance of interactions […]

“Fudged statistics on the Iraq War death toll are still circulating today”

Mike Spagat shares this story entitled, “Fudged statistics on the Iraq War death toll are still circulating today,” which discusses problems with a paper published in a scientific journal in 2006, and errors that a reporter inadvertently included in a recent news article. Spagat writes: The Lancet could argue that if [Washington Post reporter Philip] […]

Statistical Modeling, Causal Inference, and Social Science Regrets Its Decision to Hire Cannibal P-hacker as Writer-at-Large

It is not easy to admit our mistakes, particularly now, given the current media climate and general culture of intolerance on college campuses. Still, we feel that we owe our readers an apology. We should not have hired Cannibal P-hacker, an elegant scientist and thinker who, we have come to believe, after serious consideration, does […]

Don’t calculate post-hoc power using observed estimate of effect size

Aleksi Reito writes: The statement below was included in a recent issue of Annals of Surgery: But, as 80% power is difficult to achieve in surgical studies, we argue that the CONSORT and STROBE guidelines should be modified to include the disclosure of power—even if less than 80%—with the given sample size and effect size […]

“Tweeking”: The big problem is not where you think it is.

In her recent article about pizzagate, Stephanie Lee included this hilarious email from Brian Wansink, the self-styled “world-renowned eating behavior expert for over 25 years”: OK, what grabs your attention is that last bit about “tweeking” the data to manipulate the p-value, where Wansink is proposing research misconduct (from NIH: “Falsification: Manipulating research materials, equipment, […]

Narcolepsy Could Be ‘Sleeper Effect’ in Trump and Brexit Campaigns

Kevin Lewis sent along this example of what in social science is called the “ecological fallacy.” Below is a press release that I’ve changed in only a few places: UNDER EMBARGO UNTIL MARCH 8, 2018 AT 10 AM EST Media Contact: Public and Media Relations Manager Society for Personality and Social Psychology press@spsp.org Narcolepsy Could […]

“Check out table 4.”

A colleague sent along this article and writes: Check out table 4. this is ERC funded research (the very best of European science get this money). OK, now I was curious, so I scrolled through to table 4. Here it is: Yup, it’s horrible. I don’t know that I’d call it cargo cult science at […]

“It’s Always Sunny in Correlationville: Stories in Science,” or, Science should not be a game of Botticelli

There often seems to be an attitude among scientists and journal editors that, if a research team has gone to the trouble of ensuring rigor in some part of their study (whether in the design, the data collection, or the analysis, but typically rigor is associated with “p less than .05” and some random assignment […]

“We continuously increased the number of animals until statistical significance was reached to support our conclusions” . . . I think this is not so bad, actually!

For some reason, people have recently been asking me what I think of this journal article which I wrote about months ago . . . so I’ll just repeat my post here: Jordan Anaya pointed me to this post, in which Casper Albers shared this snippet from a recently-published paper from an article in Nature […]

Some clues that this study has big big problems

Paul Alper writes: This article from the New York Daily News, reproduced in the Minneapolis Star Tribune, is so terrible in so many ways. Very sad commentary regarding all aspects of statistics education and journalism. The news article, by Joe Dziemianowicz, is called “Study says drinking alcohol is key to living past 90,” with subheading, […]

When anyone claims 80% power, I’m skeptical.

A policy analyst writes: I saw you speak at ** on Bayesian methods. . . . I had been asked to consult on a large national evaluation of . . . [details removed to preserve anonymity] . . . and had suggested treading carefully around the use of Bayesian statistics in this study (basing it […]

The scandal isn’t what’s retracted, the scandal is what’s not retracted.

Andrew Han at Retraction Watch reports on a paper, “Structural stigma and all-cause mortality in sexual minority populations,” published in 2014 by Mark Hatzenbuehler, Anna Bellatorre, Yeonjin Lee, Brian Finch, Peter Muennig, and Kevin Fiscella, that claimed: Sexual minorities living in communities with high levels of anti-gay prejudice experienced a higher hazard of mortality than […]

The competing narratives of scientific revolution

Back when we were reading Karl Popper’s Logic of Scientific Discovery and Thomas Kuhn’s Structure of Scientific Revolutions, who would’ve thought that we’d be living through a scientific revolution ourselves? Scientific revolutions occur on all scales, but here let’s talk about some of the biggies: 1850-1950: Darwinian revolution in biology, changed how we think about […]