Skip to content
Archive of posts filed under the Miscellaneous Statistics category.

Opportunity for Comment!

(This is Dan) Last September, Jonah, Aki, Michael, Andrew and I wrote a paper on the role of visualization in the Bayesian workflow.  This paper is going to be published as a discussion paper in the Journal of the Royal Statistical Society Series A and the associated read paper meeting (where we present the paper and […]

What is the role of qualitative methods in addressing issues of replicability, reproducibility, and rigor?

Kara Weisman writes: I’m a PhD student in psychology, and I attended your talk at the Stanford Graduate School of Business earlier this year. I’m writing to ask you about something I remember you discussing at that talk: The possible role of qualitative methods in addressing issues of replicability, reproducibility, and rigor. In particular, I […]

Power analysis and NIH-style statistical practice: What’s the implicit model?

So. Following up on our discussion of “the 80% power lie,” I was thinking about the implicit model underlying NIH’s 80% power rule. Several commenters pointed out that, to have your study design approved by NSF, it’s not required that you demonstrate that you have 80% power for real; what’s needed is to show 80% […]

Chasing the noise in industrial A/B testing: what to do when all the low-hanging fruit have been picked?

Commenting on this post on the “80% power” lie, Roger Bohn writes: The low power problem bugged me so much in the semiconductor industry that I wrote 2 papers about around 1995. Variability estimates come naturally from routine manufacturing statistics, which in semicon were tracked carefully because they are economically important. The sample size is […]

One good and one bad response to statistics’ diversity problem

(This is Dan) As conference season rolls into gear, I thought I’d write a short post contrasting some responses by statistical societies to the conversation that the community has been having about harassment of women and minorities at workshops and conferences. ISI: Do what I say, not what I do Let’s look at a different diversity […]

Average predictive comparisons and the All Else Equal fallacy

Annie Wang writes: I’m a law student (and longtime reader of the blog), and I’m writing to flag a variant of the “All Else Equal” fallacy in ProPublica’s article on the COMPAS Risk Recidivism Algorithm. The article analyzes how statistical risk assessments, which are used in sentencing and bail hearings, are racially biased. (Although this […]

Forking paths come from choices in data processing and also from choices in analysis

Michael Wiebe writes: I’m a PhD student in economics at UBC. I’m trying to get a good understanding of the garden of forking paths, and I have some questions about your paper with Eric Loken. You describe the garden of forking paths as “researcher degrees of freedom without fishing” (#3), where the researcher only performs […]

Here are the data and code for that study of Puerto Rico deaths

A study just came out, Mortality in Puerto Rico after Hurricane Maria, by Nishant Kishore et al.: Using a representative, stratified sample, we surveyed 3299 randomly chosen households across Puerto Rico to produce an independent estimate of all-cause mortality after the hurricane. Respondents were asked about displacement, infrastructure loss, and causes of death. We calculated […]

All Fools in a Circle

A graduate student in psychology writes: Grants do not fund you unless you have pilot data – and moreover – show some statistically significant finding in your N of 20 or 40 – in essence trying to convince the grant reviewers that there is “something there” worth them providing your lab lots of money to […]

“Not statistically significant” != 0, stents edition

Doug Helmreich writes: OK, I work at a company that is involved in stents, so I’m not unbiased, but… and especially The research design is pretty cool—placebo participants got a sham surgery with no stent implanted. The results show that people with the stent did have better metrics than those with just the […]

Some experiments are just too noisy to tell us much of anything at all: Political science edition

Sointu Leikas pointed us to this published research article, “Exposure to inequality affects support for redistribution.” Leikas writes that “it seems to be a really apt example of “researcher degrees of freedom.’” Here’s the abstract of the paper: As the world’s population grows more urban, encounters between members of different socioeconomic groups occur with greater […]


Dale Lehman writes: This one’s on a topic you have blogged about often and one that I still think is under-appreciated: measurement. The Economist recently reported on this fascinating article about lightning strikes and their apparent sensitivity to shipping lanes and the associated pollution. I [Lehman] immediately wondered about whether there is a bias in […]

Comment of the year

From Jeff: “The decision to use mice for that study was terrible.” “Yeah, I know—and such small samples!” Sure, it’s only May. But I don’t think we’ll see anything better for awhile, so I’m happy to give out the award right now.

“I admire the authors for simply admitting they made an error and stating clearly and without equivocation that their original conclusions were not substantiated.”

David Allison writes: I hope you will consider covering this in your blog. I admire the authors for simply admitting they made an error and stating clearly and without equivocation that their original conclusions were not substantiated. More attention to the confusing effects of regression to the mean are warranted as is more praise for […]

The anthropic principle in statistics

The anthropic principle in physics states that we can derive certain properties of the world, or even the universe, based on the knowledge of our existence. The earth can’t be too hot or too cold, there needs to be oxygen and water, etc., which in turn implies certain things about our solar system, and so […]

The statistical significance filter leads to overoptimistic expectations of replicability

Shravan Vasishth, Daniela Mertzen, Lena Jäger, et al. write: Treating a result as publishable just because the p-value is less than 0.05 leads to overoptimistic expectations of replicability. These overoptimistic expectations arise due to Type M(agnitude) error: when underpowered studies yield significant results, effect size estimates are guaranteed to be exaggerated and noisy. These effects […]

What is “weight of evidence” in bureaucratese?

Martha Smith writes:

Garden of forking paths – poker analogy

[image of cats playing poker] Someone who wishes to remain anonymous writes: Just wanted to point out an analogy I noticed between the “garden of forking paths” concept as it relates to statistical significance testing and poker strategy (a game I’ve played as a hobby). A big part of constructing a winning poker strategy nowadays […]

How to reduce Type M errors in exploratory research?

Miao Yu writes: Recently, I found this piece [a news article by Janet Pelley, Sulfur dioxide pollution tied to degraded sperm quality, published in Chemical & Engineering News] and the original paper [Inverse Association between Ambient Sulfur Dioxide Exposure and Semen Quality in Wuhan, China, by Yuewei Liu, published in Environmental Science & Technology]. Air […]

Aki’s favorite scientific books (so far)

A month ago I (Aki) started a series of tweets about “scientific books which have had big influence on me…”. They are partially in time order, but I can’t remember the exact order. I may have forgotten some, and some stretched the original idea, but I can recommend all of them. I have collected all […]