Archive of posts filed under the Miscellaneous Statistics category.

Chasing the noise in industrial A/B testing: what to do when all the low-hanging fruit have been picked?

Commenting on this post on the “80% power” lie, Roger Bohn writes: The low power problem bugged me so much in the semiconductor industry that I wrote 2 papers about around 1995. Variability estimates come naturally from routine manufacturing statistics, which in semicon were tracked carefully because they are economically important. The sample size is […]

One good and one bad response to statistics’ diversity problem

(This is Dan) As conference season rolls into gear, I thought I’d write a short post contrasting some responses by statistical societies to the conversation that the community has been having about harassment of women and minorities at workshops and conferences. ISI: Do what I say, not what I do Let’s look at a different diversity […]

Average predictive comparisons and the All Else Equal fallacy

Annie Wang writes: I’m a law student (and longtime reader of the blog), and I’m writing to flag a variant of the “All Else Equal” fallacy in ProPublica’s article on the COMPAS Risk Recidivism Algorithm. The article analyzes how statistical risk assessments, which are used in sentencing and bail hearings, are racially biased. (Although this […]

Forking paths come from choices in data processing and also from choices in analysis

Michael Wiebe writes: I’m a PhD student in economics at UBC. I’m trying to get a good understanding of the garden of forking paths, and I have some questions about your paper with Eric Loken. You describe the garden of forking paths as “researcher degrees of freedom without fishing” (#3), where the researcher only performs […]

Here are the data and code for that study of Puerto Rico deaths

A study just came out, Mortality in Puerto Rico after Hurricane Maria, by Nishant Kishore et al.: Using a representative, stratified sample, we surveyed 3299 randomly chosen households across Puerto Rico to produce an independent estimate of all-cause mortality after the hurricane. Respondents were asked about displacement, infrastructure loss, and causes of death. We calculated […]

All Fools in a Circle

A graduate student in psychology writes: Grants do not fund you unless you have pilot data – and moreover – show some statistically significant finding in your N of 20 or 40 – in essence trying to convince the grant reviewers that there is “something there” worth them providing your lab lots of money to […]

“Not statistically significant” != 0, stents edition

Doug Helmreich writes: OK, I work at a company that is involved in stents, so I’m not unbiased, but… http://www3.imperial.ac.uk/newsandeventspggrp/imperialcollege/newssummary/news_2-11-2017-15-52-46 and especially https://www.nytimes.com/2017/11/02/health/heart-disease-stents.html The research design is pretty cool—placebo participants got a sham surgery with no stent implanted. The results show that people with the stent did have better metrics than those with just the […]

Some experiments are just too noisy to tell us much of anything at all: Political science edition

Sointu Leikas pointed us to this published research article, “Exposure to inequality affects support for redistribution.” Leikas writes that “it seems to be a really apt example of “researcher degrees of freedom.’” Here’s the abstract of the paper: As the world’s population grows more urban, encounters between members of different socioeconomic groups occur with greater […]

Kaleidoscope

Dale Lehman writes: This one’s on a topic you have blogged about often and one that I still think is under-appreciated: measurement. The Economist recently reported on this fascinating article about lightning strikes and their apparent sensitivity to shipping lanes and the associated pollution. I [Lehman] immediately wondered about whether there is a bias in […]

Comment of the year

From Jeff: “The decision to use mice for that study was terrible.” “Yeah, I know—and such small samples!” Sure, it’s only May. But I don’t think we’ll see anything better for awhile, so I’m happy to give out the award right now.

“I admire the authors for simply admitting they made an error and stating clearly and without equivocation that their original conclusions were not substantiated.”

David Allison writes: I hope you will consider covering this in your blog. I admire the authors for simply admitting they made an error and stating clearly and without equivocation that their original conclusions were not substantiated. More attention to the confusing effects of regression to the mean are warranted as is more praise for […]

The anthropic principle in statistics

The anthropic principle in physics states that we can derive certain properties of the world, or even the universe, based on the knowledge of our existence. The earth can’t be too hot or too cold, there needs to be oxygen and water, etc., which in turn implies certain things about our solar system, and so […]

The statistical significance filter leads to overoptimistic expectations of replicability

Shravan Vasishth, Daniela Mertzen, Lena Jäger, et al. write: Treating a result as publishable just because the p-value is less than 0.05 leads to overoptimistic expectations of replicability. These overoptimistic expectations arise due to Type M(agnitude) error: when underpowered studies yield significant results, effect size estimates are guaranteed to be exaggerated and noisy. These effects […]

What is “weight of evidence” in bureaucratese?

Martha Smith writes:

Garden of forking paths – poker analogy

[image of cats playing poker] Someone who wishes to remain anonymous writes: Just wanted to point out an analogy I noticed between the “garden of forking paths” concept as it relates to statistical significance testing and poker strategy (a game I’ve played as a hobby). A big part of constructing a winning poker strategy nowadays […]

How to reduce Type M errors in exploratory research?

Miao Yu writes: Recently, I found this piece [a news article by Janet Pelley, Sulfur dioxide pollution tied to degraded sperm quality, published in Chemical & Engineering News] and the original paper [Inverse Association between Ambient Sulfur Dioxide Exposure and Semen Quality in Wuhan, China, by Yuewei Liu, published in Environmental Science & Technology]. Air […]

Aki’s favorite scientific books (so far)

A month ago I (Aki) started a series of tweets about “scientific books which have had big influence on me…”. They are partially in time order, but I can’t remember the exact order. I may have forgotten some, and some stretched the original idea, but I can recommend all of them. I have collected all […]

Doomsday! Problems with interpreting a confidence interval when there is no evidence for the assumed sampling model

Mark Brown pointed me to a credulous news article in the Washington Post, “We have a pretty good idea of when humans will go extinct,” which goes: A Princeton University astrophysicist named J. Richard Gott has a surprisingly precise answer to that question . . . to understand how he arrived at it and what […]

Walter Benjamin on storytelling

After we discussed my paper with Thomas Basbøll, “When do stories work? Evidence and illustration in the social sciences,” Jager Hartman wrote to me: Here is a link to the work by Walter Benjamin I think of when I think of storytelling. He uses storytelling throughout his works and critiques done on his works are […]

“We continuously increased the number of animals until statistical significance was reached to support our conclusions” . . . I think this is not so bad, actually!

Jordan Anaya pointed me to this post, in which Casper Albers shared this snippet from a recently-published paper from an article in Nature Communications: The subsequent twitter discussion is all about “false discovery rate” and statistical significance, which I think completely misses the point. The problems Before I get to why I think the quoted […]