Skip to content
Archive of posts filed under the Miscellaneous Statistics category.

The failure of null hypothesis significance testing when studying incremental changes, and what to do about it

A few months ago I wrote a post, “Cage match: Null-hypothesis-significance-testing meets incrementalism. Nobody comes out alive.” I soon after turned it into an article, published in Personality and Social Psychology Bulletin, with the title given above and the following abstract: A standard mode of inference in social and behavioral science is to establish stylized […]

Walk a Crooked MiIe

An academic researcher writes: I was wondering if you might have any insight or thoughts about a problem that has really been bothering me. I have taken a winding way through academia, and I am seriously considering a career shift that would allow me to do work that more directly translates to societal good and […]

Yes, you can do statistical inference from nonrandom samples. Which is a good thing, considering that nonrandom samples are pretty much all we’ve got.

Luiz Caseiro writes: 1. P-values and Confidence Intervals are used to draw inferences about a population from a sample. Is that right? 2. As far as I researched, standard statistical softwares usually compute confidence intervals (CI) and p-values assuming that we have a simple random sample. Is that right? 3. If we have another kind […]

Two steps forward, one step back

Alex Gamma writes in with what he describes as “an amusing little story” from two years ago: When Deaton & Case published their study, and after your re-analysis had uncovered the missing age-correction, I’ve pointed out this issue to several news blogs that reported on the study, but were not aware of the problem (only […]

Bin Yu and Karl Kumbier: “Artificial Intelligence and Statistics”

Yu and Kumbier write: Artificial intelligence (AI) is intrinsically data-driven. It calls for the application of statistical concepts through human-machine collaboration during generation of data, development of algo- rithms, and evaluation of results. This paper discusses how such human-machine collaboration can be approached through the statistical concepts of population, question of interest, representativeness of training […]

“Little Data” etc.: My talk at NYU this Friday, 8 Dec 2017

I’ll be talking at the NYU business school, in the department of information, operations, and management sciences, this Fri, 8 Dec 2017, at 12:30, in room KMC 4-90 (wherever that is): Little Data: How Traditional Statistical Ideas Remain Relevant in a Big-Data World; or, The Statistical Crisis in Science; or, Open Problems in Bayesian Data […]

The “80% power” lie

OK, so this is nothing new. Greg Francis said it, and Uri Simonsohn said it, Ulrich Schimmack said it, lots of people have said it. But it’s worth saying again. To get NIH funding, you need to demonstrate (that is, convincingly claim) that your study has 80% power. I hate the term “power” as it’s […]

Orphan drugs and forking paths: I’d prefer a multilevel model but to be honest I’ve never fit such a model for this sort of problem

Amos Elberg writes: I’m writing to let you know about a drug trial you may find interesting from a statistical perspective. As you may know, the relatively recent “orphan drug” laws allow (basically) companies that can prove an off-patent drug treats an otherwise untreatable illness, to obtain intellectual property protection for otherwise generic or dead […]

A new definition of the nerd?

Jonathan Falk points to this book excerpt by Michael Lewis, who writes: A lot of what people did and said when they “predicted” things, Morey now realized, was phony: pretending to know things rather than actually knowing things. There were a great many interesting questions in the world to which the only honest answer was, […]

Oooh, I hate all talk of false positive, false negative, false discovery, etc.

A correspondent writes: I think this short post on p value, bayes, and false discovery rate contains some misinterpretations. My reply: Oooh, I hate all talk of false positive, false negative, false discovery, etc. I posted this not because I care about someone, somewhere, being “wrong on the internet.” Rather, I just think there’s so […]

Expediting organised experience: What statistics should be?

The above diagram is by John F. Sowa and it depicts a high level view of C.S. Peirce’s classification of the sciences of discovery (you have been warned). The dotted lines indicate what on the right should be informed by what is on the left. I think there is a missing spot (or better an […]

What’s the point of a robustness check?

Diomides Mavroyiannis writes: I am currently a doctoral student in economics in France, I’ve been reading your blog fo awhile and I have this question that’s bugging me. I often go to seminars where speakers present their statistical evidence for various theses. I was wondering if you could shed light on robustness checks, what is […]

“Five ways to fix statistics”

Nature magazine just published a short feature on statistics and the replication crisis, featuring the following five op-ed-sized bits: Jeff Leek: Adjust for human cognition Blake McShane, Andrew Gelman, David Gal, Christian Robert, and Jennifer Tackett: Abandon statistical significance David Colquhoun: State false-positive risk, too Michele Nuijten: Share analysis plans and results Steven Goodman: Change […]

No to inferential thresholds

Harry Crane points us to this new paper, “Why ‘Redefining Statistical Significance’ Will Not Improve Reproducibility and Could Make the Replication Crisis Worse,” and writes: Quick summary: Benjamin et al. claim that FPR would improve by factors greater than 2 and replication rates would double under their plan. That analysis ignores the existence and impact […]

No no no no no on “The oldest human lived to 122. Why no person will likely break her record.”

I came across this news article by Brian Resnick entitled: The oldest human lived to 122. Why no person will likely break her record. Even with better medicine, living past 120 years will be extremely unlikely. I was skeptical, and I really didn’t buy it after reading the research article, “Evidence for a limit to […]

I hate that “Iron Law” thing

Dahyeon Jeong wrote: While I was reading your today’s post “Some people are so easy to contact and some people aren’t”, I’ve come across your older posts including “Edlin’s rule for routinely scaling down published estimates.” In this post you write: Also, yeah, that Iron Law thing sounds horribly misleading. I’d not heard that particular […]

What should this student do? His bosses want him to p-hack and they don’t even know it!

Someone writes: I’m currently a PhD student in the social sciences department of a university. I recently got involved with a group of professors working on a project which involved some costly data-collection. None of them have any real statistical prowess, so they came to me to perform their analyses, which I was happy to […]

The Statistical Crisis in Science—and How to Move Forward (my talk next Monday 6pm at Columbia)

I’m speaking Mon 13 Nov, 6pm, at Low Library Rotunda at Columbia: The Statistical Crisis in Science—and How to Move Forward Using examples ranging from elections to birthdays to policy analysis, Professor Andrew Gelman will discuss ways in which statistical methods have failed, leading to a replication crisis in much of science, as well as […]

The time reversal heuristic (priming and voting edition)

Ed Yong writes: Over the past decade, social psychologists have dazzled us with studies showing that huge social problems can seemingly be rectified through simple tricks. A small grammatical tweak in a survey delivered to people the day before an election greatly increases voter turnout. A 15-minute writing exercise narrows the achievement gap between black […]

Post-publication review succeeds again! (Two-lines edition.)

A couple months ago, Uri Simonsohn posted online a suggested statistical method for detecting nonmonotonicity in data. He called it: “Two-lines: The First Valid Test of U-Shaped Relationships.” With a title like that, I guess you’re asking for it. And, indeed, awhile later I received an email from Yair Heller identifying some problems with Uri’s […]