Skip to content
Archive of posts filed under the Miscellaneous Statistics category.

No to inferential thresholds

Harry Crane points us to this new paper, “Why ‘Redefining Statistical Significance’ Will Not Improve Reproducibility and Could Make the Replication Crisis Worse,” and writes: Quick summary: Benjamin et al. claim that FPR would improve by factors greater than 2 and replication rates would double under their plan. That analysis ignores the existence and impact […]

No no no no no on “The oldest human lived to 122. Why no person will likely break her record.”

I came across this news article by Brian Resnick entitled: The oldest human lived to 122. Why no person will likely break her record. Even with better medicine, living past 120 years will be extremely unlikely. I was skeptical, and I really didn’t buy it after reading the research article, “Evidence for a limit to […]

I hate that “Iron Law” thing

Dahyeon Jeong wrote: While I was reading your today’s post “Some people are so easy to contact and some people aren’t”, I’ve come across your older posts including “Edlin’s rule for routinely scaling down published estimates.” In this post you write: Also, yeah, that Iron Law thing sounds horribly misleading. I’d not heard that particular […]

What should this student do? His bosses want him to p-hack and they don’t even know it!

Someone writes: I’m currently a PhD student in the social sciences department of a university. I recently got involved with a group of professors working on a project which involved some costly data-collection. None of them have any real statistical prowess, so they came to me to perform their analyses, which I was happy to […]

The Statistical Crisis in Science—and How to Move Forward (my talk next Monday 6pm at Columbia)

I’m speaking Mon 13 Nov, 6pm, at Low Library Rotunda at Columbia: The Statistical Crisis in Science—and How to Move Forward Using examples ranging from elections to birthdays to policy analysis, Professor Andrew Gelman will discuss ways in which statistical methods have failed, leading to a replication crisis in much of science, as well as […]

The time reversal heuristic (priming and voting edition)

Ed Yong writes: Over the past decade, social psychologists have dazzled us with studies showing that huge social problems can seemingly be rectified through simple tricks. A small grammatical tweak in a survey delivered to people the day before an election greatly increases voter turnout. A 15-minute writing exercise narrows the achievement gap between black […]

Post-publication review succeeds again! (Two-lines edition.)

A couple months ago, Uri Simonsohn posted online a suggested statistical method for detecting nonmonotonicity in data. He called it: “Two-lines: The First Valid Test of U-Shaped Relationships.” With a title like that, I guess you’re asking for it. And, indeed, awhile later I received an email from Yair Heller identifying some problems with Uri’s […]

More thoughts on that “What percent of Americans would you say are gay or lesbian?” survey

We had some discussion yesterday about this Gallup poll that asked respondents to guess the percentage of Americans who are gay. The average response was 23%—and this stunningly high number was not just driven by outliers: more than half the respondents estimated the proportion gay as 20% or more. All this is in stark contrast […]

Statistical Significance and the Dichotomization of Evidence (McShane and Gal’s paper, with discussions by Berry, Briggs, Gelman and Carlin, and Laber and Shedden)

Blake McShane sent along this paper by himself and David Gal, which begins: In light of recent concerns about reproducibility and replicability, the ASA issued a Statement on Statistical Significance and p-values aimed at those who are not primarily statisticians. While the ASA Statement notes that statistical significance and p-values are “commonly misused and misinterpreted,” […]

“Americans Greatly Overestimate Percent Gay, Lesbian in U.S.”

This sort of thing is not new but it’s still amusing. From a Gallup report by Frank Newport: The American public estimates on average that 23% of Americans are gay or lesbian, little changed from Americans’ 25% estimate in 2011, and only slightly higher than separate 2002 estimates of the gay and lesbian population. These […]

Contour as a verb

Our love is like the border between Greece and Albania – The Mountain Goats (In which I am uncharacteristically brief) Andrew’s answer to recent post reminded me of one of my favourite questions: how do you visualise uncertainty in spatial maps.  An interesting subspecies of this question relates to exactly how you can plot a contour […]

“Quality control” (rather than “hypothesis testing” or “inference” or “discovery”) as a better metaphor for the statistical processes of science

I’ve been thinking for awhile that the default ways in which statisticians think about science—and which scientists think about statistics—are seriously flawed, sometimes even crippling scientific inquiry in some subfields, in the way that bad philosophy can do. Here’s what I think are some of the default modes of thought: – Hypothesis testing, in which […]

My favorite definition of statistical significance

From my 2009 paper with Weakliem: Throughout, we use the term statistically significant in the conventional way, to mean that an estimate is at least two standard errors away from some “null hypothesis” or prespecified value that would indicate no effect present. An estimate is statistically insignificant if the observed value could reasonably be explained […]

This Friday at noon, join this online colloquium on replication and reproducibility, featuring experts in economics, statistics, and psychology!

Justin Esarey writes: This Friday, October 27th at noon Eastern time, the International Methods Colloquium will host a roundtable discussion on the reproducibility crisis in social sciences and a recent proposal to impose a stricter threshold for statistical significance. The discussion is motivated by a paper, “Redefine statistical significance,” recently published in Nature Human Behavior (and available […]

The Publicity Factory: How even serious research gets exaggerated by the process of scientific publication and reporting

The starting point is that we’ve seen a lot of talk about frivolous science, headline-bait such as the study that said that married women are more likely to vote for Mitt Romney when ovulating, or the study that said that girl-named hurricanes are more deadly than boy-named hurricanes, and at this point some of these […]

How to discuss your research findings without getting into “hypothesis testing”?

Zachary Horne writes: I regularly read your blog and have recently started using Stan. One thing that you’ve brought up in the discussion of nhst [null hypothesis significance testing] is the idea that hypothesis testing itself is problematic. However, because I am an experimental psychologist, one thing I do (or I think I’m doing anyway) […]

“La critique est la vie de la science”: I kinda get annoyed when people set themselves up as the voice of reason but don’t ever get around to explaining what’s the unreasonable thing they dislike.

Someone pointed me to a blog post, Negative Psychology, from 2014 by Jim Coan about the replication crisis in psychology. My reaction: I find it hard to make sense of what he is saying because he doesn’t offer any examples of the “negative psychology” phenomenon that he discussing. I kinda get annoyed when people set […]

Freelance orphans: “33 comparisons, 4 are statistically significant: much more than the 1.65 that would be expected by chance alone, so what’s the problem??”

From someone who would prefer to remain anonymous: As you may know, the relatively recent “orphan drug” laws allow (basically) companies that can prove an off-patent drug treats an otherwise untreatable illness, to obtain intellectual property protection for otherwise generic or dead drugs. This has led to a new business of trying large numbers of […]

Workshop on Interpretable Machine Learning

Andrew Gordon Wilson sends along this conference announcement: NIPS 2017 Symposium Interpretable Machine Learning Long Beach, California, USA December 7, 2017 Call for Papers: We invite researchers to submit their recent work on interpretable machine learning from a wide range of approaches, including (1) methods that are designed to be more interpretable from the start, […]

Please contribute to this list of the top 10 do’s and don’ts for doing better science

Demis Glasford does research in social psychology and asks: I was wondering if you had ever considered publishing a top ten ‘do’s/don’ts’ for those of us that are committed to doing better science, but don’t necessarily have the time to devote to all of these issues [of statistics and research methods]. Obviously, there is a […]