Skip to content
Archive of posts filed under the Miscellaneous Statistics category.

Don’t calculate post-hoc power using observed estimate of effect size

Aleksi Reito writes: The statement below was included in a recent issue of Annals of Surgery: But, as 80% power is difficult to achieve in surgical studies, we argue that the CONSORT and STROBE guidelines should be modified to include the disclosure of power—even if less than 80%—with the given sample size and effect size […]

“Tweeking”: The big problem is not where you think it is.

In her recent article about pizzagate, Stephanie Lee included this hilarious email from Brian Wansink, the self-styled “world-renowned eating behavior expert for over 25 years”: OK, what grabs your attention is that last bit about “tweeking” the data to manipulate the p-value, where Wansink is proposing research misconduct (from NIH: “Falsification: Manipulating research materials, equipment, […]

A psychology researcher uses Stan, multiverse, and open data exploration to explore human memory

Under the heading, “An example of Stan to the rescue, multiverse analysis, and psychologists trying to do well,” Greg Cox writes: I’m currently a postdoc at Syracuse University studying how human memory works. I wanted to forward a paper of ours [“Information and Processes Underlying Semantic and Episodic Memory Across Tasks, Items, and Individuals,” by […]

Columbia Data Science Institute art contest

This is a great idea! Unfortunately, only students at Columbia can submit. I encourage other institutions to do such contests too. We did something similar at Columbia, maybe 10 or 15 years ago? It went well, we just didn’t have the energy to do it again every year, as we’d initially planned. So I’m very […]

High-profile statistical errors occur in the physical sciences too, it’s not just a problem in social science.

In an email with subject line, “Article full of forking paths,” John Williams writes: I thought you might be interested in this article by John Sabo et al., which was the cover article for the Dec. 8 issue of Science. The article is dumb in various ways, some of which are described in the technical […]

Bothered by non-monotonicity? Here’s ONE QUICK TRICK to make you happy.

We’re often modeling non-monotonic functions. For example, performance at just about any task increases with age (babies can’t do much!) and then eventually decreases (dead people can’t do much either!). Here’s an example from a few years ago: A function g(x) that increases and then decreases can be modeled by a quadratic, or some more […]

The gaps between 1, 2, and 3 are just too large.

Someone who wishes to remain anonymous points to a new study of David Yeager et al. on educational mindset interventions (link from Alex Tabarrok) and asks: On the blog we talk a lot about bad practice and what not to do. Might this be an example of how *to do* things? Or did they just […]

Some clues that this study has big big problems

Paul Alper writes: This article from the New York Daily News, reproduced in the Minneapolis Star Tribune, is so terrible in so many ways. Very sad commentary regarding all aspects of statistics education and journalism. The news article, by Joe Dziemianowicz, is called “Study says drinking alcohol is key to living past 90,” with subheading, […]

“To get started, I suggest coming up with a simple but reasonable model for missingness, then simulate fake complete data followed by a fake missingness pattern, and check that you can recover your missing-data model and your complete data model in that fake-data situation. You can then proceed from there. But if you can’t even do it with fake data, you’re sunk.”

Alex Konkel writes on a topic that never goes out of style: I’m working on a data analysis plan and am hoping you might help clarify something you wrote regarding missing data. I’m somewhat familiar with multiple imputation and some of the available methods, and I’m also becoming more familiar with Bayesian modeling like in […]

In statistics, we talk about uncertainty without it being viewed as undesirable

Lauren Kennedy writes: I’ve noticed that statistics (or at least applied statistics) has this nice ability to talk about uncertainty without it being viewed as undesirable. Stan Con had that atmosphere and I think it just makes everyone so much more willing to debug, discuss and generate new ideas. Indeed, in statistics I’ve seen fierce […]

When anyone claims 80% power, I’m skeptical.

A policy analyst writes: I saw you speak at ** on Bayesian methods. . . . I had been asked to consult on a large national evaluation of . . . [details removed to preserve anonymity] . . . and had suggested treading carefully around the use of Bayesian statistics in this study (basing it […]

Data concerns when interpreting comparisons of gender equality between countries

A journalist pointed me to this research article, “Gender equality and sex differences in personality: evidence from a large, multi-national sample,” by Tim Kaiser (see also news report by Angela Lashbrook here), which states: A large, multinational (N = 926,383) dataset was used to examine sex differences in Big Five facet scores for 70 countries. […]

No, I don’t think it’s the file drawer effect

Someone named Andrew Certain writes: I’ve been reading your blog since your appearance on Econtalk . . . explaining the ways in which statistics are misused/misinterpreted in low-sample/high-noise studies. . . . I recently came across a meta-analysis on stereotype threat [a reanalysis by Emil Kirkegaard] by that identified a clear relationship between smaller sample […]

“Usefully skeptical science journalism”

Dean Eckles writes: I like this Wired piece on the challenges of learning about how technologies are affecting us and children. The journalist introducing a nice analogy (that he had in mind before talking with me — I’m briefly quoted) between the challenges in nutrition (and observational epidemiology more generally) and in studying “addictive” technologies. […]

Response to Rafa: Why I don’t think ROC [receiver operating characteristic] works as a model for science

Someone pointed me to this post from a few years ago where Rafael Irizarry argues that scientific “pessimists” such as myself are, at least in some fields, “missing a critical point: that in practice, there is an inverse relationship between increasing rates of true discoveries and decreasing rates of false discoveries and that true discoveries […]

From no-data to data: The awkward transition

I was going to write a post with the above title, but now I don’t remember what I was going to say!

“The idea of replication is central not just to scientific practice but also to formal statistics . . . Frequentist statistics relies on the reference set of repeated experiments, and Bayesian statistics relies on the prior distribution which represents the population of effects.”

Rolf Zwaan (who we last encountered here in “From zero to Ted talk in 18 simple steps”), Alexander Etz, Richard Lucas, and M. Brent Donnellan wrote an article, “Making replication mainstream,” which begins: Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component […]

If you have a measure, it will be gamed (politics edition).

They sometimes call it Campbell’s Law: New York Governor Andrew Cuomo is not exactly known for drumming up grassroots enthusiasm and small donor contributions, so it was quite a surprise on Monday when his reelection campaign reported that more than half of his campaign contributors this year gave $250 or less. But wait—a closer examination […]

The statistical checklist: Could there be a list of guidelines to help analysts do better work?

[image of cat with a checklist] Paul Cuffe writes: Your idea of “researcher degrees of freedom” [actually not my idea; the phrase comes from Simmons, Nelson, and Simonsohn] really resonates with me: I’m continually surprised by how many researchers freestyle their way through a statistical analysis, using whatever tests, and presenting whatever results, strikes their […]

He wants to model a proportion given some predictors that sum to 1

Joël Gombin writes: I’m wondering what your take would be on the following problem. I’d like to model a proportion (e.g., the share of the vote for a given party at some territorial level) in function of some compositional data (e.g., the sociodemographic makeup of the voting population), and this, in a multilevel fashion (allowing […]