Skip to content
Archive of entries posted by

“Tweeking”: The big problem is not where you think it is.

In her recent article about pizzagate, Stephanie Lee included this hilarious email from Brian Wansink, the self-styled “world-renowned eating behavior expert for over 25 years”: OK, what grabs your attention is that last bit about “tweeking” the data to manipulate the p-value, where Wansink is proposing research misconduct (from NIH: “Falsification: Manipulating research materials, equipment, […]

Multilevel data collection and analysis for weight training (with R code)

[image of cat lifting weights] A graduate student who wishes to remain anonymous writes: I was wondering if you could answer an elementary question which came to mind after reading your article with Carlin on retrospective power analysis. Consider the field of exercise science, and in particular studies on people who lift weights. (I sometimes […]

A psychology researcher uses Stan, multiverse, and open data exploration to explore human memory

Under the heading, “An example of Stan to the rescue, multiverse analysis, and psychologists trying to do well,” Greg Cox writes: I’m currently a postdoc at Syracuse University studying how human memory works. I wanted to forward a paper of ours [“Information and Processes Underlying Semantic and Episodic Memory Across Tasks, Items, and Individuals,” by […]

How to graph a function of 4 variables using a grid

This came up in response to a student’s question. I wrote that, in general, you can plot a function y(x) on a simple graph. You can plot y(x,x2) by plotting y vs x and then having several lines showing different values of x2 (for example, x2=0, x2=0.5, x2=1, x2=1.5, x2=2, etc). You can plot y(x,x2,x3,x4) […]

Post-publication peer review: who’s qualified?

Gabriel Power writes: I don’t recall that you addressed this point in your posts on post-publication peer review [for example, here and here — ed.]. Who would be allowed to post reviews of a paper? Anyone? Only researchers? Only experts? Science is not a democracy. A study is not valid because a majority of people […]

A couple more papers on genetic diversity as an explanation for why Africa and remote Andean countries are so poor while Europe and North America are so wealthy

Back in 2013, I wrote a post regarding a controversial claim that high genetic diversity, or low genetic diversity, is bad for the economy: Two economics professors, Quamrul Ashraf and Oded Galor, wrote a paper, “The Out of Africa Hypothesis, Human Genetic Diversity, and Comparative Economic Development,” that is scheduled to appear in the American […]

The hot hand—in darts!

Roland Langrock writes: Since on your blog you’ve regularly been discussing hot hand literature – which we closely followed – I’m writing to share with you a new working paper we wrote on a potential hot hand pattern in professional darts. We use state-space models in which a continuous-valued latent “hotness” variable, modeled as an […]

What to do when your measured outcome doesn’t quite line up with what you’re interested in?

Matthew Poes writes: I’m writing a research memo discussing the importance of precisely aligning the outcome measures to the intervention activities. I’m making the point that an evaluation of the outcomes for a given intervention may net null results for many reasons, one of which could simply be that you are looking in the wrong […]

Don’t get fooled by observational correlations

Gabriel Power writes: Here’s something a little different: clever classrooms, according to which physical characteristics of classrooms cause greater learning. And the effects are large! Moving from the worst to the best design implies a gain of 67% of one year’s worth of learning! Aside from the dubiously large effect size, it looks like the […]

Columbia Data Science Institute art contest

This is a great idea! Unfortunately, only students at Columbia can submit. I encourage other institutions to do such contests too. We did something similar at Columbia, maybe 10 or 15 years ago? It went well, we just didn’t have the energy to do it again every year, as we’d initially planned. So I’m very […]

High-profile statistical errors occur in the physical sciences too, it’s not just a problem in social science.

In an email with subject line, “Article full of forking paths,” John Williams writes: I thought you might be interested in this article by John Sabo et al., which was the cover article for the Dec. 8 issue of Science. The article is dumb in various ways, some of which are described in the technical […]

Echo Chamber Incites Online Mob to Attack Math Profs

The story starts as follows: There’s evidence for greater variability in the distribution of men, compared to women, in various domains. Two math professors, Theodore Hill and Sergei Tabachnikov, wrote an article exploring a mathematical model for the evolution of this difference in variation, and send the article to the Mathematical Intelligencer, a magazine that […]

N=1 survey tells me Cynthia Nixon will lose by a lot (no joke)

Yes, you can learn a lot from N=1, as long as you have some auxiliary information. The other day I was talking with a friend who’s planning to vote for Andrew Cuomo in the primary. What about Cynthia Nixon? My friend wasn’t even considering voting for her. Now, my friend is, I think, in the […]

Discussion of effects of growth mindset: Let’s not demand unrealistic effect sizes.

Shreeharsh Kelkar writes: As a regular reader of your blog, I wanted to ask you if you had taken a look at the recent debate about growth mindset [see earlier discussions here and here] that happened on theconversation.com. Here’s the first salvo by Brooke McNamara, and then the response by Carol Dweck herself. The debate […]

Narcolepsy Could Be ‘Sleeper Effect’ in Trump and Brexit Campaigns

Kevin Lewis sent along this example of what in social science is called the “ecological fallacy.” Below is a press release that I’ve changed in only a few places: UNDER EMBARGO UNTIL MARCH 8, 2018 AT 10 AM EST Media Contact: Public and Media Relations Manager Society for Personality and Social Psychology press@spsp.org Narcolepsy Could […]

Mouse Among the Cats

Colleen Flaherty asks: Do you get asked to peer review a lot? I’m guessing you do… This new very short paper says it’s not a crisis, though, since only the people who publish the most are getting asked to review a lot… The authors pose two solutions: either we need to “democratize” the system of […]

Researchers.one: A souped-up Arxiv with pre- and post-publication review

Harry Crane writes: I’m writing to call your attention to a new peer review and publication platform, called RESEARCHERS.ONE, that I have recently launched with Ryan Martin. The platform can be found at https://www.researchers.one. Given past discussions I’ve seen on your website, I think this new platform might interest you and your readers. We’d also […]

What if a big study is done and nobody reports it?

Paul Alper writes: Your blog often contains criticisms of articles which get too much publicity. Here is an instance of the obverse (inverse? reverse?) where a worthy publication dealing with a serious medical condition is virtually ignored. From Michael Joyce at the ever-reliable and informative Healthnewsreview.org: Prostate cancer screening: massive study gets minimal coverage. Why? […]

“Check out table 4.”

A colleague sent along this article and writes: Check out table 4. this is ERC funded research (the very best of European science get this money). OK, now I was curious, so I scrolled through to table 4. Here it is: Yup, it’s horrible. I don’t know that I’d call it cargo cult science at […]

“It’s Always Sunny in Correlationville: Stories in Science,” or, Science should not be a game of Botticelli

There often seems to be an attitude among scientists and journal editors that, if a research team has gone to the trouble of ensuring rigor in some part of their study (whether in the design, the data collection, or the analysis, but typically rigor is associated with “p less than .05” and some random assignment […]