Skip to content
Archive of posts filed under the Miscellaneous Science category.

What is the role of qualitative methods in addressing issues of replicability, reproducibility, and rigor?

Kara Weisman writes: I’m a PhD student in psychology, and I attended your talk at the Stanford Graduate School of Business earlier this year. I’m writing to ask you about something I remember you discussing at that talk: The possible role of qualitative methods in addressing issues of replicability, reproducibility, and rigor. In particular, I […]

Global shifts in the phenological synchrony of species interactions over recent decades

Heather Kharouba et al. write: Phenological responses to climate change (e.g., earlier leaf-out or egg hatch date) are now well documented and clearly linked to rising temperatures in recent decades. Such shifts in the phenologies of interacting species may lead to shifts in their synchrony, with cascading community and ecosystem consequences . . . We […]

A style of argument can be effective in an intellectual backwater but fail in the big leagues—but maybe it’s a good thing to have these different research communities

Following on a post on Tom Wolfe’s evolution-denial trolling, Thanatos Savehn pointed to this obituary, “Jerry A. Fodor, Philosopher Who Plumbed the Mind’s Depths, Dies at 82,” which had lots of interesting items, including this: “We think that what is needed,” they wrote, “is to cut the tree at its roots: to show that Darwin’s […]

Ambiguities with the supposed non-replication of ego depletion

Baruch Eitam writes: I am teaching a seminar for graduate students in the social track and I decided to dedicate the first 4-6 classes to understanding the methodological crises in psychology, its reasons and some proposed solutions. In one of the classes I had the students read this paper which reports an attempt to reproduce […]

“Peeriodicals”: A new system of virtual journals

Brandon Stell writes: The PubPeer Foundation will soon be launching a new scientific discussion project: “Peeriodicals”. As you will discover by following the link below, Peeriodicals are virtual journals with you as Editor-in-chief. I love this idea, in part because it reminds me of my proposal to replace journals with recommender systems. The only post […]

The statistical significance filter leads to overoptimistic expectations of replicability

Shravan Vasishth, Daniela Mertzen, Lena Jäger, et al. write: Treating a result as publishable just because the p-value is less than 0.05 leads to overoptimistic expectations of replicability. These overoptimistic expectations arise due to Type M(agnitude) error: when underpowered studies yield significant results, effect size estimates are guaranteed to be exaggerated and noisy. These effects […]

How to think about research, and research criticism, and research criticism criticism, and research criticism criticism criticism?

Some people pointed me to this article, “Issues with data and analyses: Errors, underlying themes, and potential solutions,” by Andrew Brown, Kathryn Kaiser, and David Allison. They discuss “why focusing on errors [in science] is important,” “underlying themes of errors and their contributing factors, “the prevalence and consequences of errors,” and “how to improve conditions […]

Evaluating Sigmund Freud: Should we compare him to biologists or economists?

This post is about how we should think about Freud, not about how we should think about biology or economics. So. There’s this whole thing about Sigmund Freud being a bad scientist. Or maybe I should say a bad person and a terrible scientist. The “bad person” thing isn’t so relevant, but the “terrible scientist” […]

What killed alchemy?

Here’s the answer according to David Wootton’s 2015 book, “The invention of science: a new history of the scientific revolution” (sent to me by Javier Benitez): What killed alchemy was the insistence that experiments must be openly reported in publications which presented a clear account of what had happened, and they must then be replicated, […]

Why is the replication crisis centered on social psychology?

We had a post on this a couple years ago, but the topic came up again, and here are my latest thoughts. Psychology has several features that contribute to the replication crisis: – Psychology is a relatively open and uncompetitive field (compared for example to biology). Many researchers will share their data. – Psychology is […]

A model for scientific research programmes that include both “exploratory phenomenon-driven research” and “theory-testing science”

John Christie points us to an article by Klaus Fiedler, What Constitutes Strong Psychological Science? The (Neglected) Role of Diagnosticity and A Priori Theorizing, which begins: A Bayesian perspective on Ioannidis’s (2005) memorable statement that “Most Published Research Findings Are False” suggests a seemingly inescapable trade-off: It appears as if research hypotheses are based either […]

Carol Nickerson investigates an unfounded claim of “17 replications”

Carol Nickerson sends along this report in which she carefully looks into the claim that the effect of power posing on feelings of power has replicated 17 times. Also relevant to the discussion is this post from a few months ago by Joe Simmons, Leif Nelson, and Uri Simonsohn. I am writing about this because […]

Tools for detecting junk science? Transparency is the key.

In an article to appear in the journal Child Development, “Distinguishing polemic from commentary in science,” physicist David Grimes and psychologist Dorothy Bishop write: Exposure to nonionizing radiation used in wireless communication remains a contentious topic in the public mind—while the overwhelming scientific evidence to date suggests that microwave and radio frequencies used in modern […]

The all-important distinction between truth and evidence

Yesterday we discussed a sad but all-too-familiar story of a little research project that got published and hyped beyond recognition. The published paper was called, “The more you play, the more aggressive you become: A long-term experimental study of cumulative violent video game effects on hostile expectations and aggressive behavior,” but actually that title was […]

More bad news in the scientific literature: A 3-day study is called “long term,” and nobody even seems to notice the problem. Whassup with that??

Someone pointed me to this article, “The more you play, the more aggressive you become: A long-term experimental study of cumulative violent video game effects on hostile expectations and aggressive behavior,” by Youssef Hasan, Laurent Bègue, Michael Scharkow, and Brad Bushman. My correspondent was suspicious of the error bars in Figure 1. I actually think […]

Yet another IRB horror story

The IRB (institutional review board) is this weird bureaucracy, often staffed by helpful and well-meaning people but generally out of control, as it operates on an if-it’s-not-allowed-it’s-forbidden principle. As an example, Jonathan Falk points us to this Kafkaesque story from Scott Alexander, which ends up like this: Faced with submitting twenty-seven new pieces of paperwork […]

“It’s not just that the emperor has no clothes, it’s more like the emperor has been standing in the public square for fifteen years screaming, I’m naked! I’m naked! Look at me! And the scientific establishment is like, Wow, what a beautiful outfit.”

Somebody pointed Nick Brown to another paper by notorious eating behavior researcher Brian Wansink. Here’s Brown: I have that one in my collection of PDFs. I see I downloaded it on January 7, 2017, which was 3 days before our preprint went live. Probably I skimmed it and didn’t pay much further attention. I don’t […]

Gaydar and the fallacy of objective measurement

Greggor Mattson, Dan Simpson, and I wrote this paper, which begins: Recent media coverage of studies about “gaydar,” the supposed ability to detect another’s sexual orientation through visual cues, reveal problems in which the ideals of scientific precision strip the context from intrinsically social phenomena. This fallacy of objective measurement, as we term it, leads […]

Reasons for an optimistic take on science: there are not “growing problems with research and publication practices.” Rather, there have been, and continue to be, huge problems with research and publication practices, but we’ve made progress in recognizing these problems.

Javier Benitez points us to an article by Daniele Fanelli, “Is science really facing a reproducibility crisis, and do we need it to?”, published in the Proceedings of the National Academy of Sciences, which begins: Efforts to improve the reproducibility and integrity of science are typically justified by a narrative of crisis, according to which […]

A more formal take on the multiverse

You’ve heard of multiverse analysis, which is an attempt to map out the garden of forking paths. Others are interested in this topic too. Carol Nickerson pointed me to this paper by Jan Wacker with a more formal version of the multiverse idea.