Skip to content
Archive of posts filed under the Miscellaneous Science category.

Post-publication peer review: who’s qualified?

Gabriel Power writes: I don’t recall that you addressed this point in your posts on post-publication peer review [for example, here and here — ed.]. Who would be allowed to post reviews of a paper? Anyone? Only researchers? Only experts? Science is not a democracy. A study is not valid because a majority of people […]

What to do when your measured outcome doesn’t quite line up with what you’re interested in?

Matthew Poes writes: I’m writing a research memo discussing the importance of precisely aligning the outcome measures to the intervention activities. I’m making the point that an evaluation of the outcomes for a given intervention may net null results for many reasons, one of which could simply be that you are looking in the wrong […]

Mouse Among the Cats

Colleen Flaherty asks: Do you get asked to peer review a lot? I’m guessing you do… This new very short paper says it’s not a crisis, though, since only the people who publish the most are getting asked to review a lot… The authors pose two solutions: either we need to “democratize” the system of […]

Researchers.one: A souped-up Arxiv with pre- and post-publication review

Harry Crane writes: I’m writing to call your attention to a new peer review and publication platform, called RESEARCHERS.ONE, that I have recently launched with Ryan Martin. The platform can be found at https://www.researchers.one. Given past discussions I’ve seen on your website, I think this new platform might interest you and your readers. We’d also […]

“It’s Always Sunny in Correlationville: Stories in Science,” or, Science should not be a game of Botticelli

There often seems to be an attitude among scientists and journal editors that, if a research team has gone to the trouble of ensuring rigor in some part of their study (whether in the design, the data collection, or the analysis, but typically rigor is associated with “p less than .05” and some random assignment […]

Isaac Newton : Alchemy :: Michael Jordan : Golf

Not realizing the domain-specificity of their successes.

Hey—take this psychological science replication quiz!

Rob Wilbin writes: I made this quiz where people try to guess ahead of time which results will replicate and which won’t in order to give then a more nuanced understanding of replication issues in psych. Based on this week’s Nature replication paper. It includes quotes and p-values from the original study if people want […]

“Identification of and correction for publication bias,” and another discussion of how forking paths is not the same thing as file drawer

Max Kasy and Isaiah Andrews sent along this paper, which begins: Some empirical results are more likely to be published than others. Such selective publication leads to biased estimates and distorted inference. This paper proposes two approaches for identifying the conditional probability of publication as a function of a study’s results, the first based on […]

Problems in a published article on food security in the Lower Mekong Basin

John Williams points us to this article, “Designing river flows to improve food security futures in the Lower Mekong Basin,” by John Sabo et al., featured in the journal Science. Williams writes: The article exhibits multiple forking paths, a lack of theory, and abundant jargon. It is also very carelessly written and reviewed. For example, […]

The competing narratives of scientific revolution

Back when we were reading Karl Popper’s Logic of Scientific Discovery and Thomas Kuhn’s Structure of Scientific Revolutions, who would’ve thought that we’d be living through a scientific revolution ourselves? Scientific revolutions occur on all scales, but here let’s talk about some of the biggies: 1850-1950: Darwinian revolution in biology, changed how we think about […]

Let’s get hysterical

Following up on our discussion of hysteresis in the scientific community, Nick Brown points us to this article from 2014, “Excellence by Nonsense: The Competition for Publications in Modern Science,” by Mathias Binswanger, who writes: To ensure the efficient use of scarce funds, the government forces universities and professors, together with their academic staff, to […]

How feminism has made me a better scientist

Feminism is not a branch of science. It is not a set of testable propositions about the observable world, nor is it any single research method. From my own perspective, feminism is a political movement associated with successes such as votes for women, setbacks such as the failed Equal Rights Amendment, and continuing struggles in […]

Some thoughts after reading “Bad Blood: Secrets and Lies in a Silicon Valley Startup”

I just read the above-titled John Carreyrou book, and it’s as excellent as everyone says it is. I suppose it’s the mark of any compelling story that it will bring to mind other things you’ve been thinking about, and in this case I saw many connections between the story of Theranos—a company that raised billions […]

Let’s be open about the evidence for the benefits of open science

A reader who wishes to remain anonymous writes: I would be curious to hear your thoughts on is motivated reasoning among open science advocates. In particular, I’ve noticed that papers arguing for open practices have seriously bad/nonexistent causal identification strategies. Examples: Kidwell et al. 2017, Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method […]

Response to Rafa: Why I don’t think ROC [receiver operating characteristic] works as a model for science

Someone pointed me to this post from a few years ago where Rafael Irizarry argues that scientific “pessimists” such as myself are, at least in some fields, “missing a critical point: that in practice, there is an inverse relationship between increasing rates of true discoveries and decreasing rates of false discoveries and that true discoveries […]

The replication crisis and the political process

Jackson Monroe writes: I thought you might be interested in an article [by Dan McLaughlin] in NRO that discusses the replication crisis as part of a broadside against all public health research and social science. It seemed as though the author might be twisting the nature of the replication crisis toward his partisan ends, but […]

Revisiting “Is the scientific paper a fraud?”

Javier Benitez points us to this article from 2014 by Susan Howitt and Anna Wilson, which has subtitle, “The way textbooks and scientific research articles are being used to teach undergraduate students could convey a misleading image of scientific research,” and begins: In 1963, Peter Medawar gave a talk, Is the scientific paper a fraud?, […]

How to think about an accelerating string of research successes?

While reading this post by Seth Frey on famous scientists who couldn’t let go of bad ideas, I followed a link to this post by David Gorski from 2010 entitled, “Luc Montagnier: The Nobel disease strikes again.” The quick story is that Montagnier endorsed some dubious theories. Here’s Gorski: He only won the Nobel Prize […]

Advice on soft skills for academics

Julia Hirschberg sent this along to the natural language processing mailing list at Columbia: here are some slides from last spring’s CRA-W Grad Cohort and previous years that might be of interest. all sorts of topics such as interviewing, building confidence, finding a thesis topic, preparing your thesis proposal, publishing, entrepreneurialism, and a very interesting […]

Journals and refereeing: toward a new equilibrium

As we’ve discussed before (see also here), one of the difficulties of moving from our current system of review of scientific journal articles, to a new model of post-publication review, is that any major change seems likely to break the current “gift economy” system in which thousands of scientists put in millions of hours providing […]