Skip to content
Archive of posts filed under the Decision Theory category.

Using partial pooling when preparing data for machine learning applications

Geoffrey Simmons writes: I reached out to John Mount/Nina Zumel over at Win Vector with a suggestion for their vtreat package, which automates many common challenges in preparing data for machine learning applications. The default behavior for impact coding high-cardinality variables had been a naive bayes approach, which I found to be problematic due its multi-modal output (assigning […]

It’s all about Hurricane Andrew: Do patterns in post-disaster donations demonstrate egotism?

Jim Windle points to this post discussing a paper by Jesse Chandler, Tiffany M. Griffin, and Nicholas Sorensen, “In the ‘I’ of the Storm: Shared Initials Increase Disaster Donations.” I took a quick look and didn’t notice anything clearly wrong with the paper, but there did seem to be some opportunities for forking paths, in […]

The Millennium Villages Project: a retrospective, observational, endline evaluation

Shira Mitchell et al. write (preprint version here if that link doesn’t work): The Millennium Villages Project (MVP) was a 10 year, multisector, rural development project, initiated in 2005, operating across ten sites in ten sub-Saharan African countries to achieve the Millennium Development Goals (MDGs). . . . In this endline evaluation of the MVP, […]

Don’t define reproducibility based on p-values

Lizzie Wolkovich writes: I just got asked to comment on this article [“Genotypic variability enhances the reproducibility of an ecological study,” by Alexandru Milcu et al. ]—I have yet to have time to fully sort out their stats but the first thing that hit me about it was they seem to be suggesting a way […]

A possible defense of cargo cult science?

Someone writes: I’ve been a follower of your blog and your continual coverage of “cargo cult science”. Since this type of science tends to be more influential and common than the (idealized) non-“cargo cult” stuff, I’ve been trying to find ways of reassuring myself that this type of science isn’t a bad thing (because if […]

This one’s important: How to better analyze cancer drug trials using multilevel models.

Paul Alper points us to this news article, “Cancer Conundrum—Too Many Drug Trials, Too Few Patients,” by Gina Kolata, who writes: With the arrival of two revolutionary treatment strategies, immunotherapy and personalized medicine, cancer researchers have found new hope — and a problem that is perhaps unprecedented in medical research. There are too many experimental […]

Heuristics and Biases? Laplace was there, 200 years ago.

In an article entitled Laplace’s Theories of Cognitive Illusions, Heuristics, and Biases, Josh “hot hand” Miller and I write: In his book from the early 1800s, Essai Philosophique sur les Probabilités, the mathematician Pierre-Simon de Laplace anticipated many ideas developed in the 1970s in cognitive psychology and behavioral economics, explaining human tendencies to deviate from […]

Bayesian inference for A/B testing: Lauren Kennedy and I speak at the NYC Women in Machine Learning and Data Science meetup tomorrow (Tues 27 Mar) 7pm

Here it is: Bayesian inference for A/B testing Andrew Gelman, Department of Statistics and Department of Political Science, Columbia University Lauren Kennedy, Columbia Population Research Center, Columbia University Suppose we want to use empirical data to compare two or more decisions or treatment options. Classical statistical methods based on statistical significance and p-values break down […]

Spatial patterns in crime: Where’s he gonna strike next?

Wouter Steenbeek writes: I am a criminologist and mostly do spatial analyses of crime patterns: where does crime occur and why in these neighborhoods / at these locations, and so on. Currently, I am thinking about offender decision-making behavior, specifically his ‘location choice’ of where to offend. Hey, how about criminologists instead of looking to […]

Request for a cat picture

Could someone please send me a photo (that I’d have permission to share on this blog) that connects a cat to “heuristics and biases” or “behavioral economics”? Thanks. P.S. Javier Benítez points us to this page of free stock photos of cats. Cool! Still, if anyone has anything particularly appropriate to the topic above, just […]

“The problem of infra-marginality in outcome tests for discrimination”

Camelia Simoiu, Sam Corbett-Davies, and Sharad Goel write: Outcome tests are a popular method for detecting bias in lending, hiring, and policing decisions. These tests operate by comparing the success rate of decisions across groups. For example, if loans made to minority applicants are observed to be repaid more often than loans made to whites, […]

What We Talk About When We Talk About Bias

Shira Mitchell wrote: I gave a talk today at Mathematica about NHST in low power settings (Type M/S errors). It was fun and the discussion was great. One thing that came up is bias from doing some kind of regularization/shrinkage/partial-pooling versus selection bias (confounding, nonrandom samples, etc). One difference (I think?) is that the first […]

Gaydar and the fallacy of objective measurement

Greggor Mattson, Dan Simpson, and I wrote this paper, which begins: Recent media coverage of studies about “gaydar,” the supposed ability to detect another’s sexual orientation through visual cues, reveal problems in which the ideals of scientific precision strip the context from intrinsically social phenomena. This fallacy of objective measurement, as we term it, leads […]

“Like a harbor clotted with sunken vessels”

After writing this post on an error in one of my published papers, I got to thinking about the general problem of mistakes in the scientific literature. Retraction is not a serious solution to the problem. And there are lots of people out there who simply refuse to admit, let alone correct, their published errors: […]

Who’s afraid of prediction markets? (Hanson vs. Thicke)

In a post entitled, “Compare Institutions To Institutions, Not To Perfection,” Robin Hanson slams a recent paper by Michael Thicke that criticizes prediction markets. Hanson summarizes: Unfortunately many responses to reform proposals fit the above pattern: reject the reform because it isn’t as good as perfection, ignoring the fact that the status quo is nothing […]

I’ll use this line in my talk this Wednesday at the Society for Research on Educational Effectivness

I had a conversation with a policy analyst about the design of studies for program evaluation—the post is scheduled to appear in a few months—and he expressed some frustration: The idea of evidence based policy has put a gun to our heads as researchers to give binary responses with absolute confidence to a question that […]

The graphs tell the story. Now I want to fit this story into a bigger framework so it all makes sense again.

Paul Alper points us to these graphs: Pretty stunning. I mean, really stunning. Why are we just hearing about this now, given that the pattern is a decade old? And what’s this: “Data for the U.S. ends in 2007”? Huh? Also, it’s surprising how high the rates were for Japan, Italy, and Germany in the […]

Let’s face it, I know nothing about spies.

I saw this news article: Multiple federal agencies investigated claims that former Indiana basketball coach Bobby Knight groped and verbally sexually harassed several female employees when he gave a speech at the National Geospatial-Intelligence Agency in July 2015, according to newly-released documents. . . . he had slapped a “senior woman … on [her] butt” […]

Bob likes the big audience

In response to a colleague who was a bit scared of posting some work up on the internet for all to see, Bob Carpenter writes: I like the big audience for two reasons related to computer science principles. The first benefit is the same reason it’s scary. The big audience is likely to find flaws. […]

Of rabbits and cannons

When does it make sense to shoot a rabbit with a cannon? I was reminded of this question recently when I happened to come across this exchange in the comments section from a couple years ago, in the context of the finding patterns in the frequencies of births on different days: Rahul: Yes, inverting a […]