Skip to content
Archive of posts filed under the Decision Theory category.

“Pointwise mutual information as test statistics”

Christian Bartels writes: Most of us will probably agree that making good decisions under uncertainty based on limited data is highly important but remains challenging. We have decision theory that provides a framework to reduce risks of decisions under uncertainty with typical frequentist test statistics being examples for controlling errors in absence of prior knowledge. […]

Replin’ ain’t easy: My very first preregistration

I’m doing my first preregistered replication. And it’s a lot of work! We’ve been discussing this for awhile—here’s something I published in 2013 in response to proposals by James Moneghan and by Macartan Humphreys, Raul Sanchez de la Sierra, and Peter van der Windt for preregistration in political science, here’s a blog discussion (“Preregistration: what’s […]

About that claim that police are less likely to shoot blacks than whites

Josh Miller writes: Did you see this splashy NYT headline, “Surprising New Evidence Shows Bias in Police Use of Force but Not in Shootings”? It’s actually looks like a cool study overall, with granular data, and a ton of leg work, and rich set of results that extend beyond the attention grabbing headline that is […]

Of polls and prediction markets: More on #BrexitFail

David “Xbox poll” Rothschild and I wrote an article for Slate on how political prediction markets can get things wrong. The short story is that in settings where direct information is not easily available (for example, in elections where polls are not viewed as trustworthy forecasts, whether because of problems in polling or anticipated volatility […]

“Most notably, the vast majority of Americans support criminalizing data fraud, and many also believe the offense deserves a sentence of incarceration.”

Justin Pickett sends along this paper he wrote with Sean Roche: Data fraud and selective reporting both present serious threats to the credibility of science. However, there remains considerable disagreement among scientists about how best to sanction data fraud, and about the ethicality of selective reporting. OK, let’s move away from asking scientists. Let’s ask […]

On deck this week

Mon: “Most notably, the vast majority of Americans support criminalizing data fraud, and many also believe the offense deserves a sentence of incarceration.” Tues: Some insider stuff on the Stan refactor Wed: I know you guys think I have no filter, but . . . Thurs: Bigmilk strikes again Fri: “Pointwise mutual information as test […]

Causal and predictive inference in policy research

Todd Rogers pointed me to a paper by Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ziad Obermeyer that begins: Empirical policy research often focuses on causal inference. Since policy choices seem to depend on understanding the counterfactual—what happens with and without a policy—this tight link of causality and policy seems natural. While this link holds […]

On deck this week

Mon: Americans (used to) love world government Tues: “Positive Results Are Better for Your Career” Wed: “I would like to share some sad stories from economics related to these issues” Thurs: Happiness formulas Fri: “Participants reported being hungrier when they walked into the café (mean = 7.38, SD = 2.20) than when they walked out […]

Informative priors for treatment effects

Biostatistician Garnett McMillan writes: A PI recently completed a randomized trial where the experimental treatment showed a large, but not quite statistically significant (p=0.08) improvement over placebo. The investigators wanted to know how many additional subjects would be needed to achieve significance. This is a common question, which is very hard to answer for non-statistical […]

On deck this week

Mon: How is Brexit different than Texit, Quexit, or Scotxit? Tues: Should this paper in Psychological Science be retracted? The data do not conclusively demonstrate the claim, nor do they provide strong evidence in favor. The data are, however, consistent with the claim (as well as being consistent with no effect) Wed: Individual and aggregate […]

On deck this week

Mon: Clarke’s Law: Any sufficiently crappy research is indistinguishable from fraud Tues: Reduced-dimensionality parameterizations for linear models with interactions Wed: Time-reversal heuristic as randomization, and p < .05 as conflict of interest declaration Thurs: It comes down to reality and it’s fine with me cause I’ve let it slide Fri: Can a census-tract-level regression analysis […]

The answer is the Edlin factor

Garnett McMillan writes: You have argued about the pervasive role of the Garden of Forking Paths in published research. Given this influence, do you think that it is sensible to use published research to inform priors in new studies? My reply: Yes, I think you can use published research but in doing so you should […]

On deck this week

Mon: They threatened to sue Mike Spagat but that’s not shutting him up Tues: “Smaller Share of Women Ages 65 and Older Are Living Alone,” before and after age adjusment Wed: Objects of the class “Pauline Kael” Thurs: research-lies-allegations-windpipe-surgery Fri: Hey—here’s a tip from the biology literature: If your correlation is .02, try binning your […]

On deck this week

Social problems with a paper in Social Problems Donald Trump and Joe McCarthy “What is a good, convincing example in which p-values are useful?” “How One Study Produced a Bunch of Untrue Headlines About Tattoos Strengthening Your Immune System” No, I’m not convinced by this one either. How to design a survey so that Mister […]

On deck this week

Mon: All that really important statistics stuff that isn’t in the statistics textbooks Tues: Who marries whom? Wed: Gray graphs look pretty Thurs: Freak Punts on Leicester Bet Fri: Who falls for the education reform hype? Sat: Taking responsibility for your statistical conclusions: You must decide what variation to compare to. Sun: Researchers demonstrate new […]

On deck this week

Mon: Splitsville for Thiel and Kasparov? Tues: Here’s something I know nothing about Wed: The “power pose” of the 6th century B.C. Thurs: “99.60% for women and 99.58% for men, P < 0.05.” Fri: Stan on the beach Sat: Michael Lacour vs John Bargh and Amy Cuddy Sun: Should he major in political science and […]

Is fraac Scott Adams?

tl;dr: If you value your time, don’t read this post.

On deck this week

Birthdays, baseball, zombies, luxury . . . and fraac!

Peer review abuse flashback

Our recent discussion of the problems with peer review reminded me of this amusing/horrifying story from a few years ago, when some researchers noticed a data coding error in a published paper Once it was noticed, the error was obvious: But the authors of the original paper had that never-back-down attitude. So instead of thanking […]

Big Belly Roti on Amsterdam Ave and 123 St

Josh “hot hand” Miller was in town and a bunch of us went to this new Caribbean place around the corner. It was good! P.S. The other hot hand guy, Sanjurjo, is not in town.