Exhibit A:  Why we (usually) don’t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness 5, 189-211. (Andrew Gelman, Jennifer Hill, and Masanao Yajima) Exhibit B: The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis […]
In one of life’s horrible ironies, I wrote a paper “Why we (usually) don’t have to worry about multiple comparisons” but now I spend lots of time worrying about multiple comparisons
Shravan Vasishth writes: I saw on your blog post that you listed aggregation as one of the desirable things to do. Do you agree with the following argument? I want to point out a problem with repeated measures ANOVA in talk: In a planned experiment, say a 2×2 design, when we do a repeated measures […]
David Rothschild (coauthor of the Xbox study, the Mythical Swing Voter paper, and of course the notorious Aapor note) will be speaking Friday 10 Oct in the Economics and Big Data meetup in NYC. His title: “How to disrupt the multi-billion dollar survey research industry: information aggregation using non-representative polling data.” Should be fun! P.P.S. […]
In a news article regarding difficulties in using panel surveys to measure the unemployment rate, David Leonhardt writes: The main factor is technology. It’s a major cause of today’s response-rate problems – but it’s also the solution. For decades, survey research has revolved around the telephone, and it’s worked very well. But Americans’ relationship with […]
Anti-Abortion Democrats, Jimmy Carter Republicans, and the Missing Leap Day Babies: Living with Uncertainty but Still Learning To learn about the human world, we should accept uncertainty and embrace variation. We illustrate this concept with various examples from our recent research (the above examples are with Yair Ghitza and Aki Vehtari) and discuss more generally […]
How does inference for next year’s data differ from inference for unobserved data from the current year?
Juliet Price writes: I recently came across your blog post from 2009 about how statistical analysis differs when analyzing an entire population rather than a sample. I understand the part about conceptualizing the problem as involving a stochastic data generating process, however, I have a query about the paragraph on ‘making predictions about future cases, […]
Kevin Van Horn sent me an email with the above title (ok, he wrote MRP, but it’s the same idea) and the following content: I’m working on a problem that at first seemed like a clear case where multilevel modeling would be useful. As I’ve dug into it I’ve found that it doesn’t quite fit […]
James Keirstead writes: I’m working on some regressions for UK cities and have a question about how to interpret regression coefficients. . . . In a typical regression, one would be working with data from a sample and so the standard errors on the coefficients can be interpreted as reflecting the uncertainty in the choice […]
Chris Skovron writes: A colleague sent the attached image from Indonesia. For whatever reason, it seems appropriate that Mr. P is a delicious salty snack with the tagline “good times.” Indeed. MRP has made the New York Times and Indonesian snack food. What more can we ask for?
Nate Delaney-Busch writes: I’m a PhD student of cognitive neuroscience at Tufts, and a question came recently with my colleagues about the difficulty of random sampling in cases of highly controlled stimulus sets, and I thought I would drop a line to see if you had any reading suggestions for us. Let’s say I wanted […]