Erik van Zwet writes: I saw you re-posted your Bayes-solves-multiple-testing demo. Thanks for linking to my paper in the PPS! I think it would help people’s understanding if you explicitly made the connection with your observation that Bayesians are frequentists: … Continue reading
Pascal Jordan writes: I stumbled upon a paper which I think you might find interesting to read or discuss: “Ignored evident multiplicity harms replicability—adjusting for it offers a remedy” from Yoav Zeevi, Sofi Astashenko, Yoav Benjamini. It deals with the … Continue reading
Kevin Lewis points us to this article by Joachim Vosgerau, Uri Simonsohn, Leif Nelson, and Joseph Simmons, which begins: Several researchers have relied on, or advocated for, internal meta-analysis, which involves statistically aggregating multiple studies in a paper . . … Continue reading
I’m rerunning this one from 2016 because it came up at work recently, and I think the general topic is as important as it’s ever been. flat priors consistently give bad inferences. Or, to put it another way, the routine … Continue reading
Kleber Neves writes: I’ve been a long-time reader of your blog, eventually becoming more involved with the “replication crisis” and such (currently, I work with the Brazilian Reproducibility Initiative). Anyway, as I’m now going deeper into statistics, I feel like … Continue reading
Justin Chumbley writes: I have mused on drafting a simple paper inspired by your paper “Why we (usually) don’t have to worry about multiple comparisons”. The initial idea is simply to revisit frequentist “weak FWER” or “omnibus tests” (which assume … Continue reading
Mark White writes: I have a question regarding using multilevel models for multiple comparisons, per your 2012 paper and many blog posts. I am in a situation where I do randomized experiments, and I have a lot of additional demographic … Continue reading
[cat picture] The following email came in: I’m in a PhD program (poli sci) with a heavy emphasis on methods. One thing that my statistics courses emphasize, but that doesn’t get much attention in my poli sci courses, is the … Continue reading
I promised I wouldn’t do any new blogging until January but I’m here at this conference and someone asked me a question about the above slide from my talk. The point of the story in that slide is that flat … Continue reading
Exhibit A: [2012] Why we (usually) don’t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness 5, 189-211. (Andrew Gelman, Jennifer Hill, and Masanao Yajima) Exhibit B: The garden of forking paths: Why multiple comparisons can be … Continue reading
Joe Northrup writes: I have a question about correcting for multiple comparisons in a Bayesian regression model. I believe I understand the argument in your 2012 paper in Journal of Research on Educational Effectiveness that when you have a hierarchical … Continue reading
Yarden Katz writes: I’m probably not the first to point this out, but just in case, you might be interested in this article by T. Florian Jaeger, Daniel Pontillo, and Peter Graff on a statistical dispute [regarding the claim, “Phonemic … Continue reading
John Transue sent it in with the following thoughtful comment: I’d imagine you’ve already received this, but just in case, here’s a cartoon you’d like. At first blush it seems to go against your advice (more nuanced than what I’m … Continue reading
Bill Harris writes: I’ve read your paper and presentation showing why you don’t usually worry about multiple comparisons. I see how that applies when you are comparing results across multiple settings (states, etc.). Does the same principle hold when you … Continue reading
Nick Allum writes: Continue reading
Monday 2 Nov, 5-6:30pm at the Methodology Institute, LSE. No link to the seminar on the webpage, so I’ll give you the information here: Why we (usually) don’t worry about multiple comparisons Applied researchers often find themselves making statistical inferences … Continue reading
Hamdan Yousuf writes: I was reading your Kanazawa letter to the editor and I was interested in your discussion of multiple comparisons. This might be an elementary issue but I don’t quite understand when the issue of multiple comparisons arises, … Continue reading
Here’s the paper (with Jennifer and Masanao), and here’s the abstract: The problem of multiple comparisons can disappear when viewed from a Bayesian perspective. We propose building multilevel models in the settings where multiple comparisons arise. These address the multiple … Continue reading
I told Phil Stark about this paper on Type S error rates and this paper on statistical challenges in estimating small effects, and he replied with these references on Type S errors and multiple comparisons: Benjamini, Y. and Stark, P.B., … Continue reading
Statisticians often get worried about multiple comparisons and have various procedures for adjusting p-values. It’s only very rarely come up in my own research, though. Here’s a presentation explaining why, from a workshop of educational researchers to which I was … Continue reading