It’s at 4pm; see here for details.

Statisticians talk a lot about what to do with your data. We can go further by considering what comes before data analysis: design of experiments and data collection. Here are some recommendations for design and data collection:

Recommendation 1. Consider measurements that address the underlying construct of interest.

The concepts of validity and reliability of measurement are well known in psychology but are often forgotten in experimental design and analysis. Often we see exposure or treatment measures and outcome measures that connect only indirectly to substantive research goals. This can be seen in the frequent disconnect between the title and abstract of a research paper, on one hand, and the actual experiment, on the other. A notorious example in psychology is a paper that referred in its title to a “long-term experimental study” that in fact was conducted for only three days.

Our recommendation goes in two directions. First, set up your design and data collection to measure what you want to learn about. If you are interested in long-term effects, conduct a long-term study if possible. Second, to the extent that it is not possible to take measurements that align with your inferential goals, be open about this gap and explicit about the theoretical assumptions or external information that you are using to support your more general conclusions.

Recommendation 2. When designing an experiment, consider realistic effect sizes.

There is a tendency to overestimate effect sizes when designing a study. Part of this is optimism and availability bias: it is natural for researchers who have thought hard about a particular effect to think that it will be important, to envision scenarios where the treatment will have large effects and not to think so much about cases where it will have no effect or where it will be overwhelmed by other possible influences on the outcome. In addition, past results will be much more likely to be published if they have reached a significance threshold, and this results in literature reviews that vastly overestimate effect sizes.

Overestimation of effect sizes leads to overconfidence in design, with researchers being satisfied by small sample size and sloppy measurements in a mistaken belief that the underlying effect is so large that it can be detected even with a very crude inference. And this causes three problems. First, it is a waste of resources to conduct an experiment that is so noisy that there is essentially no chance of learning anything useful, and this sort of work can crowd out the more careful sorts of studies that would be needed to detect realistic effect sizes. Second, a false expectation of high power creates a cycle of hype and disappointment that can discredit a field of research. Third, the process of overestimation can be self-perpetuating, with a noisy experiment being analyzed until apparently statistically-significant results appear, leading to another overestimate to add to the literature. These problems arise not just in statistical power analysis (where the goal is to design an experiment with a high probability of yielding a statistically significant result) but also applies to more general design analyses where inferences will be summarized by estimates and uncertainties.

Recommendation 3. Simulate your data collection and analysis on the computer first.

In the past, we have designed experiments and gathered data on the hope that the results would lead to insight and possible publication—but then the actual data would end up too noisy, and we would realize in retrospect that our study never really had a chance of answering the questions we wanted to ask. Such an experience is not a complete waste— we learn from our mistakes and can use them to design future studies—but we can often do better by preceding any data collection with a computer simulation.

Simulating a study can be more challenging than conducting a traditional power analysis. The simulation does not require any mathematical calculations; the challenge is the need to specify all aspects of the new study. For example, if the analysis will use regression on pre-treatment predictors, these must be simulated too, and the simulated model for the outcome should include the possibility of interactions.

Beyond the obvious benefit of revealing designs that look to be too noisy to detect main effects or interactions of interest, the construction of the simulation focuses our ideas by forcing us to make hard choices in assuming the structure and sizes of effects. In the simulation we can make assumptions about variation in measurement and in treatment effects, which can facilitate the first two recommendations above.

Recommendation 4. Design in light of analysis.

In his book, The Chess Mysteries of Sherlock Holmes, logician Raymond Smullyan (1979) wrote, “To know the past, one must first know the future.” The application of this principle to statistics is that design and data collection should be aligned with how you plan to analyze your data. As Lee Sechest put it, “The central issue is the validity of the inferences about the construct rather than the validity of the measure per se.”

One place this arises is in the collection of pre-treatment variables. If there is concern about imbalance between treatment and control groups in an observational study or an experiment with dropout, it is a good idea to think about such problems ahead of time and gather information on the participants to use in post-data adjustments. Along similar lines, it can make sense to recruit a broad range of participants and record information on them to facilitate generalizations from the data to larger populations of interest. A model to address problems of representativeness should include treatment interactions so that effects can vary by characteristics of the person and scenario.

In summary, we can most effectively learn from experiment if we think plan the design and data collection ahead of time, which involves: (1) using measurement that relates well to underlying constructs of interest, (2) considering realistic effect sizes and variation, (3) simulating experiments on the computer before collecting any data, and (4) keeping analysis plans in mind in the design stage.

The background on this short paper was that I was asked by Joel Huber from the Journal of Consumer Psychology to comment on an article by Michel Wedel and David Gal, Beyond statistical significance: Five principles for the new era of data analysis and reporting. Their recommendations were: (1) summarize evidence in a continuous way, (2) recognize that rejection of statistical model A should not be taken as evidence in favor of preferred alternative B, (3) use substantive theory to generalize from experimental data to the real world, (4) report all the data rather than choosing a single summary, (5) report all steps of data collection and analysis. (That’s my rephrasing; you can go to the Wedel and Gal article to see their full version.)

Emma Pierson sends along this statistics quiz (screen-shotted below—you can see the answers people gave to each question in the bar graphs):

Pierson writes:

In spite of the simplicity and ubiquity of the setup, people were very bad at this quiz—on every question, the majority answer was incorrect, and on the last two questions, performance was worse than random guessing. Of course, Twitter is a weird sample, but I also spoke to a number of genuine experts—professors of statistics, computer science, and economics at top universities—who found these questions unintuitive as well. I thought this might be of interest to your blog because you might be able to provide some useful intuition or diagnose why people’s intuitions lead them astray here.

I responded: I don’t quite understand your questions. From question 1, it look like “a”, “noise”, and “p” are vectors; I’m picturing length 100 because that’s what I typically do in simulation experiments for class.. But then in question 2, it looks like “a” is a scalar, but then the bias would be E(a_hat – a), not a_hat – a. And then the correlation you’re discussing is across multiple studies. In question 3, it again seems like “a” and “p” are vectors. I guess I understand questions 1 and 3 but not question 2. I agree that it’s disturbing that so many people get this wrong. I wonder whether the ambiguous specification is part of the problem. Or maybe I’m missing something?

Pierson responded:

Here’s a simulation of the setup I had in mind (hopefully a correct simulation—I don’t really use R these days, but I think you do):

a = rnorm(500000)
noise = rnorm(500000)
p = a + noise
d = data.frame(a, p)

# question 1
print(lm(a ~ p, data=d))
print(lm(p ~ a, data=d))

# question 2
a_hat = predict(lm(a ~ p, data=d), d)
print(cor(a_hat - a, a))
print(cor(a_hat - a, a_hat))

# question 3
print(cor(p - a, a))
print(cor(p - a, p))

Always hard to know what people on Twitter are thinking, but I don’t think (based on my conversations with people) that misunderstandings explain the wrong answers? People seemed to understand the setup pretty clearly, and I also didn’t get any confused questions about it on Twitter.

To which I replied: Oh, you were defining a_hat as the point prediction: I didn’t get that part. I agree that these things confuse people. It reminds me of that other false intuition people have, that if a is correlated with b, and b is correlated with c, then a must be correlated with c. People also seem to have the intuition that a can be correlated with b, even while b is uncorrelated with a. And then there’s a whole literature in cognitive psychology on super-additive probabilities (Pr(A) + Pr(not-A) seems to be greater than 1) or sub-additive probabilities (Pr(A) + Pr(not-A) seems to be less than 1). Lots of confusion around probability. It’s a jungle out there.

This is one of the motivations for the work of Gigerenzer etc. on reframing problems in terms of natural frequencies so as to avoid the confusingness of probability.

In our discussion of the statistician Carl Morris, Phil pointed to an interview of Morris by Jim Albert from 2014 which contained this passage:

In the spring of 1970, some years after our Stanford graduations, we talked one evening outside the statistics department at Stanford and decided to write a paper together. What should it be about? Brad [Efron] suggested, “Let’s work on Stein’s estimator.” Because so few understood it back then, and because we both admired Charles Stein so much for his genius and his humanity, we chose this topic, hoping we could honor him by showing that his estimator could work well with real data.

Stein already had proved remarkable theorems about the dominance of his shrinkage estimators over the sample mean vector, but there also needed to be a really convincing applied example. For that, we chose baseball batting average data because we not only could use the batting averages of the players early in the season, but because we also later could observe how those batters fared for the season’s remainder—a much longer period of time.

What struck me about this quote was that there was such a long delay between the theoretical work and “a really convincing applied example.” Also, the “applied example” was only kind of applied. Yeah, sure, it was a real data, and it addressed a real problem—assessing performance based on noisy information—but it was what might best be called a stylized data example.

Don’t get me wrong; I think stylized data examples are great. Here are some other instances of stylized data examples in statistics:
– The 8 schools
– The Minnesota radon survey
– The Bangladesh arsenic survey
– Forecasting the 1992 presidential election
– The speed-of-light measurements.

What do these and many other examples have in common, beside that my colleagues and I used them to demonstrate methods in our books?

They are all real data, they are all related to real applied problems (in education research, environmental hazards, political science, and physics) and real statistical problems (estimating causal effects, small-area estimation, decision making under uncertainty, hierarchical forecasting, model checking), and they’re all kind of artificial, typically using only a small amount of the relevant information for the problem at hand.

Still, I’ve found stylized data examples to be very helpful, perhaps for similar reasons as Efron and Morris:

1. The realness of the problem helps sustain our intuition and also to give a sense of real progress being made by new methods, in a way that is more understandable and convincing than, say, a reduction in mean squared error.

2. The data are real and so we can be surprised sometimes! This is related to the idea of good stories being immutable.

Indeed, sometimes researchers demonstrate their methods with stylized data examples and the result is not convincing. Here’s an example from a few years ago, where a colleague and I expressed skepticism about a certain method that had been demonstrated on two social-science examples. I was bothered by both examples, and indeed my problems with these examples gave me more understanding as to why I didn’t like the method. So the stylized data examples were useful here too, even if not the way the original author intended.

In section 2 of this article from 2014 I discussed different “ways of knowing” in statistics:

How do we decide to believe in the e↵ectiveness of a statistical method? Here are a few potential sources of evidence (I leave the list unnumbered so as not to imply any order of priority):

• Mathematical theory (e.g., coherence of inference or convergence)
• Computer simulations (e.g., demonstrating approximate coverage of interval estimates under some range of deviations from an assumed model)
• Solutions to toy problems (e.g., comparing the partial pooling estimate for the eight schools to the no pooling or complete pooling estimates)
• Improved performance on benchmark problems (e.g., getting better predictions for the Boston Housing Data)
• Cross-validation and external validation of predictions
• Success as recognized in a field of application (e.g., our estimates of the incumbency advantage in congressional elections)
• Success in the marketplace (under the theory that if people are willing to pay for something, it is likely to have something to offer)

None of these is enough on its own. Theory and simulations are only as good as their assumptions; results from toy problems and benchmarks don’t necessarily generalize to applications of interest; cross-validation and external validation can work for some sorts of predictions but not others; and subject-matter experts and paying customers can be fooled.

The very imperfections of each of these sorts of evidence gives a clue as to why it makes sense to care about all of them. We can’t know for sure so it makes sense to have many ways of knowing. . . .

For more thoughts on this topic, see this follow-up paper with Keith O’Rourke.

In the above list of bullet points I described the 8 schools as a “toy problem,” but now I’m more inclined to call it a stylized data example. “Toy” isn’t quite right; these are data from real students in real schools!

Let me also distinguish stylized data examples from numerical illustrations of a method that happen to use real data. Introductory statistics books are full of examples like that. You’re at the chapter on the t test or whatever and they demonstrate it with data from some experiment in the literature. “Real data,” yes, but not really a “real example” in that there’s no engagement with the applied context; the data are just there to show how the method works. In contrast, the intro stat book by Llaudet and Imai uses what I’d call real examples. Still with the edges smoothed, but what I’d call legit stylized data examples.

It’s my impression that the use of stylized data examples has been standard in statistics research and education for awhile. Not always, but often, enough so that it’s not a surprise to see them. The remark by Carl Morris in that interview makes me think that this represents a change, that things were different 50 years ago and before.

And I guess that’s right—there really has been a change. When I think of the first statistics course I took in college, the data were all either completely fake or they were numerical illustrations. And even Tukey’s classic EDA book from 1977 is full of who-cares examples like the monthly temperature series in Yuma, Arizona. At that point, Tukey had decades of experience with real problems and real data in all sorts of application areas—yet when writing one of his most enduring works, he went with the fake? Why? I think because that’s how it was done back then. You have your theory, you have your methods, and the point of the methods research article or book is to show how to do it, full stop. In the tradition of Snedecor and Cochran’s classic book on statistical methods. Different methods but the same general approach. But something changed, and maybe the 1970s was the pivotal period. Maybe the Steve Stiglers of the future can figure this one out.

From Active Statistics: Stories, Games, Problems, and Hands-on Demonstrations for Applied Regression and Causal Inference:

Consider the analogy of learning statistics to learning a new language. We tell our students that the class is particularly difficult because they’re learning two foreign languages—statistics and R—both of which can be challenging if you do not have a strong background in math and programming.

The challenge is to learn how to engage in and understand conversations, not merely memorize stock phrases and rules of grammar.