Statistics is associated with random numbers: normal distributions, probability distributions more generally, random sampling, randomized experimentation.

But I don’t think these are the most important parts of statistics.

I thought about this when rereading this post that I wrote awhile ago but happened to appear yesterday. Here’s the relevant bit:

In his self-experimentation, Seth lived the contradiction between the two tenets of evidence-based medicine: (1) Try everything, measure everything, record everything; and (2) Make general recommendations based on statistical evidence rather than anecdotes.

Seth’s ideas were extremely evidence-based in that they were based on data that he gathered himself or that people personally sent in to him, and he did use the statistical evidence of his self-measurements, but he did not put in much effort to reduce, control, or adjust for biases in his measurements, nor did he systematically gather data on multiple people.

I think that these are the most important parts of statistics:
(a) to reduce, control, or adjust for biases and variation in measurement, and
(b) to systematically gather data on multiple cases.
This all should be obvious, but I don’t think it comes out clearly in textbooks, including my own. We get distracted by the shiny mathematical objects.

And, yes, random sampling and randomized experimentation are important, as is statistical inference in all its mathematical glory—our BDA book is full of math—, but you want those sweet, sweet measurements as your starting point.

Last year we discussed the exciting new introductory social-science statistics textbook by Elena Llaudet and Kosuke Imai.

Since then, Llaudet has created a website with tons of materials for instructors.

This is the book that I plan to teach from, next time I teach introductory statistics. As it is, I recommend it as a reference for students in more advanced classes such as Applied Regression and Causal Inference, if they want a clean refresher from first principles.

The students in my class on applied regression and causal inference are required to contribute to a shared online document. Before each class, every student needs to enter something in the document, either a question on the reading/homework or a response to some other student’s question. Then in class we discuss some of the questions that came up.

This can work really well! For example, from today’s class:

Student #1: I am a bit confused on the formula from figure 6.4. The initial equation is y = 30 + 0.54x, but I am a bit confused on how we derive to the equivalent equation y = 63.9 + 0.54(x − 62.5)

Student #2: If you multiply it out: y = 63.9 + 0.54(x − 62.5) = 63.9 + 0.54x − 33.75 ≈ 30 + 0.54x. So you have the same equation in the end. But it’s transformed because an intercept where x = 0 doesn’t make a lot of practical sense in some cases (like height, where no one can be 0 inches tall). So we write that equation instead, centering it around the mean, to give more meaning to the intercept.

Sometimes lots of students ask the same or similar questions, and we definitely want to discuss live. For example:

Student #3: How exactly is the residual standard deviation calculated, and what does it mean conceptually? How can we understand it in comparison to standard error and standard deviation?

Student #4: I also had a question about this. I find the terms “standard error,” “standard deviation,” “standard deviation of the residual,” and “standard error of the true parameter values” to at times be used interchangeably, which can be very confusing. For instance, when I read the instructions for problem 1a., I assumed we were being asked to take the standard error, meaning the standard deviation divided by the square of n. Yet, in the book it makes clear we should use the sigma score produced by the stan_glm model. Is there a way for us to remember which concept is being invoked and when going forward, or is this something we just have to discern through context and our intuition?

Student #5: Yeah, I have the same questions here. It seems that the residual standard deviation is to describe the standard deviation of the error terms in the function.

Student #6: I also have this question : is uncertainty and standard error the same thing? And is MD_SD equal to the standard error that is sd/squared(n)? And using lm, is the uncertainty printed the standard deviation or error?

So I talked about standard deviation and standard error in class. It didn’t go well. I said how the standard error is the estimated standard deviation of a parameter estimate, I wrote some notation on the board, and I did some explaining. Nothing I said was wrong, but I don’t think it helped.

And, no kidding it didn’t help! Of course my explanation didn’t help. This was a topic that the students were already confused about. More talkety talk from me is not gonna solve that problem.

Here’s what I think I should’ve done. Instead of giving this definition and explanation, I should’ve just said that the definition and explanation are in the book, and I should’ve used the class time to go over an example. For example, just a few minutes earlier we’d discussed in class this fitted model predicting the incumbent candidate for president’s two-party vote share given percentage growth in per capita income in the year preceding the election, using all the elections after World War 2:

1948-2020:

            Median MAD_SD
(Intercept) 46.7    1.4
growth       2.8    0.6
Auxiliary parameter(s):
      Median MAD_SD
sigma 3.7 0.7

So I could’ve just gone through each of the sd’s in the display, along with sigma, and asked the students in pairs to explain to each other the meaning of each of these. Then the next thing would be to ask what would happen to these numbers if N were multiplied by 2, or by 4.

At this point, if students wanted to ask their questions on standard deviation and standard error, we could do it in the context of this example that they’ve just been talking about. We could also loop back to their homework assignment.

My mistake was to “explain” rather than to demonstrate. Explanation is a dead end. Demonstration is forward-looking and allows students to participate, which is absolutely necessary if you want them to learn something.

And if we want to follow up, we can go through the other fitted models we discussed, breaking the data into two parts:

1948-1988:

            Median MAD_SD
(Intercept) 44.8    2.7
growth       3.5    1.0
Auxiliary parameter(s):
      Median MAD_SD
sigma 4.5 1.2

1992-2020:

            Median MAD_SD
(Intercept) 48.4    1.5
growth       1.6    1.1
Auxiliary parameter(s):
      Median MAD_SD
sigma 2.8 0.8

I think this alternative way of teaching, using examples rather than explanation, would’ve worked better for three reasons:

1. The explanation’s still there in the book for students to read when they’re ready.

2. It should be easier to follow the idea in the context of a familiar example than in abstract framing in terms of estimates and sampling distributions.

3. Even if they still don’t fully understand the concept, in the meantime they may have learned a useful related skill which will move them forward.

We had a fun discussion the other day after my talk in London on statistics teaching.

One question that arose was how our classes should change in reaction to the new chatbots that can write essays and answer homework questions.

My first reaction is that the chatbots will be helpful, not so much for doing homework or checking answers, as there are lots of examples where the chatbot gives the worst of both worlds: a wrong answer but with a superficially convincing explanation, but rather for helping with coding. Students are always telling me they got stuck for hours and hours trying to figure out the code to make some graph or do some analysis, and indeed, even routine tasks such as graphing a scatterplot with fitted regression line or poststratification accounting for uncertainty can be tricky to code in R or Python or whatever, so if the chatbot can help with that, great! The point is not for the chatbot to give the answer but rather to think of it as a sort of super-google that will help you find the answers to frequently asked questions on the internet.

So, yeah, a chatbot, if used well, can help with your homework, especially if the problem is to make some graph or do some statistical analysis and you have to figure out what commands to use on the computer. On the downside, it can get in the way of doing homework if you use it as a replacement for thinking, for example by entering your homework problem to the chat window, getting a solution as output, and then editing that slightly before turning it in. Editing a chatbot response isn’t nothing—it’s a real skill to take something that might be wrong and rewrite it as necessary to make sense, indeed this is a skill that recipients of the Founders Award from the American Statistical Association seem to struggle with, and indeed it could be useful in life—but it’s not really what we want to be teaching in a statistics class, either. It’s a skill, it’s just not statistics.

So, how does the existence and ready availability of powerful chatbots change our statistics teaching? To start, we can’t really give take-home essay assignments. But I wasn’t giving take-home essay assignments, or take-home final exam problems, anyway. I’ll still give regular homework assignments, but I can’t assume that students will do them. One option is to require the solutions be hand-written (except for those that require code and computer-made graphs); even if all you do is copy over something that came off the computer, that itself could be a way to help learn.

The big thing, though, is to rethink the balance between students’ time spent in class and time spent at home. Traditionally in university classes, classroom time is for following along the lecture and for asking questions, and home time is for reading the book and doing homework. Now, I think students need to spending most of classroom time working on problems, as we can’t assume they’ll be doing that at home. From this perspective, home is the time for students to prepare for class (by reading the book and doing practice problems) and for them to review the material after.

That works for me—it’s pretty much how I’ve been trying to teach anyway in recent years—but I wonder if the balance of hours needs to change too. Currently I’m assuming students will spend 10 hours a week on a course: that’s 3 hours in the classroom (2 weekly classes of 1.5 hours each; I’d prefer 3 classes a week, each an hour long, but unfortunately it seems that this will never again be happening at Columbia—something to do with its U.S. News ranking strategy, perhaps?), 1 hour in a help session with the teaching assistant if that is available, and 6 hours at home, which perhaps is 2 hours going through the textbook and 4 hours on homework.

Anyway, I’m just wondering: if students learn by doing, and if they’re going to be doing the doing in class and not at home, then maybe we should have more classroom hours? For each course, 4 or 5 hours per week instead of just 3? I know this is a non-starter too: in recent years, classes have started moving from two 1.5-hour meetings per week to one 2-hour meeting. Students and teachers alike find it so much more convenient to have fewer classroom hours to schedule. But I’m really thinking that more live hours would be better. Not 4 or 5 hours of lecture; 4 or 5 hours of supervised problem solving. That’s where I think we should go, even if I have no idea how to get there.

David Kane sends along this article he just wrote with the above title and the following abstract:

A community is a collection of people who know and care about each other. The vast majority of college courses are not communities. This is especially true of statistics and data science courses, both because our classes are larger and because we are more likely to lecture. However, it is possible to create a community in your classroom. This article offers an idiosyncratic set of practices for creating community. I have used these techniques successfully in first and second semester statistics courses with enrollments ranging from 40 to 120. The key steps are knowing names, cold calling, classroom seating, a shallow learning curve, Study Halls, Recitations and rotating-one-on-one final project presentations.

There’s some overlap with the ideas in chapters 1 and 2 of my forthcoming book with Aki, “Active Statistics: Stories, Games, Problems, and Hands-on Demonstrations for Applied Regression and Causal Inference,” but Kane has a bunch of ideas that are different from (but complementary) to ours. I recommend our book (along with Rohan Alexander’s Telling Stories with Data, Elena Llaudet and Kosuke Imai’s Data Analysis for Social Science, our own, Regression and Other Stories); I also recommend Kane’s article. Kane, like Aki and me in our new book, focus on ways to get students actively involved in class. I hadn’t thought about this in terms of “community,” but that seems like a good way to frame it.

I’m sure there’s lots more out there; the above list of resource is focused on modern introductions to applied statistics and active ways to learn and teach this material.

Zach del Rosario writes:

I recently published work investigating longstanding probability errors in aircraft design. I’m actually a bit sad that I had to write this paper, as the error is as simple as

f(quantile(X)) ==not necessarily equal to== quantile(f(X))

with X a random variable and f an arbitrary function. Despite the simplicity of this error, this tacit assumption has been in use in aircraft design for at least the past 60 years (based on the revision history of the Code of Federal Regulations).

Since completing this work, I’ve begun a research pivot to try to understand how university training could lead to so many engineers missing such a simple issue related to probability / uncertainty. I’m particularly curious to hear the experiences of your other readers. At this stage in my career I’m looking for inspiration on how to frame what is ultimately an education / social science question: How widespread are these kinds of issues in engineering, and what can we do from an education perspective to address them?

I have no idea! It’s my general impression that probability is intuitive to only a small minority of people. Math is hard, and probability is even harder! On the other hand, there’s research by Gigerenzer et al. that suggests that people understand probabilistic concepts much better when they are framed in terms of frequencies. So maybe instead of asking about the quantile of a distribution, you could say, “Imagine you have 1000 airplanes. How many would have . . .”, or whatever, and get better intuition that way. I’m just guessing, though; maybe some readers have better ideas.

Brett Cooper writes:

I read your book Regression and Other Stories. As a beginner and community college student, I wonder if I may be able to ask you a couple of simple and clarifying questions.

1. Multiple Regression and Collinearity

I created 2 linear regression models using variables from a simple dataset. The first model is a simple linear regression model with a regressor X1 and a specific response variable Y. The other model is a multiple linear regression model which includes X1, X2, X3 and Y. The statistical summary for the simple linear regression model assigns a positive and statistically significant coefficient beta1 to X1 indicating a positive linear association between X1 and Y. However, the statistical summary for the multiple regression model shows that the coefficient beta1 for X1 has changed sign, changed magnitude and is not statistically significant anymore.
Did that happen because of the undesirable but often unavoidable effect of collinearity between X1 and one or both of the other two predictors X2 and X3?

Based on my understanding, it is to be expected that, even with zero multicollinearity, the regression coefficient associated with certain predictor X change, in magnitude or even sign, when the predictors X are considered together in a multiple linear regression model instead of separately in relation to the response variable Y. Is that correct?
However, is it “normal” for the coefficients’ sign and for the p-value for the same predictor X to change when switching to a different model? A change of coefficient sign for X would indicate an opposite behavior between X and Y in going from a simple to a multiple regression model…

— As a rule of thumb, before creating a multiple regression model involving Y,X1,X2,X3, would it recommended and useful to first create the simple regression models, i.e. Y =beta1*X1+beta0, Y =beta2*X2+beta0, Y=beta3*X3+beta0, and compute their regression coefficients? Or should we jump straight to the multivariate model Y= beta1*X1+beta2*X2+beta3*X3 and evaluate the regression coefficients at that point?

2. Multicollinearity

— Multicollinearity affects the interpretability (impacts the accuracy of the regression coefficients) of our model but not its predictive power. Multicollinearity can have different sources: it can originate from the data itself but also from the structure of the model. For example, model Y = beta1*X1 + beta2* X1^2 + beta3 * (X1*X2) has interdependent terms. Surprisingly, multicollinearity would NOT be present between X1 and the term X1^2 even if they are quadratically dependent…Is that correct? What about the interaction term (X1*X3) and the term X1? Or would multicollinearity be present and only be reduced if we mean center the variables X1, X2, X3?

— Correlation means linear dependence. Is multicollinearity only caused by the presence of linear dependence or do other types of dependence (curvilinear, etc.) between predictors also cause collinearity in the model?

3. Variable Transformation (feature scaling)

–Certain statistical models require their input predictors to be scaled before they can be used to build the model itself so the variables can all be on equal footing.
Some models, like decision trees, don’t require scaling at all. Scaling variables (linear or nonlinear scaling) is generally useful when the involved variables, the Xs and the Y, have very different ranges. My understanding is that predictor variables with large ranges would automatically receive large regression coefficients even if their relative importance is lower in comparison to other predictors. Is that correct and true for most models?

— In some cases, scaling seems optional and only improves the interpretability of the association between Y and the Xs (computed correlation may be tiny only and we can increase it by scaling the variables). That said, is the scaling of the predictors X and/or response variable Y necessary and critical for the creation of an accurate and correct simple or multiple linear regression model? Or does scaling only help with the interpretability of the regression coefficients?

My reply:

1a. When you add predictors to a model, you can expect the coefficients of the original predictors to change. Once they can change, yes, they can change sign: there’s nothing special about zero. As for statistical significance and p-values: sure, they can change too. One way to see this is to imagine N = 1 million: then even small changes in the coefficients will correspond to huge changes in p-values.

1b. Yes, when you fit a big model, I recommend fitting a series of little models to build up to it, and you can look at how the predictions of interest change as the model builds up.

2. I’m not completely sure about your questions on multicollinearity. To put it another way: the answers to these questions are not obvious to me, and I recommend figuring these out by just simulating them in R.

3. I think that scaling is important for the interpretation of parameters (see here) and if you’re going to use Bayesian priors (see here). There’s also nonlinear scaling (logs, etc.) or combining predictors, which will change your model entirely.