Are smaller schools better? Another example of artifacts in rank data

Alex Tabarrok writes, regarding the example in Section 3 of this paper,

Another nice illustration of the importance of weighting comes from high-stakes schemes that reward schools for improving test scores. North Carolina, for example, gives significant monetary awards to schools that raise their grades the most over the year. The smallest decile of schools has been awarded the highest-honors (top-25 in the state) 27% of the time while schools in the largest decile have received that honor only about 1% of the time. Students (and parents) are naturally led to believe that small schools are better. But just as with the cancer data, the worst schools also come from the smallest decile. The reason, of course, is the same as with the cancer data small changes in incoming student cohorts make the variance of the score changes much larger at the smaller schools. There are some nice graphs and discussion in

Kane, T. and D. O. Staiger. 2002. The Promise and Pitfalls of Using Imprecise School Accountability Measures. Journal of Economic Perspectives 16 (4):91-114.

It’s scary to think of policies being implemented based on the fallacy of looking at the highest-ranking cases and ignoring sample size. But most of my students every year get the cancer-rate example wrong–that’s one reason it’s a good example!–so I guess it’s not a surprise that policymakers can make the mistake too. And even though people point out the error, it can be hard to get the message out. (For example, Kane and Staiger hadn’t head of my paper with Phil Price on the topic, and until recently, I hadn’t heard of Kane and Staiger’s paper either.)

2 thoughts on “Are smaller schools better? Another example of artifacts in rank data

  1. Dude, how can anyone still not have heard of our paper? The title alone should make it famous.

    But seriously, the fact that smaller samples lead to more stochastic variability is one of the most famous statistical facts ever. Everybody knows this (well, maybe not "everybody", but most people), and can recognize it effects when it's pointed out to them. What is interesting to me is that people don't recognize it when it isn't pointed out to them.

    –Phil

Comments are closed.