Just in time for Halloween, here’s a horror story for you . . .
Howard Wainer writes:
In my book “Uneducated Guesses” in the chapter on value-added models, I discuss how the treatment of missing data can have a profound effect on the estimates of teacher scores. I made up how a principal might send the best students on a field trip at the beginning of the year when the ‘pre-test’ was given (and their scores would be imputed from the students who showed up) and that the bottom half of the class would have a matching field trip on the day of the post test. Everyone laughed.
But apparently someone decided to take it seriously.
You can’t make this stuff up.
This sort of thing is not surprising but it’s worth keeping in mind. That a measurement system can be gamed, does not mean it’s useless, but part of good measurement is to consider these problems.