Howard Wainer writes:
When we focus only on the differences between groups, we too easily lose track of the big picture. Nowhere is this more obvious than in the current public discussions of the size of the gap in test scores that is observed between racial groups. It has been noted that in New Jersey the gap between the average scores of white and black students on the well-developed scale of the National Assessment of Educational Progress (NAEP) has shrunk by only about 25 percent over the past two decades. The conclusion drawn was that even though the change is in the right direction, it is far too slow.
But focusing on the difference blinds us to what has been a remarkable success in education over the past 20 years. Although the direction and size of student improvements are considered across many subject areas and many age groups, I will describe just one — 4th grade mathematics. . . . there have been steep gains for both racial groups over this period (somewhat steeper gains for blacks than for whites). . . . New Jersey’s black students performed as well in 2011 as New Jersey’s white students did in 1992. Given the consequential differences in wealth between these two groups, which has always been inextricably connected with student performance, reaching this mark is an accomplishment worthy of applause, not criticism.
He concludes:
If we couple our concerns about American education and the remarkable success shown in this data, it seems sensible to try to understand what was going on, so that we can do more of it. . . . A little more than 20 years ago, several suits challenging the way that public schools were financed . . . The courts decided that in order for the mandated “equal educational opportunity” to be true, per-pupil expenditures in all school districts should be about equal. In order for that to happen, given the vast differences in the tax base across different communities, the state had to step in and augment the school budgets of poorer districts. The fact that substantially increased funding has accompanied these substantial improvements in student performance must be considered as a prime candidate in any search for cause.
Howard sent this to me and I passed it on to my various contacts in journalism. I didn’t hear back from anyone—I guess it was deemed not exciting enough to appear in any major newspaper or magazine, so it eventually ended up in “NJ Spotlight.” I like it, but maybe the problem was that it wasn’t topical enough. Maybe Howard should’ve sat on the piece for awhile and saved it to time with some test-scores report? These things come out at regular intervals and are usually good for a headline or too.
Maybe a graph would explain this faster than three paragraphs of prose?
Aleks:
To empasize the same point for meta-analysis, we developed the L’Abbe plot, to “force” people not to lose sight of the nuissance parameters (e.g. control rate, control mean).
Unfortunately most just use forrest plots of some (poorly) chosen outcome comparison :-(
Almost any plot of the raw summaries available would likely suffice.
Didn’t you tell him his graph is terrible? Why does he have time running from right to left?
Good point!
My impression is that states that have pursued inter-district equalization most seriously have on average lagged states which have kept Strayer-Haig type funding formulas. Last time I looked, New Jersey, for example had the greatest variance in per-student funding in the US (that may no longer be the case). I wouldn’t push this too hard, we are talking about 50 observations, basically, and you can eliminate the inverse relationship simply by removing Hawaii and California from the data set. But the evidence is quite strong that even significant inter-district equalization in the states where it has occurred is not generally associated with improvements in student test scores (Kentucky seems to be an exception).
US schools often get a bad rap. http://super-economy.blogspot.com/2010/12/amazing-truth-about-pisa-scores-usa.html
So, on Wainer’s point about progress, I agree. Only bad news seems newsworthy. But he is a little quick to jump to causality. My hunch is that we are likely to find better explanations in greater emphasis on early childhood education, intra district reallocation of resources, especially from high schools to elementary schools, and programs like Texas’, which focus on pass rates rather than mean or median test scores. Others would stress greater competition and charter schools, although the evidence on that point isn’t strong (some people have gotten strong results on both sides of the issue, but too often it seems to me that their results are suspiciously consistent with their priors).
When something seems to good to be true, it usually isn’t. I read that “New Jersey’s black students performed as well in 2011 as New Jersey’s white students did in 1992” and I immediately think, did we really achieve this amazing degree of progress (improving quality of education to compensate for a full 1 sigma in IQ in less than 20 years), or is it possible that 1992 scores on that test aren’t directly comparable with 2011 scores?
Nameless – it’s about student performance, not IQ. Quite a difference.
Exactly. IQs are the same, but either textbooks, or teaching practices, or tests themselves changed so much and so quickly that today’s students at the 16th percentile by IQ today get the same test scores as students at the 50th percentile 20 years ago.
Turns out that NAEP has a separate “long-term” assessment scale that is supposed to be used for intertemporal comparisons. In 4th grade mathematics, it does show _some_ progress, though not as much as Howard Wainer’s op-ed:
1992: blacks 208, whites 235 (gap 27)
2008: blacks 224, whites 250 (gap 26)
In 12th grade mathematics, there’s no statistically significant progress at all:
1992: blacks 286, whites 312
2008: blacks 287, whites 314
Rewriting some of these 4th grader numbers to compare to Wainer’s point:
208 blacks in 1992
224 blacks in 2008
235 whites in 1992
Blacks in 1992 were 27 points behind whites; in 2008 they were 11 points behind where whites had been in 1992. If the tests from the different years really are comparable then this does show a lot of progress, although only about 40% as much as Wainer said. And at least it discredits the idea that math test scores are some sort of measure of “innate” IQ.