Andrew has pointed to Jonathan Livengood’s analysis of the correlation between poverty and PISA results, whereby schools with poorer students get poorer test results. I’d have written a comment, but then I couldn’t have inserted a chart.
Andrew points out that a causal analysis is needed. This reminds me of an intervention that has been done before: take a child out of poverty, and bring him up in a better-off family. What’s going to happen? There have been several studies examining correlations between adoptive and biological parents’ IQ (assuming IQ is a test analogous to the math and verbal tests, and that parent IQ is analogous to the quality of instruction – but the point is in the analysis not in the metric). This is the result (from Adoption Strategies by Robin P Corley in Encyclopedia of Life Sciences):
So, while it did make a difference at an early age, with increasing age of the adopted child, the intelligence of adoptive parents might not be making any difference whatsoever in the long run. At the same time, the high IQ parents could have been raising their own child, and it would probably take the same amount of resources.
There are conscientious people who might not choose to have a child because they wouldn’t be able to afford to provide to their own standard (their apartment is too small, for example, or they don’t have enough security and stability while being a graduate student). On the other hand, people with less comprehension might neglect this and impose their child on society without the means to provide for him. Is it good for society to ask the first group to pay taxes, and reallocate the funds to the second group? I don’t know, but it’s a very important question.
I am no expert, especially not in psychology, education, sociology or biology. Moreover, there is a lot more than just IQ: ethics and constructive pro-social behavior are probably more important, and might be explained a lot better by nurture than nature.
I do know that I get anxious whenever a correlation analysis tries to look like a causal analysis. A frequent scenario introduces an outcome (test performance) with a highly correlated predictor (say poverty), and suggests that reducing poverty will improve the outcome. The problem is that poverty is correlated with a number of other predictors. A solution I have found is to understand that multiple predictors information about the outcome overlaps – a tool I use is interaction analysis, whereby we explicate that two predictors’ information overlaps (in contrast to regression coefficients which misleadingly separate the contributions of each predictors). But the real solution is a study of interventions, and the twin and adoptive studies with a longer time horizon are pretty rigorous. I’d be curious about similarly rigorous studies of educational interventions, or about the flaws in the twin and adoptive studies.
[Feb 7, 8:30am] An email points out a potential flaw in the correlation analysis:
The thing which these people systematically missed, was that we don’t really care at all about the correlation between the adopted child’s IQ and that of the adopted parent. The right measure of effect is to look at the difference in IQ level.
Example to drive home the point: Suppose the IQ of every adoptive parent is 120, while the IQ of the biological parents is Normal(100,15), as is that of the biological control siblings is, but that of the adopted children is Normal(110,15). The correlation between adopted children and adopted parents would be exactly zero (because the adopted parents are all so similar), but clearly adoption would have had a massive effect. And, yes, adopted parents, especially in these studies, are very different from the norm, and similar to each other: I don’t know about the Colorado study, but in the famous Minnesota twins study, the mean IQ of the adoptive fathers was indeed 120, as compared to a state average of 105.
The review paper you link to is, so far as I can tell, completely silent about these obvious-seeming points.
I would add that correlations are going to be especially misleading for causal inference in any situation where a variable is being regulated towards some goal level, because, if the regulation is successful. It’s like arguing that the temperature in my kitchen is causally irrelevant to the temperature in my freezer — it’s uncorrelated, but only because a lot of complicated machinery does a lot of work to keep it that way! With that thought in mind, read this.
Indeed, the model based on correlation doesn’t capture the improvement in the average IQ of what the adoptive child would have if brought up in an orphanage or by unwilling or incapable biological parents (as arguably all children put up for adoption are) vs being brought up in a well-functioning family (as probably all adoptive families are). And comments like these are precisely why we should discuss these topics systematically, so that better models can be developed and studied! As a European I am regularly surprised how politicized this topic seems to be in the US. It’s an important question that needs more rigor.
Thanks for the emails and comments, they’re the main reason why I still write these blog posts.
Maybe adoptive children grow in a different environment. They may receive less attention than natural children and they may be traumatized by their situation. It may explain the lack of correlation between AP and AC's IQ.
"I get anxious whenever a correlation analysis tries to look like a causal analysis…"
I do too, especially when the causal structure I'd posit is a lot more complicated. Just off the top of my head, my default model might include:
* A direct impact of parental contribution on learning. Upstream of this would be things like:
1. Value parent(s) place(s) on child's learning (relative to other values, I guess)
2. Parental time available to devote to child's learning. Upstream of this would be things like:
a. Number/age of other children
b. Hours spent working
c. Parental health
3. Parental knowledge specific to what the child is currently learning
a. Parental educational achievement
4. General parental literacy
5. Parental disposable income – the closest thing to "poverty" in the model, I think. Of course upstream of this would be some things elsewhere in the model, some of which imply temporal feedback loops:
a. Parental educational achievement
b. Parental IQ
c. Parental health
…and new things like:
d. Parental value placed on monetary income versus other things of value (e.g. time for leisure and direct contribution to child's educational attainment).
e. Parental "financial savvy"
* A direct impact of child's "IQ" on rate of learning. Upstream of that, things like…
1. Parental IQ (again)
2. Child's lead exposure
3. Child's nutritional status
4. Various other characteristics of the child's home environment, perhaps.
* Impacts of various characteristics the school environment on rate of learning:
1. Parental involvement for other children
2. Class size
3. Teacher skill
etc…
Obviously this isn't a full model (limited time, imagination, and a vague awareness of the "enough already" nature of it) and I make no claim for parsimony or testability – let alone that it's a "good" model for any given purpose.
My only point is that even a rough sketch of *some* of the factors my default understanding says are causally important yields a model that's a *lot* more complex than [poverty] –> [poor educational outcome]. What's more, there doesn't seem to be a strong reason to give poverty primacy of place as a causal factor.
Now, it might turn out to be a good "indicator", especially considering its dependence on a laundry list of causal factors, some of which also enter in elsewhere via feedback loops. That's what I think may drive the strong observed correlation.
Of course it could be that parental disposable income is so impactful that it swamps everything else. It could also be that the model is just wildly wrong about what causes what, and most of it is bunk. But my inclination is to be skeptical of an explanation that comes down to "it's the poverty, stupid".
The problem of similar, "high quality" adoptive parents was addressed in Bruce Sacerdote's studies of Korean adoptees in America. The adoption agency had a policy of randomly assigning adoptees to families, with the result that there was a lot of variation in the income and educational levels of the adoptive parents. While he did not study IQ, he found that parental income and education were only weakly associated with adoptee educational and income outcomes, which were similar across SES levels.
See here, here, and here (note that the adoptees were on average six years younger than the non-adoptees).
I found a copy of the underlying paper via a Google search for anyone who is interested:
http://www.dictionaryofgenetics.com/files/encyclo…
"As a European I am regularly surprised how politicized this topic seems to be in the US."
I think the issue is that there seems to be a desired policy outcome (here in the US) that key groups each want. Plus, education has turned into a hot button topic to traise political support (notice how much Obama has focsued on it lately). Everyone wants to interpret the data in the most favorable light for their policy conclusions.
This makes it difficult to argue for moderate interpretations and more careful testing.
JL, your first two links are the same.
Technology has advanced to the point where we have better measures of heritability than possible through twin-adoption studies.
http://blogs.discovermagazine.com/gnxp/2011/02/wh…
I agree with Joseph on how education has been turned into a hot button, but unfortunately not too much is being done about it. Yes, the talk is there but there is no action.
But anyways back to the topic. I feel that schools in poorer environments have poorer people..not just financially but mentally as well. It's the whole atmosphere of the place. The teachers are slack. The students aren't serious. It just a revolving circle of nothing good. But if we get teachers to teach that aren't from that environment and are from successful environments then things will be completely different. There aren't that many teachers willing to take that action though.
Faizal Nisar
[spam link deleted]
I think Livengood's analysis is particularly interesting because he uses State poverty levels. This provides stronger evidence for a non-genetic causal link than if he had used individual poverty status as a hypothesized causal variable.
We can think of his analysis as the reduced form relationship underlying an estimation in which State is used as an instrument for poverty. More specifically, if we wish to make the case that the poverty/education link is due to genetics, we need to show that the gene pool is very different in the different states. (Eyeballing his plots, I guess the relationship would still be there even if you controlled for ethnic composition in each state).
The correlation he presents seems to me to be strongly suggestive of some economic impact on child outcomes. It might not be directly via parental income though. Parental poverty might be associated with low tax revenue and hence low expenditure on schools.
It is vey hard to do rigorous experimental studies of educational interventions.
Why?
1) There are many many natural variations across space and time. Therefore, real natural experients are almost impossible to find. Too much changes to often.
2) Setting up such experiments is politically problematic. For such things to happen is schools you need the district's approval. And to be a rigorous experiment you need different groups (control group, for example) and random assignment. But some parents will object to their kid being experimented on, and refuse to be in the experimental group. Other parents will insist on the best (newest?) program and refuse to the in the control group. If they find out later which group their kid was in, they will raise holy hell. This kind of ruckus can get an superintendents fired and board members elected out of office.
We don't like experimenting with kids, and certainly not with our OWN kids. Threaten my kid's chance for the best (whichever way I see it) and I will mess with you, but good.
So, truly "rigorous studies of educational interventions" are very very hard to come by.
(and then there are other methodological questions. Are instructors randmly assigned to treatments, too? Which factors at each site need to be controlled for? Which populations or setting might the experiment be generalized to? Etc. etc..)
You also have to contend with things like the Hawthorne effect which plays hell with most educational research.
Ceolof and Mark: Pleading hardship in science is no defence.
Yes there are real difficulties and actually even insurmountable obstacles but I used to hear this all the time in the surgery department and most often somewhere someone did manage to do an RCT in spite of the hardships.
Think in a way it go back to a quote from Lotka, should you study important questions or questions you can answer?
Getting a credible analysis from a Non-RCT is arguably much much harder than conducting an RCT (if ethical).
K?
Wonks Anon, thanks, it should've been this one: http://www.marginalrevolution.com/marginalrevolut…
I can't speak for Ceolof, but I think you've misread my position. It's not that these obstacles are insurmountable or even that they are all that difficult to overcome; it's that they are largely ignored.
Research standards in education are historically not very high and studies tend to be done on the cheap. Add to that the fact that the field is incredibly politicized and that many of the researchers have publicly eschewed the traditional role as objective observer for that of advocate (see Ravitch for examples) and you have a recipe for disaster.
Despite the issues Ceolof raised, I'm certain that we could come up with and conduct good, well-designed RCTs. (for example, if he had wanted to, Joel Klein could easily have authorized the public schools to keep the lottery losers in separate classes thus eliminating concerns about peer effects). The trouble is the flawed studies we've been seeing are presented as conclusive and even smart, statistically literate commentators like Alex Tabarrok will treat them as such.
Mark: sorry if my post seemed judgemental, not my intention.
Just recalling how some when learning about these difficulties do tend to try to make do without.
For one example of where I did, see Polyspecific intravenous immununoglobulin therapy in streptococcal toxic shock syndrome – a comparative observational study. Clinical Infectious Diseases 28:800-807, 1999.
I revisited this as a case study last month and was pleased to find out someone did do an RCT.
Then not so pleased to find out it failed due to lack of recruitment
(untreated mortality appears to be 60% and treated 30% – really nasty disease).
K?