Theodore Vasiloudis writes:
I came upon this article by Laura Hamilton, an assistant professor in the University of California at Merced, that claims that “The more money that parents provide for higher education, the lower the grades their children earn.”
I can’t help but feel that there something wrong with the basis of the study or a confounding factor causing this apparent correlation, and since you often comment on studies on your blog I thought you might find this study interesting.
My reply: I have to admit that the description above made me suspicious of the study before I even looked at it. On first thought, I’d expect the effect of parent’s financial contributions to be positive (as they free the student from the need to get a job during college), but not negative. Hamilton argues that “parental investments create a disincentive for student achievement,” which may be—but I’m generally suspicious of arguments in which the rebound is bigger than the main effect.
So I clicked through to the study and indeed found a problem, and it’s the one you might expect if you follow this sort of thing. Hamilton regresses college grades on the amount parents spent on college, and finds a negative correlation: more parental spending is associated with lower grades.
The problem I’m worrying about is selection, what is sometimes called “survivorship bias.” If you’re not doing so well in college but it’s being paid for, you might stay. But if you’re paying for it yourself, you might just quit. This would induce the negative correlation in the data, but not at all through the causal story that Hamilton is telling, that students who are parentally-supported “dial down their academic efforts.” Couldn’t it just be that these students are working as hard as they ever would, they’re just more likely to stay in school?
To say it another way: it seems completely plausible to me that (a) there could be a negative correlation between parental support and student grades, conditional on students being in college, but (b) paying more of your college student’s tuition would not (on average) lower his or her grades.
Yes, Hamilton runs an analysis controlling for students’ college admissions test scores, but (a) admissions test scores don’t tell the whole story about students’ abilities and readiness for college, and (b) she also controls for post-treatment variables such as student’s major, full-time status, and—amazingly—whether the student is employed during school. You shouldn’t control for these things! You really really shouldn’t control for whether a student is working during school, if you’re trying to estimate the effect of parental support.
Hamilton also performs an analysis on a different dataset including within-student comparisons when student aid varied over time. This is fine but it has the same problem that it doesn’t seem that students get counted after they drop out. Again, an observed correlation does not necessarily correspond to a causal effect—and I don’t say this in a vague “correlation does not equal causation” sense but more specifically that there’s a clear selection problem going on here (as well as a comparison conditional on intermediate outcomes). Hamilton writes, “These results provide strong evidence that selectivity processes are not driving the negative relationship between parental aid and GPA,” but I don’t see it.
As a side note, I’m slightly bothered by this graph from the paper:
I mean, sure, a graph is better than a table full of numbers such as “-11.503***,” but . . . are there really many families making $5,000 or $15,000 a year and giving their kids $40,000 to pay for college? I guess it’s possible but it seems unlikely to me. A footnote says, “The funds needed to provide parental aid can come from a variety of sources . . . Parents may thus offer funds that exceed income earned in a given year.” Still, I can’t imagine it happens that often. And, when it does, I’d expect the impact on the student to be different from funds supplied by a richer family. I can’t imagine it feels so much like free money to a student if parents are paying more for college than they (the parents) make in a year. So, to the extent the graph makes sense and even if I were to believe the author’s data analysis and theoretical model, I wouldn’t believe the model would hold in this case.
I’m surprised the reviewers for the American Sociological Review didn’t catch these problems, which are pretty standard issues in causal inference: survivorship bias and controlling for intermediate variables. Or maybe they wanted to publish the paper because it represents empirical work—even if flawed—on an important topic.
Or maybe there’s something I’m missing. That’s certainly possible—it happens all the time!—and I’d be happy to be corrected on any of my points above.
P.S. Oooh! This study was also mentioned (uncritically) in the Times and on Gawker. So it must be real.
P.P.S. More from Joseph Delaney.
Andrew:
I buy your selection story, yet there is another perhaps simpler story: students do badly so parents spend more money on their education (e.g. they want to make up for their not so brilliant children).
As for why ASR went along with it, my money is on it having something to do with confirmation bias. Who does not want to lampoon and blame the rich for everything? It would be a nice topic for a PhD Comic http://www.phdcomics.com/comics.php.
Not read piece but:
“If you’re not so smart you decide not to go to college, but if someone is paying for the free ride you go”.
or
“If you are busy building computers in your garage you might not go to college, but if someone ….”
or
“If you are romantically engaged and want to wander the world in a VW van you might not go to college, but if things turn sour and someone….”
or
you get the point.
Some of these are time varying factors, other not. Fixed effects etc are unlikely to cut it here. Moreover, in all these cases the counterfactual is not “Don’t pay and your daughter will get better grades” but rather “don’t pay and she’ll get on with life out of college”. I suppose many parents with money don’t like the latter outcome.
Conditional on selection there appears to be a positive effect of cash incentives on student performance according to RCTs http://www.huffingtonpost.com/2012/06/21/study-finds-chicago-stude_n_1616910.html
Maybe parents ought to (i) pay their children more and (ii) make it conditional on academic performance!
Granted I didn’t read the paper, but my first thought was: bright rich kids don’t pay sticker price for school nearly as often as not-so-bright rich kids, so it ends up being that their parents are paying more.
Yeah, this was my thought, too.
What could be done to remedy this? Only look at first semester students, before anyone has had a chance to drop out?
With regards to the employment control, it is also entirely possible that this is due to a bad reviewer complaining “you didn’t control for ___.”
There could be sinister reasons for people to push the meme in question. What does the flawed study seem to suggest at a policy level? Education cuts, cuts, cuts! If we eliminate student aid completely, we’ll see grades soar!
I am more curious about: parents pay for their kids do get better grades. This is pretty obvious with Ivy League. I don’t think anyone is really poor, though I can’t say it is causal.
@Sam: Interestingly, we have randomized evidence for government support (which, unlike family support, can be randomized) and the main effect is in the opposite direction. So the policy conclusions is that *parents* should reduce the amount that they give because people are more responsible with government grant money than family money.
If that sounds odd then maybe the authors of the study should have gotten worried . . .
Suggestions for ASR editors:
– Judge whether the research question is important. If it is, presumably any answer is interesting (yes/no, significant/not significant);
– Judge whether the research designs is credible. A credible design gives Nature a reasonable chance to contradict the researchers;
– Ignore the answer reported in the manuscript.
Do not base your decision on:
– preconceptions;
– “interesting findings” (often no more than speculation backed up by unreported specification searches);
– significance.
These likely flaws went right through the skeptical economists at Marginal Revolution as well:
http://marginalrevolution.com/marginalrevolution/2013/01/dont-pay-for-all-of-your-kids-college-education.html
“I should note that this piece includes all of the appropriate controls” although Cowen hedges this with “but still we do not know how good those controls are and perhaps parental paying practices are proxying for other features of the situation.”
Interestingly enough, many of the commenters at Tyler’s blog pointed out the selection bias problem.
If we believe that both grades and money contribute to getting through college (and it’s hard not to), then even if money has zero effect on grades, money and grades will be negatively correlated if we condition on getting through college. Which this study appears to do.
In other words, if we believe that the following causal DAG is a subgraph of the true system
Grades → Graduation ← Money
then conditioning on Graduation creates a correlation between Grades and Money that has nothing to do with causation.
Re: “I’d expect the effect of parent’s financial contributions to be positive (as they free the student from the need to get a job during college)”
I have a bachelor’s degree from a foreign university and a master’s from one of the UC’s.
One of my biggest surprises in the University of California was that the amount of time I was supposed to spend studying was drastically smaller than what I was used to back home.
When I was getting my bachelor’s, I had to attend lectures and seminars 30-40 hours a week. Classes would start at 9 a.m. and run till 5..7 p.m., with a lunch break in the middle of the day. There would be homework on top of that.
In the United States, being present on campus for 12 hours/week was considered “full-time”. I actually managed to get my master’s while working full-time off-campus, with some flexibility in the hours I was putting in at work.
So please forgive me if I don’t buy the claim that freeing the U.S. student from having to get a part time job has a non-negligible negative effect on his grades.
Obviously, that should be, “forcing the U.S. student to get a part time job …” I have two degrees, but I still can’t always express my thoughts correctly at the first attempt.
I attended a commuter college, graduating in 1972. The prevailing thought then was that 12-16 hours of work a week would have only a trivial effect on grades — you would have less time, but your time management skills would improve.
I do not know what current research suggests.
I’m inclined to snark, but instead I’ll just mention that both actual class/lab and homework load (as well as difficulty of the word) vary enormously between majors and (less so) between schools. Your personal experience is not representative of all US undergraduates.
Well, I’ll also add that while the stereotypes have some truth to them with regard to, say, the large US universities and between the humanities and STEM; when you start looking at particular majors and particular schools, you’ll find far more very strong exceptions to these stereotypes than most people expect. There are certain STEM degrees which are relatively easy and certain humanities and fine arts degrees which are extremely difficult. There are certain well-regarded schools which are relatively easy and certain less-well-regarded schools which are very difficult.
There positively, absolutely are certain undergraduate degrees in the US where a student cannot reasonably be expected to even work part-time, much less full-time. There are undergraduate degrees in the US which require an amount of classroom/lab time equivalent to what you describe in your foreign experience. There are undergraduate degrees in the US which require an average student to spend at least that much time on homework. So please reconsider your assertion that there are no students in the US for whom part-time work will adversely affect their grades.
Well, my assertion is not that there are _no_ students in the US for whom part-time work would adversely affect grades. It is possible that there are some marginal students in certain programs an certain universities for whom just the combined class+homework load is so large that getting a job would be harmful. My assertion is that the typical student in a typical university has much more free time than he knows what to do with.
To move from assertions to facts, I found a site called “National Survey of Student Engagement” with assorted statistics on university students. Among other things, it says that the median senior in a U.S. university reports spending 14 hours/week on “Preparing for class (studying, reading, writing, doing homework or lab work, analyzing data, rehearsing, and other academic activities)”. Once we correct for social desirability bias, it could be closer to 10.
Strangely, the survey does not ask about hours spent in the classroom. Maybe I don’t know how to ask Google the right question, but this data is hard to come by. I did find information that, at most universities, an undergraduate student is expected to take 15 credits per semester, and each credit is 15-16 contact hours.
So, our typical student spends 25 hours/week studying during the school year (just over 1/2 of the calendar year), out of 100 waking hours, and he spends 0 hours/week studying between semesters.
I rest my case.
a) At my school students are likely to underreport hours spent on preparing for class. b) Furthermore, they forget that they in exam periods spend 70 hours a week.
c) You forgot to specify that the survey had also explained that students spend 0 hours pr. week between semesters…or?
d) And I am to belive that students have half the year off in a typical american university? You surely aren’t forgetting preparing for exams etc.?
I rest my case. Up to you to assess what I mean with “case”.
“And I am to belive that students have half the year off in a typical american university? You surely aren’t forgetting preparing for exams etc.?”
Official University of California, San Diego academic calendar:
http://blink.ucsd.edu/instructors/resources/academic/calendars/2012.html
Fall quarter: 9/27 to 12/15 (11.5 weeks)
Winter quarter: 1/7 to 3/23 (11 weeks)
Spring quarter: 4/1 to 6/14 (11 weeks)
total: 33.5 weeks or 64% of the year, including exams.
Nice cherry picking Nameless.
You could also have chosen this one: http://exploredegrees.stanford.edu/academiccalendar/#text –> different results.
The courses are designed so that the professors expect the typical student to spend (on average) 2-3 hours of work outside of class for each credit hour of the class. So, for an average student taking 15 credits of classes that’s 15 hours of class time* and 30-45 hours of homework/studying bringing the numbers of hours spend a week to 45-60. A standard work week is 40 hours, so it is reasonable to call 15 hours full time. You almost have a point about the number of days off of classes in the year (summer break is long enough to interfer with learning), but we go to college to learn and brains work best when they are not under undue stress. A system where students are in classes 15 hours a week, at work 40 hours, and doing homework for 30 hours for 4 years sets up students to fail. Giving 8 hours of sleep per day (sleep deprivation severly limits learning so our system should set aside that time for them to sleep) that’s only 27 hours of free time a week or 4 hours a day. Free time which will be largely spent on maintenance activities like eating, bathing, and traveling.
*Assuming no laboratory classes, where 1 credit hour can be 3-4 hours of class time.**
**Why yes, I was a science major, how did you know?
I understand the selection problem, but what’s wrong with controlling for whether the student is employed? I mean if there is also a direct effect of parental aid as well as an indirect effect via whether the student has to take employment?
Dave:
The problem with controlling for intermediate variables is well known but subtle. In short: if you control for the intermediate outcome you can apparently find a causal effect even though none is there. See section 9.7 of my book with Jennifer for a clear (I hope) discussion.
Here’s more the question, what is meant by “intermediate variables”? I mean this isn’t like a single simple designed intervention. We didn’t have for example a bunch of students, and we gave some of them a bunch of “parental money” and others not, and then based on that one-time grant we could look at a later time and see that they had jobs or not, and also what their grades were. Clearly, the effect of the early grant would be to force more of the non-grant people to get jobs than the grant people, but in the context of a fluid tradeoff between loans, work, and parental grant money it’s hard to say what’s an “intermediate outcome” and what’s a “treatment”.
Daniel:
Given Hamilton’s claims, “parental investments create a disincentive for student achievement” and “Parental aid decreases student GPA,” I think it’s fair to say that she is considering counterfactuals in which parents pay less or more of their kids’ college costs. In that case, I’d consider students’ work status to come after the treatment.
Interestingly, the author adresses survivor bias in the NY Times piece:
” The higher graduation rate of students whose parents paid their way is not surprising, she said, since many students leave college for financial reasons. (…)
Oddly, a lot of the parents who contributed the most money didn’t get the best returns on their investment (…) Their students were more likely to stay and graduate, but their G.P.A.’s were mediocre at best, and some I didn’t see study even once.”
My insta-theory(tm): endowment effect – i.e. it’s so much easier to play meritocracy if your parents are wealthy.
‘I’m generally suspicious of arguments in which the rebound is bigger than the main effect’ should be up with Occam’s razor as basic methodological principle
Geoff:
Give it a name and I’ll put it in the lexicon. The Rebound Principle? The Rebound Rule?
The Shaquille O’Neil principle?
Err, Shaquille O’Neal that is
A couple comments:
1) I’d like to see an analysis of the effect working part-time has on student grades, i.e., all other things being equal what, if any, effect does working have on student grades?
2) That graph sets off all kinds of alarm bells. What about confidence intervals? I have a hard time buying that those curves correspond to high fidelity fits to data. A simple continuous curve with negligible uncertainty in each case? Really? There’s sufficiently little variation for students in the $5k income, $35k contribution bucket that confidence intervals aren’t significant? Really?!? And what about that bridge in Brooklyn you say I can buy cheap?
The survivor bias is a valid point. Also there is something more fundamentally flawed with her data process. She says it comes from a database of all 4 year schools. But many listed as full-time undergrads now (is it 9 or 12 credits?) are really employee getting paid to advance their skills. Examples are active military personnel, med techs going on for BSRN degrees, various technicians studying engineering. So they often get reimbursed by their grade and study material they have some knowledge of. Also many scholarship students avoid classes with risk of low grades if their money is grade-dependent. She should have limited her data to traditional 18-24 year old full-time student who are claimed as dependents. Then there would be a meaningful comparison the impact parental of support.
We need a serious discussion of undergraduate life and it sounds like her book is a start. It sounds like she is another social scientist who feels her qualitative research is not good enough and needs numbers to back an argument. You have posted a few examples.
Pingback: Somewhere else, part 30 | Freakonometrics
Isn’t student aid contingent on maintaining a certain GPA? I know a lot of scholarships do this, so it is possible that students whose GPAs fall below the threshold drop out when the funding dries up, while others might continue. There is a definite survivor bias in the analysis.
The parent’s increased load Over past 10 years http://blog.collegetuitioncompare.com/2012/12/tuition-rates-changes-chart-10-years.html