“16 and Pregnant”

Ted Joyce writes:

In December 2015 the AER published an article, “Media Influences on Social Outcomes: The Impact of MTV’s 16 and Pregnant on Teen Childbearing,” by Melissa Kearney and Phil Levine [KL]. The NBER working paper of this article appeared in January of 2014. It received huge media attention as the authors claimed the show was responsible for 25% of the decline in teen childbearing from 2008-2010. . . .

Joyce and his colleagues [David Jaeger, Robert Kaestner] were skeptical:

So we buy the Nielsen data and we acquire the confidential birth certificate data and we have a go. Attached is the second version of that effort. . . .

In brief, KL are trying to identify the effect of a nationally broadcast program on teen fertility amidst large secular trends and at the nadir of the Great Recession. There is no time variation in the show across units and their IV strategy lacks anything resembling an exogenous instrument. Even a DD interpretation fails the parallel trend assumption. We feel it was their use of Twitter data and Google searches that captured the imagination of the reviewers, but here too we show all their social media results collapse as soon as we use all THEIR data and not just a selected sample.

The paper by Jaeger, Joyce, and Kaestner is called, “Did Reality TV Really Cause a Decline in Teenage Childbearing? A Cautionary Tale of Evaluating Identifying Assumptions,” and here’s their key claim:

We find that controlling for differential time trends in birth rates by a market’s pre-treatment racial/ethnic composition or unemployment rate cause Kearney and Levine’s results to disappear, invalidating the parallel trends assumption necessary for a causal interpretation. Extending the pre-treatment period and estimating placebo tests, we find evidence of an “effect” long before 16 and Pregnant started broadcasting.

I have not had a chance to read any of these papers in detail so I’m just presenting the controversy here without any endorsement (or anti-endorsement) of Jaeger, Joyce, and Kaestner’s argument.

When these sorts of things come up on the blog, some readers feel a bit cheated, that I’m bringing up this live issue and not giving my own take on it. All I can say is that some takes are easier to come by than others, and I think there’s value in all sorts of posts. Sometimes I can do a careful investigation of my own or follow a debate closely enough to have an informed opinion, sometimes I can share a controversy and let the experts chew on it. Science is full of uncertainty and turmoil and it’s not so bad to sometimes present a scientific dispute without trying to resolve it myself.

Also, this one’s about causal inference, which is one of our core blog topics.

12 thoughts on ““16 and Pregnant”

  1. A digression. This study is an example of a number of themes that have been highlighted on this blog in recent years. I haven’t read the analysis yet, but it appears to be a worthy attempt to examine how robust a study’s finding are. However, I would point out that the critique does not provide the data they used, nor does the original study. The NBER paper does not provide the data (although some NBER studies do), and the AER publication only provides some of the data. In particular, despite the AER’s open data policy (note this the next time you want to cite the increasing trend of open data policies in economics journals – this is my experience in at least 50% of the cases – the data that is provided contains a .pdf saying that the data cannot be provided), the readme file says:

    “All of the Vital Statistics data using county identifiers and the Nielsen ratings data used in this project are confidential and cannot be shared. We have included all of the Google Trends and Twitter data (purchased from Topsy Labs, which is no longer in operation). We have also included all of the programs used to generate the results reported in our analysis.”

    I accept the fact that Nielsen data is proprietary – but I really don’t accept the fact that it cannot be released with the study. Is this particular Nielsen data set really of commercial value if it were released? Arguably, releasing the data might be good for Nielsen’s business by letting others see how the data may be used. Also, all of the papers are written in publication style – difficult to read, with figures and tables disassociated from the text.

    So, on one hand, this is the publication system at work – a provocative study is published, a replication is attempted, many of the modeling details are there for public examination and comment. On the other hand, the amount of time and expertise required to wade through these lengthy and hard to digest papers means that very little careful examination is likely to occur. Finally, getting the data to independently analyze is not possible (at least without spending considerable money and time trying to obtain it). One step forward, one step back? (that’s my best attempt to say it is a glass half-full)

  2. Dale,

    I completely sympathize with the unavailability of key data. I pleaded with Nielsen to allow Kearney and Levine (KL) to give us access to the highly aggregated ratings data in their study. Nielsen said no and we had to raise the money. But I will ask again given the data are at least six years old. I can also ask if we can make available the birth data at the DMA level (designated market area). We had to aggregate up from the county level which is why the data are confidential.

    What is available are the twitter and google search data which I can send you along with our STATA programs. We feel the internet data made the study feel novel by using “big data.” But as we demonstrate, none of their results stand up to very simple and we feel more appropriate specifications. Our re-analysis of the internet data is important for two reasons. First, scaling searches or tweets is challenging. For example, Michael Jackson died on 25 June 2009, the same day that “Amber,” the third episode of season 1 of 16 and Pregnant, aired. KL weight their analyses by the number of tweets. Consequently, this would get more weight in KL’s regressions if Michael Jackson’s death caused a spike in Twitter activity. Second, without the internet data there is no evidence of a mechanism by which 16 and Pregnant affected teen birth rates. This becomes important because it’s unclear a priori if the show discouraged or encouraged teen childbearing among different segments of the audience. In subsequent exchanges with us, KL minimize the importance of the internet data. Yet, the internet analysis is far an aside. The presentation of the data and discussion of the internet results take up half of the results section of their AER paper. They conclude, “In all of these approaches using high frequency data, we believe that the results plausibly provide causal estimates of the impact of the show” (KL 2015, p. 3621). The “high frequency data” they refer to are the twitter and google search data.

  3. The second of our two-part series on 16 and Pregnant can be found here: http://www.djaeger.org/research/pubs/IREE-Final.PDF. Here is the abstract:

    We replicate and extend the analysis of the positive association between social media (Google searches and tweets) and the MTV program 16 and Pregnant recently published by Melissa Kearney and Phillip Levine (“Media Influences on Social Outcomes: The Impact of MTV’s 16 and Pregnant on Teen Childbearing,” American Economic Review 105(12):3597–3632, December 2015). We find that the relationship disappears or even turns negative when we include in the analysis periods when new episodes of 16 and Pregnant were not being broadcast. The results are also sensitive to the use of weights. Our results suggest that the causal chain from social media to fertility behavior that Kearney and Levine claim is fragile or non-existent.

Leave a Reply to Ted Joyce Cancel reply

Your email address will not be published. Required fields are marked *