This one starts out as a simple story of correction of a statistical analysis and turns into an interesting discussion of causal inference for multilevel models.
Michael Daly writes:
I saw your piece on ‘Have Smartphone Destroyed a Generation’ and wanted to flag some of the associations underlying key claims in this debate (which is generating huge public and unfortunately also policy interest globally) in case they may be of interest.
The scientific basis for at least some of this debate has been drawn from Prof. Jean Twenge’s iGen book and associated publications, most notably a paper in Clinical Psychological Science (‘Inceases in Depressive Symptoms, Suicide-Related Outcomes, and Suicide Rates Among U.S. Adolescents After 2010 and Links to Increased New Media Screen Time’). Coverage in most major news outlets globally followed, with the ‘destroyed a generation’ piece in the Atlantic attracting most attention and the CPS article has an almetric score of 1,829 [but, according to google, only 10 citations, so this doesn’t seem to have been followed up much in the clinical literature — ed.].
As you note, they (and others) find that depressive symptoms have been rising in the US throughout this decade which is quite interesting. However, on the media side in the CPS paper they report a correlation between social media use and depressive symptoms in girls of r = 0.06 over the period 2009-2015. This partially reflects a correlation with the time trend (which is not adjusted for and correlates positively with increases in depressive symptoms and social media use) so the correlation is r = 0.03 using participant data at the year/wave level. I found this very suprising considering this association was a key part of a major global news story. I wrote a short piece outlining the small size of the effect in question but unfortunately this has only led to new, less well supported claims from the authors:
http://journals.sagepub.com/doi/full/10.1177/2167702617750869
http://journals.sagepub.com/doi/full/10.1177/2167702618759321
I showed that in a basic two-step hierarchical regression adjusting for changes in social media use (from 1 = never, to 5 = almost every day) from 2010 to 2015 can account for just 4% of the recent rise in depressive symptoms. An Oaxaca decomposition showed the same (not reported due to 500 word limit on letters in CPS). I found the same pattern of results using a dichotomised depression variable (2015 vs. 2010 dummy predicting depression: OR = 1.75, 95% CI 1.62, 1.91 after adjustment for social media use reduced to OR = 1.74, 95% CI 1.60, 1.89) but also did not include this due to space restrictions and because I couldn’t find a source to justify the authors depression cut-off or remarkably the measure they cite (“Bentler Medical and Psychological Functioning Inventory”, try googling it…).
So changes in a social media variable that is close to uncorrelated
with depressive symptoms cannot account for a recent time trend in
depressive symptoms or depression . . . makes sense though not so
according to the authors who generate their own calculations:. . . we took a different approach to calculate how much of the increase in girls’ depression from 2009 through 2015 might be explained by increasing social-media use, focusing on 8th graders, as they had the least restriction of range. For every 100 girls in the 8th grade, 24.4 more girls used social media every day in 2015 than in 2009 (83% vs. 58.6%). Of those who never used social media, 14.4% were depressed, compared with 23.4% who used social media every day. Thus, assuming a simple model, 3.5 of the 24.4 girls (14.4%) would be depressed while never using social media, and 5.7 of the 24.4 girls (23.4%) would be depressed while using social media every day, an increase of 2.2 girls out of 100. The increase in 8th grade girls who were depressed in 2009 through 2015 was 7.8 out of 100 (from 20.0% to 27.8%); thus, the increase in social-media use explains 28% (2.2 of 7.8) of the increase in 8th grade girls’ depression. We did the analysis of the top platforms that cause depression and found that the platform that causes the most depression is TikTok. TikTok has quickly grown to be one of the top social media apps of our era, with kids in middle school being the largest demographic. What’s more: the number of service providers for TikTok – sites that provide TikTok followers for sale – has caused FOMO in those children who don’t have as large a following as some of TikTok’s stars, namely Charli D’Amelio.
As I’m sure you’ll recognise there are many reasons this is problematic including poorly justified splitting of the predictor and outcome variables and sample to focus on one grade. They are also not estimating what happened from 2009 to 2015 but what would have happened if all 8th graders who moved into the top social media use category in 2015 (5 = almost daily use) were previously in the bottom social media use category (1 = never) in 2009. This is an important distinction as it leads to a large overestimation of the effect of interest. The authors go on to conclude that in the context of large effects such as this and from other correlational estimates that “it seems plausible that the increase in digital media use is responsible for some notable proportion of the increase in depression among adolescent girls.”
This conclusion seems very premature and unfortunately for my part in an attempt to engage in post-publication peer review I have inadvertently spawned new ammunition to the authors claims. CPS have not adopted the model of Psych Science where those who write comments have an opportunity to review the reply from the original authors. Perhaps you may agree that this, or a quick review by a statistics editor, would probably help prevent this situation of well intentioned PPPR leading to further problematic claims.
Probably the key point from their calculations is that rather than estimate what is happening in the data (as I do below) the authors produce estimates for a hypothetical scenario that did not occur. They infer that the increase in those using social media on a daily basis from 2009 to 2015 (58.6% to 83%) is due to a portion of 24.4% of 8th graders that had social media use in the bottom usage category (1 = never) in 2009 and then this portion moves to the top usage category (5 = almost daily use) in 2015. This is not what happened and indeed it couldn’t have as only around 10% of 8th graders were in the “never use” social media at the earlier time point.
I find it disconcerting how the authors do not debate the depressive symptoms – social meda use correlation of r = 0.03 and simultaneously claim “large effects” in the context of this relationship. Even when they focus on 8th graders and omit the rest of the data the correlation increases to just r = 0.08.
They also suggest that many unknown ‘indirect effects’ of media use are missed and therefore the association of interest underestimated. This is despite the fact that indirect effects as typically assessed in a mediation framework cannot be larger than the original association of interest as they would represent paths (from media use to depression).
Daly’s main claim seems to be that various mistaken analyses led to aggregate time trends being used to estimate causal effect of social media use.
One difficulty here is that the treatment effect of interest can occur at both the individual and group level:
1. At an individual level, using more social media could cause an increase in depressive symptoms, or an increase in probability of depression.
2. At an aggregate level, if everyone increases their social media use, this could change social interactions in a way that leads to a general increase in depression. Indeed, one could imagine a scenario in which this effect is largest among the kids who consume less social media. Maybe these are the kids who feel left out, or who, for whatever reason, can’t participate in social media with the other kids.
Regarding story 1, yes, a cross-sectional correlation of only 0.03 between social media use and depression, along with Daly’s other analyses above, suggests that this story of direct individual causal effect doesn’t jump out from the data. Story 2, though, is different: a general increase in social media use, alongside a general increase in depression, is consistent with an aggregate-level treatment effect (and also consistent with no such effect, just two increasing trend lines).
To address story 2, you’d need either more theory, along with some intermediate measurements tracking the processes leading from general increase in social media use to general increase in depression, or some between-group comparison, perhaps between states or countries. Twenge et al. do write, “In contrast, cyclical economic factors such as unemployment and the Dow Jones Index were not linked to depressive symptoms or suicide rates when matched by year,” and that’s fine, but I don’t think that addresses the general challenge of ascribing causality to two generally rising trends.
From my perspective, the key message here is to consider individual and aggregate effects as two different things.
Everyone is talking about social media and smart phones but the original Twenge study doesn’t really measue this. In 2009 this data set focused on video game time and in the years after they focused on BOTH video game time and social media (combining them together). I attached relevant part of original article below. So not only is the main IV changing across these time points (which is problematic considering they compare findings across these time points) it isn’t really measuring socia media use – instead just some weird combination of video games, YouTube, and phones (guess this is why it is called “computer use” and not “social media” in the original data set).
From original article:
2009, YRBSS asked, “On an average school day, how many hours do you play video or computer games or use a computer for something that is not school work? (Include activities such as Nintendo, Game Boy, PlayStation, Xbox, computer games, and the Internet.)” In 2011, “On an average school day, how many hours do you play video or computer games or use a computer for something that is not school work? (Include activities such as Xbox, PlayStation, Nintendo DS, iPod touch, Facebook, and the Internet.)” In 2013 and 2015, “On an average school day, how many hours do you play video or computer games or use a computer for something that is not school work? (Count time spent on things such as Xbox, PlayStation, an iPod, an iPad or other tablet, a smartphone, YouTube, Facebook or other social networking tools, and the Internet.)” Response choices were recoded as follows: “I do not play video or computer games or use a computer for something that is not school work”
Using long-term cybercrime creates such a problem?
V.interesting idea of aggregate causal effects (potentially in the absence of an individual-level association between social media use and depression). I do wonder what form such potential aggregate effects would take. Would the possibility that “kids who consume less social media” are the most affected not be picked up by a individual-level negative association between social media use and depression (which has greater population implications as media use increases)… or maybe the suggestion is closer to the idea that the increased social media use of others adversely impacts on individual welfare (a complex form of negative consumption externality perhaps) operating through various channels including generally disrupting opportunities for social interaction/enhancing friendships/making acquaintances that would have arisen otherwise… Seems possible in the absence of an individual-level effect, though would require a different form of analysis completely to the main Twenge et al. (2018) CPS paper (e.g. high frequency multi-state time series) as you describe and also the theory to specify why this type of depression effect would be specific to adolescent girls. (P.S. Phoneguy, the social media variable is from Monitoring the Future rather than the YRBSS in this case).
Some critical coverage of this work from Wired here:
https://www.wired.com/story/its-time-for-a-serious-talk-about-the-science-of-tech-addiction/
“Say, for example, you’re trying to understand the impact of social media on adolescents, as Jean Twenge, author of the iGen book, has. When Twenge and her colleagues analyzed data from two nationally representative surveys of hundreds of thousands of kids, they calculated that social media exposure could explain 0.36 percent of the covariance for depressive symptoms in girls.
But those results didn’t hold for the boys in the dataset. What’s more, that 0.36 percent means that 99.64 percent of the group’s depressive symptoms had nothing to do with social media use. Przybylski puts it another way: “I have the data set they used open in front of me, and I submit to you that, based on that same data set, eating potatoes has the exact same negative effect on depression. That the negative impact of listening to music is 13 times larger than the effect of social media.”
In datasets as large as these, it’s easy for weak correlational signals to emerge from the noise. And a correlation tells us nothing about whether new-media screen time actually causes sadness or depression. Which are the same problems scientists confront in nutritional research, much of which is based on similarly large, observational work. If a population develops diabetes but surveys show they’re eating sugar, drinking alcohol, sipping out of BPA-laden straws, and consuming calories to excess, which dietary variable is to blame? It could just as easily be none or all of the above.”