Shravan Vasishth is unimpressed by this evidence that was given to support the claim that being bilingual postpones symptoms of dementia:
My reaction: Seems like there could be some selection issues, no?
Shravan: Also, low sample size, and confirming what she already believes. I would be more impressed if she found evidence against the bilingual advantage.
Me: Hmmm, that last bit is tricky, as there’s also a motivation for people to find surprising, stunning results.
Shravan: Yes, but you will never find that this surprising, stunning result is something that goes against the author’s own previously published work. It always goes against someone *else*’s. I find this issue to be the most surprising and worrying of all, even more than p-hacking, that we only ever find evidence consistent with our beliefs and theories, never against.
Indeed, Shravan’s example confirms what I already thought about scientists.
Education is typically seen as ‘protective’ against dementia, presumably at least in part due to its correlation with other positive factors (SES, good brain function, and so on). The bilinguals also score numerically lower on the neuropsych/behavioral tests. I would consider a later onset of dementia to be surprising/stunning, barring actually reading the article.
Bialystok did this back in 2014 so why is it today’s topic? How far behind is the Blog? Although p-value does not have much standing in this Blog, here is what can be found with the numbers given:
For MCI
Sample N Mean StDev SE Mean
1 38 62.2 13.2 2.1
2 36 66.9 11.1 1.8
Difference = μ (1) – μ (2)
Estimate for difference: -4.70
95% CI for difference: (-10.34, 0.94)
T-Test of difference = 0 (vs ≠): T-Value = -1.66 P-Value = 0.101 DF = 71
For AD
Sample N Mean StDev SE Mean
1 35 72.2 10.3 1.7
2 40 78.80 8.80 1.4
Difference = μ (1) – μ (2)
Estimate for difference: -6.60
95% CI for difference: (-11.05, -2.15)
T-Test of difference = 0 (vs ≠): T-Value = -2.96 P-Value = 0.004 DF = 67
Below is a NYT article about Bialystok’s investigations of the benefits of bilingualism:
http://www.nytimes.com/2011/05/31/science/31conversation.html
I think I saw this article mentioned on twitter in June 2016, so Andrew’s blog is at most on a six month lag (although I think he did say once that it was on a three month lag). Someone on twitter even extrapolated as follows: “Bialystok has found this is a continuous variable, so even a little bilingualism helps although a lot helps even more”.
Table 3 in the paper is interesting, because if one were in the business of arguing that bilingualism affords no advantage, you could easily argue with the same table as follows: “Nine comparisons, corrected for multiple comparisons (absolute value of critical t=2.7, corrected alpha=0.005), showed no evidence of a bilingual advantage in these tests.”
Actually, am I mistaken or is the second comparison on the right hand side (AD) in bold for the stroop effect not statistically significant even at alpha 0.05?
(9.88-11.55)/(sqrt((3.32^2/25)+(3.69^2/29)))
[1] -1.750235
I probably calculated something wrong here?
Oh, OK, now I understand. Actually Table 3 is meant to show that the monolinguals and bilinguals are at a comparable cognitive level:
Second, the
general absence of differences between monolinguals and bilinguals
on the detailed executive function tests within each diagnostic
group at the time of the first clinic visit (see Table 3) shows that
the patients in both languages groups were at comparable cognitive
levels. … This finding confirms previous reports of
equivalent cognitive levels between monolingual and bilingual
participants, despite bilinguals being older at the time of diagnosis.
Incidentally, they note and then dismiss the role of immigrant status as follows:
Bilinguals were different from monolinguals in two respects that
might have confounded these results. First, bilinguals were more
likely than monolinguals to be immigrants, leading to the possibility
that immigration status and not bilingualism was responsible
for later onset. However, that possibility is ruled out by the
analyses showing no association between immigration status and
onset age (partial correlation, r 0.02), and no substantive
change in the effects of language group and diagnostic group on
onset and clinic appointment ages when immigration status was
included in the model. That is, immigration status had no systematic
effect on age of onset or age of first clinic visit, and inclusion
in the model did not attenuate the positive effects of bilingualism.
All this with a between-subject comparison involving 35 or so subjects!
PS I have no belief about whether bilingualism affords any kind of advantage. It might for all I know. I’m just commenting on the poor use of statistical methods here.
It’s been a while since I last read about this subject, but I think my larger problem with studies like this (and a lot of observational studies more generally), is when the claim is accompanied by the statement, “More research is needed to explain why this is.” I don’t necessarily have a problem with the result confirming what one believes if one has a well-founded theory for why they expect to find those results. I should think that would be the case. But I’m skeptical when one study just leads to another study that leads to additional studies and the research concerning the explanatory mechanism generating the results just keeps getting deferred into the future.
>”a well-founded theory”
If they had a well founded theory we would be seeing a comparison to prediction here. We don’t see that.
As a counterexample to Shravan’s typical academic reaction, let me cite a paper which I always liked: Friedman and Ostroy. “Competitivity and Competition in Auction Markets: An Experimental and Theoretical Investigation,” Economic Journal, 1995
which describes a problem and then says: “This paper began as a sharp disagreement between the two authors as to the proper explanation of the puzzle. We investigate three approaches…. The traditionalist approach, initially favored by one author,…, the institutionalist approach initially favored by the other author… and a third approach [the one they ultimately demonstrate] which only occurred to us after looking at the results of some experiments.”
The refreshing candor of this paper is one of the reasons it has stuck with me for the last 20 years.
Crude comparisons of bilingual and monolingual people are treacherous, at least in US population. Bilingualism is uncommon in the US and is found primarily in the children of immigrants. This carries with it confounding socioeconomic factors that must be accounted for in any cogent analysis of the effects of bilingualism.
Not sure about these specific results, but this mini-field in general and Bialystok in particular has had some pretty acrimonious replicability debates over the past few years. See, e.g. this paper https://www.ncbi.nlm.nih.gov/pubmed/23370226
and this story about it in the Atlantic: http://www.theatlantic.com/science/archive/2016/02/the-battle-over-bilingualism/462114/ .
I suspect that one reason that people work on such research problems is that they are unwilling to tackle the hard but important questions in their field. In bilingualism, surely the big open problem must be developing computational process models of language learning? That’s where energy needs to be directed, not toward mickey mouse questions like whether bilingualism is good for you or not.