Confirmation bias

Shravan Vasishth is unimpressed by this evidence that was given to support the claim that being bilingual postpones symptoms of dementia:

Screen Shot 2016-07-26 at 1.23.27 PM

My reaction: Seems like there could be some selection issues, no?

Shravan: Also, low sample size, and confirming what she already believes. I would be more impressed if she found evidence against the bilingual advantage.

Me: Hmmm, that last bit is tricky, as there’s also a motivation for people to find surprising, stunning results.

Shravan: Yes, but you will never find that this surprising, stunning result is something that goes against the author’s own previously published work. It always goes against someone *else*’s. I find this issue to be the most surprising and worrying of all, even more than p-hacking, that we only ever find evidence consistent with our beliefs and theories, never against.

Indeed, Shravan’s example confirms what I already thought about scientists.

10 thoughts on “Confirmation bias

  1. Education is typically seen as ‘protective’ against dementia, presumably at least in part due to its correlation with other positive factors (SES, good brain function, and so on). The bilinguals also score numerically lower on the neuropsych/behavioral tests. I would consider a later onset of dementia to be surprising/stunning, barring actually reading the article.

  2. Bialystok did this back in 2014 so why is it today’s topic? How far behind is the Blog? Although p-value does not have much standing in this Blog, here is what can be found with the numbers given:

    For MCI
    Sample N Mean StDev SE Mean
    1 38 62.2 13.2 2.1
    2 36 66.9 11.1 1.8

    Difference = μ (1) – μ (2)
    Estimate for difference: -4.70
    95% CI for difference: (-10.34, 0.94)
    T-Test of difference = 0 (vs ≠): T-Value = -1.66 P-Value = 0.101 DF = 71

    For AD
    Sample N Mean StDev SE Mean
    1 35 72.2 10.3 1.7
    2 40 78.80 8.80 1.4

    Difference = μ (1) – μ (2)
    Estimate for difference: -6.60
    95% CI for difference: (-11.05, -2.15)
    T-Test of difference = 0 (vs ≠): T-Value = -2.96 P-Value = 0.004 DF = 67

    Below is a NYT article about Bialystok’s investigations of the benefits of bilingualism:

    http://www.nytimes.com/2011/05/31/science/31conversation.html

    • I think I saw this article mentioned on twitter in June 2016, so Andrew’s blog is at most on a six month lag (although I think he did say once that it was on a three month lag). Someone on twitter even extrapolated as follows: “Bialystok has found this is a continuous variable, so even a little bilingualism helps although a lot helps even more”.

      Table 3 in the paper is interesting, because if one were in the business of arguing that bilingualism affords no advantage, you could easily argue with the same table as follows: “Nine comparisons, corrected for multiple comparisons (absolute value of critical t=2.7, corrected alpha=0.005), showed no evidence of a bilingual advantage in these tests.”

      Actually, am I mistaken or is the second comparison on the right hand side (AD) in bold for the stroop effect not statistically significant even at alpha 0.05?

      (9.88-11.55)/(sqrt((3.32^2/25)+(3.69^2/29)))
      [1] -1.750235

      I probably calculated something wrong here?

      • Oh, OK, now I understand. Actually Table 3 is meant to show that the monolinguals and bilinguals are at a comparable cognitive level:

        Second, the
        general absence of differences between monolinguals and bilinguals
        on the detailed executive function tests within each diagnostic
        group at the time of the first clinic visit (see Table 3) shows that
        the patients in both languages groups were at comparable cognitive
        levels. … This finding confirms previous reports of
        equivalent cognitive levels between monolingual and bilingual
        participants, despite bilinguals being older at the time of diagnosis.

        Incidentally, they note and then dismiss the role of immigrant status as follows:

        Bilinguals were different from monolinguals in two respects that
        might have confounded these results. First, bilinguals were more
        likely than monolinguals to be immigrants, leading to the possibility
        that immigration status and not bilingualism was responsible
        for later onset. However, that possibility is ruled out by the
        analyses showing no association between immigration status and
        onset age (partial correlation, r 0.02), and no substantive
        change in the effects of language group and diagnostic group on
        onset and clinic appointment ages when immigration status was
        included in the model. That is, immigration status had no systematic
        effect on age of onset or age of first clinic visit, and inclusion
        in the model did not attenuate the positive effects of bilingualism.

        All this with a between-subject comparison involving 35 or so subjects!

        PS I have no belief about whether bilingualism affords any kind of advantage. It might for all I know. I’m just commenting on the poor use of statistical methods here.

  3. It’s been a while since I last read about this subject, but I think my larger problem with studies like this (and a lot of observational studies more generally), is when the claim is accompanied by the statement, “More research is needed to explain why this is.” I don’t necessarily have a problem with the result confirming what one believes if one has a well-founded theory for why they expect to find those results. I should think that would be the case. But I’m skeptical when one study just leads to another study that leads to additional studies and the research concerning the explanatory mechanism generating the results just keeps getting deferred into the future.

  4. As a counterexample to Shravan’s typical academic reaction, let me cite a paper which I always liked: Friedman and Ostroy. “Competitivity and Competition in Auction Markets: An Experimental and Theoretical Investigation,” Economic Journal, 1995

    which describes a problem and then says: “This paper began as a sharp disagreement between the two authors as to the proper explanation of the puzzle. We investigate three approaches…. The traditionalist approach, initially favored by one author,…, the institutionalist approach initially favored by the other author… and a third approach [the one they ultimately demonstrate] which only occurred to us after looking at the results of some experiments.”

    The refreshing candor of this paper is one of the reasons it has stuck with me for the last 20 years.

  5. Crude comparisons of bilingual and monolingual people are treacherous, at least in US population. Bilingualism is uncommon in the US and is found primarily in the children of immigrants. This carries with it confounding socioeconomic factors that must be accounted for in any cogent analysis of the effects of bilingualism.

  6. I suspect that one reason that people work on such research problems is that they are unwilling to tackle the hard but important questions in their field. In bilingualism, surely the big open problem must be developing computational process models of language learning? That’s where energy needs to be directed, not toward mickey mouse questions like whether bilingualism is good for you or not.

Leave a Reply

Your email address will not be published. Required fields are marked *