False memories and statistical analysis

Alison George interviews Elizabeth Loftus. It’s good stuff, important stuff, and it relates to my view as a statistician that uncertainty and variation are important. Uncertainty is relevant here because there are things that people remember that never happened. Variation is important because different people remember different things.

Loftus’s work also seems relevant to the problems with pseudoscience that we’ve been discussing recently on the blog, studies where researchers follow the forms of scientific reasoning and publish in scientific journals, but what they are publishing is essentially unreplicable noise. Perhaps there’s some connection between all those people Loftus has interviewed, who remember events that never happened to them, and people such as Daryl Bem, who think they’ve computed rigorous p-values even though their analyses seem so clearly to be contingent on the data. It’s almost like a false memory, that scientists convince themselves that the analysis they happened to do after the data arrived, was exactly the analysis they’d planned to do ahead of time. Unfortunately there never seems to be a pre-data record of such a plan, but the memory is there.

7 thoughts on “False memories and statistical analysis

  1. Richard Lewontin wrote an infamous review of a report on sexual practices in the U.S. for the New York Review of Books:
    http://www.nybooks.com/articles/archives/1995/apr/20/sex-lies-and-social-science/?pagination=false
    The initial review and the exchange of letters that followed are a must read.

    Lewontin’s theme, if I remember correctly, is that we all build autobiographical stories about ourselves in order to make sense of our life. And of course our sex life is part of this story. The consequence of this is that “blinding” doesn’t matter when asked about who we are and what we’ve done because we recall “facts” from the story we’ve constructed (which we think of as fact) not from the videotape of our life.

    So, as you suggest, apres-experiment is probably no different. Our mind constructs a story of what we did and why we did it and this construction will subtlely (or not so subtlely) modify what we did in a way that makes more sense of the results in the broader context of our work, the field, etc. etc.

    It’s also not too different from the confabulation experiments of Michael Gazzaniga.

    • By the way, one of the books Lewontin reviewed was “The Social Organization of Sexuality” by Edward Laumann and other U. of Chicago sociologists. After reading it, I wrote to Laumann suggesting that his team write about The Sexual Organization of Society, using the example of how Chicago neighborhoods reflect the sexual status of residents. I, for instance, paid to live in glamorous lakefront neighborhood when I was single, moved to a cheaper but still kind of trendy neighborhood when I got engaged, and moved to an unglamorous neighborhood when we got married. In general, much about the always fascinating subject of real estate could be explained by its interactions with the mating process.

      Much to my pleased surprise, years later they published “The Sexual Organization of the City,” following closely my advice to profile different neighorhoods in Chicago based on sexual aspirations.

      http://www.goodreads.com/book/show/420078.The_Sexual_Organization_of_the_City

      The more general lesson is that subjective memories about intimate topics like sex can be cross-checked against public facts, such as real estate demographics.

  2. Jeff:

    Essentially the argument Herb Simon used when they were trying to write AI programs to do expert’s work.

    If you ask them how the did something, you just get story re-constructions.

    He suggested you get them to verbalise what is going through their minds into a tape recorder – when they are doing the work.

    (And thats how I ended up spending a miserable couple weeks trying to write a program to shop for dresses given audio transcripts of some young women shopping for dresses)

  3. I became interested in the false memory issue when allegations of “ritual satanic abuse” became relatively common. And because this followed on the heels of allegations of child abuse in day care centers which resulted in prison sentences despite or perhaps because of the wild stories of secret dungeons and elaborate abuse games that would have required a conspiracy of silence rather than the furtive acts abusers generally use. These became modern witch trials. I would tell my upper middle class, highly educated friends they were at the highest risk of being charged with ritual satanic abuse because that was the profile.

    (See, for example, the Fells Acres case for the difficulty the legal system has in recognizing fake and then in correcting mistakes. Three convictions still in effect despite no physical evidence and a well-detailed manipulative questioning process. I’m sure the former children and their families became convinced these things happened though they did not. It was interesting and sad to watch this unfurl over the years only miles from Salem.)

    Life is essentially a stochastic process, isn’t it? Lots of randomness, lots of pathways that lead to results. Most work done, IMHO, is not replicable because it is a snapshot of how the processes happened to resolve to some results that particular time. Very few things respond with the level of predictability of elementary particles and, even then, we know what we see as results can be calculated as including what we don’t see and what can’t happen.

    The beauty of good data is it includes enough of the underlying processes that it embodies a sufficient amount of reliability and thus of replicability if the terms of the data remain consistent. These are such basic ideas that it boggles my mind to see them so widely misunderstood. BTW, one of my favorite books is a bit of nonsense about Horatio Nelson from the late 19thC. The premise, after all the discussion, is that Nelson at Trafalgar employed derivative calculus when he attacked and, amazingly, he has a single equation that proves this. It never crossed his mind that he was fitting a conclusion to a single event. He thought he was testing his theory: see it predicts and here’s the proof! Lots of lousy science follows that model. And sometimes the jumps are just silly, like finding a sarcophagus with names, forged or not, that indicate names in the New Testament and ignoring the possibility that maybe other people had names like that to say “This is IT!”

    Implanted memories and the like to me reflect our grappling with the underlying mess that is reality. I compare them to the fixations people develop. Obama isn’t a citizen. Israel controls the US – an argument I see in extreme to relatively benign forms, from blunt racist irrationality to anger that somehow Israel “wants” or “expects” the US to do its bidding and attack Syria. (When the issue in the US is about how the US sees itself, how it justifies or doesn’t to itself the vast amounts spent on the military, how it projects and preserves its world power, etc.) I’m also reminded of the mental illness of a friend from college who became fixated on his father’s remarriage as though unraveling that would magically make all his issues disappear. Some argue much of psychiatric analysis falls in that category: people chasing ghosts through their imaginations in the hope the ghost will be the answer. From what I’ve read, implanted memories become that ghost made coherent and that makes them in some ways more real and certainly more valuable and harder to release.

    I had a conversation recently with a friend who prides himself on pulling apart the irrational positions of others. But he’s a gun nut. Grew up in a gun nut family. So the discussion of guns, of any regulation of guns, became about fears of “confiscation”. When I asked how he got there given the Supreme Court’s Heller decision and all the state laws in favor of guns, etc., the conversation devolved into irrational fears he couldn’t recognize. None of us are immune and when we make huge, inconsistent logical leaps we rarely see that’s what we’re doing.

    Fixations are explanations. But they are also explanations that remain stable. That isn’t the way life actually works; it’s more “you can’t step in the same stream twice”. We want stability. In a positive sense, we like having homes and I remind my children that home is the place you return to in your head so you want that place to be happy rather than sad. False memories explain and they are essentially anti-scientific, anti-statistical analysis because they take shape and achieve a form of permanence apart from the flow of random processes that create doubt. And because we want stability, we jump through absurd hoops like imagining somehow there must be secret rooms underneath the pre-school – as in the famous California case where they literally leveled the school searching for proof that didn’t exist – or that literally dozens of children can be abused by numbers of adults in elaborate rituals despite constant comings and goings by outsiders. But heck, we still waste space on national TV every time some idiot claims Jimmy Hoffa is buried there or there or there.

    • I like the idea that life is stochastic. It’s a bit odd that we (most people) want other people, e.g., research subjects to be substantially predictable and are disappointed when our correlations are .3 or so. But we (most people) do not ourselves want to be so very predictable. It’s probably a good thing for mankind that we never get things very right.

    • A somewhat cynical friend pointed out that if all the satanic cults were real we a) would notice a substanial drop in population and b) a lot of suspicious graveyards. I mean where are we going to put all the bodies?

      On the slightly more relevant issue of false memories I’m a great fan of Elizabeth Loftus’ work.

      I also have the two memories of the US President Kennedy’s assassination: One I heard of it at lunch and two at mid-afternoon recess ( I was in grade school). I suspect the lunch time is the correct verison but I am not sure. Not a serious problem but it annnoys me.

Comments are closed.