A Wikipedia whitewash

After hearing a few times about the divorce predictions of researchers John Gottman and James Murray (work that was featured in Blink with a claim that they could predict with 83 percent accuracy whether a couple would be divorced–after meeting with them for 15 minutes) and feeling some skepticism, I decided to do the Lord’s work and amend Gottman’s wikipedia entry, which had a paragraph saying:

Gottman found his methodology predicts with 90% accuracy which newlywed couples will remain married and which will divorce four to six years later. It is also 81% percent accurate in predicting which marriages will survive after seven to nine years.

I added the following:

Gottman’s claim of 81% or 90% accuracy is misleading, however, because the accuracy is measured only after fitting a model to his data. There is no evidence that he can predict the outcome of a marriage with high accuracy in advance. As Laurie Abraham writes, “For the 1998 study, which focused on videotapes of 57 newlywed couples . . . He knew the marital status of his subjects at six years, and he fed that information into a computer along with the communication patterns turned up on the videos. Then he asked the computer, in effect: Create an equation that maximizes the ability of my chosen variables to distinguish among the divorced, happy, and unhappy. . . . What Gottman did wasn’t really a prediction of the future but a formula built after the couples’ outcomes were already known. . . . The next step, however–one absolutely required by the scientific method–is to apply your equation to a fresh sample to see whether it actually works. That is especially necessary with small data slices (such as 57 couples), because patterns that appear important are more likely to be mere flukes. But Gottman never did that. Each paper he’s published heralding so-called predictions is based on a new equation created after the fact by a computer model.”

I was thinking this would just get shot down right away, but I checked on it every now and then and it was still up.

Finally, on 21 May, my paragraph was completely removed by contributor Annsy5, who also wrote:

Full disclosure: I [Annsy5] work for The Gottman Relationship Institute, which was co-founded by John Gottman, and we would like a change made to the Wikipedia entry on him.

The 3rd paragraph is made up largely of Laurie Abraham’s claims about Dr. Gottman’s research. Ms. Abraham’s claims are inaccurate, and thorough citations can be found here: http://www.gottman.com/49853/Research-FAQs.html. We would like the paragraph removed, or at least moved to a section where the details of Dr. Gottman’s research can be expanded upon.

I know that it would be a violation of the Conflict of Interest policy for me to just go in and make the changes, so I would like other editors’ input. We’re not trying to bury anything “bad” about Dr. Gottman, we just want the information that is out there to be accurate! Please advise…

I don’t know enough about Wikipedia to want to add my paragraph back in, but what’s going on here? On 23:57, 20 May 2010, Annsy5 writes “I know that it would be a violation of the Conflict of Interest policy for me to just go in and make the changes,” and then on 23:13, 21 May 2010, Annsy5 goes and removes the paragraph and all references to criticisms of Gottman’s work.

That doesn’t seem right to me. A link to a rebuttal by Gottman would be fine. But removing all criticism while leaving the disputed “90% accuracy” claim . . . that’s a bit unscholarly, no?

P.S. A commenter asked why I posted this on the blog rather than doing this on wikipedia. The reason is that I’m more interested in the wikipedia aspect of this than the marriage-counseling aspect, and I thought the blog entry might get some interesting discussion. I know nothing about Gottman and Murray beyond what I’ve written on the blog, and I’m certainly not trying to make any expert criticism of their work. What does seem to be happening is that they get their claims out in the media and don’t have much motivation to tone down the sometimes overstated claims made on their behalf. Whatever the detailed merits of Abraham’s criticisms, I thought it was uncool for them to be removed from the wikipedia pages: Her reporting is as legitimate as Gladwell’s. But I’m not the one to make any technical additions here.

25 thoughts on “A Wikipedia whitewash

  1. I went for the following edit, which acknowledges the study, and its limitations:

    In a 1998 study, Gottman predicted with 90% accuracy which newlywed couples will remain married and which will divorce four to six years later. It also predicted with 81% percent accuracy which marriages survived after seven to nine years.[3] His prediction method, which relies on Paul Ekman's method of analyzing human emotion and microexpressions, is also featured in Malcom Gladwell's book Blink and the television series The Human Face.

    Let's see if that stays up. Otherwise, prepare for an edit war, and pray for a good moderator.

  2. Cody:

    But you don't think it would be good to refer to Laurie Abraham's criticisms? Her point is that he wasn't really predicting with such high accuracy.

    In any case, it's no big deal. I was actually surprised that my paragraph lasted as long as it did.

  3. They are clearly in breach of wiki's conflict of interest guidelines (but appear to have used appropriate tags in the talk page before making an edit). Posting on your blog, though, is the wrong way to deal with this. You should be posting on the discussion page for that article. I'd revert back to that old version that includes the paragraph, or move any of the claims and counterclaims about Gottman's work into a new section.

  4. If you think it's worth the effort, one reasonable course of action would be to:

    1) Peruse the FAQ links provided by Annsy5 and attempt a neutral synthesis. Focus on the question of whether Laurie Abraham's criticisms are actually inaccurate. (Ten minutes of effort is enough to show willing.)
    2) Add back your text with any additional info from step 1 under a new heading (or possibly subheading, depending on what's already there) entitled "Criticism". In your edit summary, put an invitation to resolve any issues on the talk page.
    3) Initiate a discussion on the talk page and wait for response.

  5. Hi, Andrew.

    Yes, it was not proper for Gottman's people to remove negative information from an article. It violates the conflict-of-interest policy, [1] and it was also editing to promote a particular point of view, which is the greater sin. [2]

    I've restored a shorter version of Abraham's criticism, and referenced their oblique disputation of it.

    What I could really use is an outside expert saying something in print that makes things clearer. I, as a Wikipedian, am forbidden to conduct original research. [3] You, however, are a noted expert in your field, and so if you look at the papers the Gottman Relationship Institute references and opine on them professionally, perhaps right here on this blog, that could well be the sort of thing that belongs in a Wikipedia article. [4] So if you get the urge, write something up and drop me a line about it.

    [1] http://en.wikipedia.org/wiki/Wikipedia:COI
    [2] http://en.wikipedia.org/wiki/Wikipedia:NPOV
    [3] http://en.wikipedia.org/wiki/Wikipedia:OR
    [4] http://en.wikipedia.org/wiki/Wikipedia:RS

  6. I think the Wiki way of handling this should be to add a "criticism" section, with a shorter version of the criticism and a cite linking to a longer version, plus perhaps a cite linking to a rebuttal.

  7. I see that the Wikipedia page now (1:31pm) contains similar text to that which was previously whitewashed, with the addition of

    The Gottman Relationship Institute claims this is incorrect, and that 6 of 7 of Gottman's studies have been properly predictive

    with a link to the Gottman Institute Research FAQ … complete with shopping cart and checkout button.

  8. Weird thing is I can predict successful couples within moments of meeting them, with huge accuracy. However, whether a bad marriage ends in divorce or is content to struggle along is down to external pressures you can't possibly see.

  9. The Gottman Institute FAQ that Matt mentions, which is linked both from the main Wiki page (right now) and the talk page, says the following:

    Six of the seven studies have been predictive—each began with a hypothesis about factors leading to divorce. Based on these factors, Dr. Gottman predicted who would divorce, then followed the couples for a pre-determined length of time. Finally, he drew conclusions about the accuracy of his predictions. He has also consistently evaluated other theoretical models that might predict differently and reported the results of these analyses (e.g., Gottman & Levenson, 2002). This is true prediction.

    So I went and looked at Gottman & Levenson (2002), which among other things says the following in the discussion section:

    The limitations of the present study are that this was a relatively small sample of couples in the first place, and that the analyses were post hoc and thus the conclusions must be taken as exploratory and hypothesis-generating.

    Glad that's cleared up.

  10. I wonder if Annsy5 will contribute to this thread and answer all our questions that were not resolved by the Faq.

  11. Sanjay: The FAQ contains this gem:

    Dr. Gottman’s ability to predict divorce among newlyweds is more clearly understood by imagining an urn that contains 130 white balls (representing couples that stayed married) and 17 red balls (representing couples that ended up divorcing) for a total of 147 balls. The chances that Dr. Gottman could blindly pick balls out of the urn and guess which were red and which were white with 90% accuracy could only happen by chance 1 x 10-19 times.

    Of course, if you guess that they are all red, you'll be 88% accurate all of the time. If instead, you guess that the first 3 balls drawn are white and the rest are red, you'll get the 90% accuracy about 0.1% of the time. Not high, but a good 16 orders of magnitude better than the FAQ suggests.

  12. Sorry for the double comment, I commented on the wrong post. (feel free to accept or delete the other one).

    As someone that does machine learning research, what bugs me the most is "accuracy" as the metric. Why no sensitivity, specificity, confusion matrices, AUROC curves, etc.?

    Oh, that's right, because they didn't do binary classification at all, as Laurie Abraham and Andrew Gelman point out.

  13. I'm not familiar with Gottman's research but I think this touches on a very interesting statistical point: the difference between explanation of variance and prediction. Assume Gottman collects his data and generates a linear combination of predictor variables to optimally explain variance in (or "predict") divorce occurence. Were he then to publish a paper based on that, wouldn't it be fair to say "With knowledge of these predictor variables, we are able to account for X% of the variance in Y"?

    In experimental work, often times, we only have one dataset with a small sample size, and in such cases, I think it would be fair to publish such data and to characterize it as "prediction." One might even say that the proposed predictive model is able to predict the response variable with such and such level of accuracy (with the understanding that the model was chosen to optimize this metric).

    A subsequent study could then be conducted to validate the initial model; either use the model generated from the test sample to predict responses in the validation sample, or generate a new model for the validation sample and compare its weights with those from the corresponding model generated from the test sample.

    Gottman's real error seems to be he doesn't seem to have attempted to propose one unified model that can be validated with multiple independent samples. He is able to generate significant predictive models within a single study (which is equivalent to finding a correlation between divorce occurrence and a given set of predictors); but use of the word "prediction" in a larger context seems to presuppose more rigorous attempts at validation.

  14. I'm surprised that paragraph stayed up too. It's so long, and it doesn't have a neutral POV. I would have expected something more like "Critics have suggested that the success of the matching technique is due to the equation being fit to the data after the fact" and a cite to Abrahamson. The rest of the verbiage isn't there to inform the reader about the criticism, but to argue for it.

  15. Noumenon: I agree that it's definitely helpful for wikipedia experts such as yourself to edit the contributions of subject-matter experts such as myself. Neither of us could do it alone. That said, I'd prefer if you were not to refer to my writing as "verbiage." Thank you.

  16. Saying that a claim is misleading would be a point of view (POV in wikipedia speak) that doesn't really belong in an wikipedia article.

    It better to state what's claimed by different parties and to describe the disagreement.

  17. If the Earth ever gets invaded by aliens and they discover Wikipedia, they'll think we're idiots: the way is "edited" is completely nonsense from an academic or professional perspective.

  18. This Gottman book is ridiculous:

    http://www.amazon.com/Mathematics-Marriage-Dynami

    Here is a write-up from slate that completely misses the point:

    http://www.slate.com/id/2081484

    The slate author assumes that the equation concerns the entire marriage. That would make sense but they don't have data for the entire marriage. The equations merely concern the initial videotaped conversation. They create non-linear models for each of the initial videotaped conversations then try to see if the parameters they select are different for the divorcing and non-divorcing couples. They get no clear results, but they wrote a book anyway.

  19. I intended to post a sarcastic comment about the desirability of a "neutral point of view," something along the lines of "Adolf Hitler: some people say he was a dynamic statesman who attempted to ensure prosperity for his countrymen; critics claim both his goals and his methods were unethical." But by god, sometimes reality is as good as any parody, and this is an example: "Hitler ultimately wanted to establish a New Order of absolute Nazi German hegemony in Europe. To achieve this, he pursued a foreign policy with the declared goal of seizing Lebensraum ("living space") for the Aryan people; directing the resources of the state towards this goal."

    Strange punctuation there, but I think we can agree that it presents a neutral point of view. And indeed, the whole article seems to avoid value judgments. Personally, I don't think that's a good thing, but I admit a certain admiration for the writers and editors who have collectively written an entire, long article about Hitler without using the words "evil" or "immoral", or even "bad", except in a very short section about Hitler's "legacy." It's like that guy who wrote an entire novel without using the letter "e": OK, it's interesting that it can be done, but is the result really better than it would be if you didn't have the constraint?

  20. Hmm, I didn't even realize I was being a jerk there (nor the true connotations of "verbiage"). I guess I thought I was commenting on the paragraph, but I was really commenting on the author's judgment about what fits in Wikipedia, and should have been more tactful or helpful. Sorry.

  21. Yes, the only evidence that invading aliens would have the idiocy of the human race would be an online collaborative encyclopedia. That makes perfect sense.

    re: "the way is "edited" is completely nonsense from an academic or professional perspective."

    I'm not conceding the point that it is "completely nonsense from an academic or professional perspective," but why should it be held to those standards anyway? It is not an academic or professional publication. It is a self describe "encyclopedia that anyone can edit." By its very definition it is an amateur undertaking. And yet it still has thousands of high quality articles…

  22. Andrew: here we run up against some perverse effects of Wikipedia rules. You're allowed to quote and link to Laurie Abraham's critique, which (as we've discussed) I think is not quite right. On the other hand, because of the "no original content" rule you are not allowed to post your own useful observation that the "80% accuracy" figure is essentially meaningless, for reasons having nothing to do with post hoc fitting (see Sanjay and Justin's comments above.) The obvious solution is that one of the commenters here should go amend the Wikipedia page to include that.

    Also, to Lemmy Caution above, I wrote that Slate piece and I still think I got the point of the book right!

Comments are closed.