God, Guns, and Gaydar: The Laws of Probability Push You to Overestimate Small Groups

Earlier today, Nate criticized a U.S. military survey that asks troops the question, “Do you currently serve with a male or female Service member you believe to be homosexual.” [emphasis added] As Nate points out, by asking this question in such a speculative way, “it would seem that you’ll be picking up a tremendous number of false positives–soldiers who are believed to be gay, but aren’t–and that these false positives will swamp any instances in which soldiers (in spite of DADT) are actually somewhat open about their same-sex attractions.”

This is a general problem in survey research. In an article in Chance magazine in 1997, “The myth of millions of annual self-defense gun uses: a case study of survey overestimates of rare events” [see here for related references], David Hemenway uses the false-positive, false-negative reasoning to explain this bias in terms of probability theory. Misclassifications that induce seemingly minor biases in estimates of certain small probabilities can lead to large errors in estimated frequencies. Hemenway discusses this effect in the context of traditional medical risk problems and then argues that this bias has caused researchers to drastically overestimate the number of times that guns have been used for self defense. Direct extrapolations from surveys suggest 2.5 million self-defense gun uses per year in the United States, but Hemenway shows how response errors could be causing this estimate to be too high by a factor of 10.

Here are a couple more examples from Hemenway’s 1997 article:

The National Rifle Association reports 3 million dues-paying members, or about 1.5% of American adults. In national random telephone surveys, however, 4-10% of respondents claim that they are dues-paying NRA members. Similarly, although Sports Illustrated reports that fewer than 3% of American households purchase the magazine, in national surveys 15% of respondents claim that they are current subscribers.

Gays are estimated to be about 3% of the general population (whether the percentage is higher or lower in the military, I have no idea), so you can see how it can be very difficult to interpret the results of “gaydar” questions.

P.S. This post really is about guns and gaydar, not so much about God, but to maintain consistency with the above title, I’ll link to this note on the persistent overreporting of church attendance in national surveys.

8 thoughts on “God, Guns, and Gaydar: The Laws of Probability Push You to Overestimate Small Groups

  1. "Gays are estimated to be about 3% of the general population (whether the percentage is higher or lower in the military, I have no idea)"

    You can estimate the percentage of male homosexuals in a profession from the number of deaths by AIDS during the 1980s and 1990s. The U.S. military did not, on the whole, have a major problem with AIDS, suggesting that in a volunteer military, you get men volunteering who like the idea of breaking stuff and killing people, and they tend to be heterosexual.

    Similarly, deaths from AIDS were low for most professional sports, except figure skating, where they were very high: e.g., both men's gold medalists from 1970s Winter Olympics died of AIDS.

    In contrast, fashion design (e.g., Perry Ellis) and dance/choreography (e.g., Nureyev) were decimated by AIDS.

    Among famous musicians, despite lots of heroin use, pianists and singers, especially show tunes types, were more likely to die of AIDS. Few guitar rock heroes died of AIDS.

    I read the New York Times obituaries daily in 1993, and prominent men who died before, say, age 60 tended to be unmarried, and tended to be said to have died either of AIDS or some AIDS-related complication. There were often references at the end to a long time companion.

    Fortunately, after that, better medical techniques reduced the AIDS death rates considerably. Still, we have a huge amount of evidence available to anybody who wants to look. Estimating rates of homosexuality by profession from AIDS deaths would make a good research project for somebody with tenure and a thick skin.

  2. It's great that Andrew has called attention to the Hemenway paper which is a great paper.

    Notice that the logic here compelling for surveys of conflict violence. Even within war-torn environments violent deaths are still generally rare at the household level. This means that even a small percentage of false positives (recording violent deaths in households where there weren't any) would overwhelm false negatives and cause large upward bias in violent-death estimates.

    Mike Spagat

  3. "You can estimate the percentage of male homosexuals in a profession from the number of deaths by AIDS during the 1980s and 1990s. The U.S. military did not, on the whole, have a major problem with AIDS, suggesting that in a volunteer military, you get men volunteering who like the idea of breaking stuff and killing people, and they tend to be heterosexual."

    Or, one can argue that homosexuals in the miltary were unlikely to engage in unprotected sexual intercourse with multiple same sex partners as compared to 'famous musicians, especially show tunes types.' I'll buy a link with AIDS and risky sexual behavior, although risky sexual behavior is a favorite pastime of both hetero and homo, sexuals.

  4. Getting back to the survey question: do we know what the surveyor was trying to determine? If they're trying to determine the number of gays in the military, the question is ridiculous for the reasons stated. But if they're trying to determine the attitude of members of the military to serving with gays, it's probably one of the right questions. For instance, if a lot of people currently think they're serving with gays now, they will presumably be untroubled by ending "don't ask, don't tell." So I'm not convinced this is a bad question. I'm not saying it's not. It depends.

  5. "I'll buy a link with AIDS and risky sexual behavior, although risky sexual behavior is a favorite pastime of both hetero and homo, sexuals."

    That's why all those wild and crazy risk-taking U.S. Marines died of AIDS while San Francisco librarians went largely untouched.

    Oh, except the opposite actually happened.

    That's why Jimmy Page of Led Zeppelin died of AIDS — groupies and heroin injections killed him.

    Oh, except he's alive and fine. In fact, very few rock musicians died of AIDS, the exception that validates my rule being Freddie Mercury, whose band was named Queen not by coincidence. (Queen's guitarist, Brian May, by the way, recently earned his Ph.D. in astrophysics.)

    Having lived through the AIDS epidemic, I am astonished by how far the basic lessons learned about this vast news story have been shoved down the memory hole only a generation later.

  6. I like the Hemenway article's discussion of false-postive, false-negative errors, but his opening discussion on use of firearms to defend against burglary is quite flawed.

    The first problem is his assumption that the "burglaries" reported by responders in the telephone survey meet the definition of "burglary" in the victimization studies that he cites. This is surely incorrect. The pattern I have seen is "I heard a noise outside and got my shotgun, went to check it out and the guy outside ran away." While Mr. Shotgun may report this as use of a firearm to repel a burglary, that incident would certainly not be classified (absent some evidence of actual or attempted entry) as an actual or attempted burglary. Most likely a wino was rummaging in the garbage, or some kids were trying to steal a garden gnome.

    Leaving aside that definitional problem, a more serious one is the assumption that burglary victims are no more likely than the general population to possess firearms and use them to repel attacks. This is a surpassingly odd assumption: one would expect persons at higher risk of victimization to be more likely to possess firearms and be prepared to use them in defense.

    The oddest thing, however, is Hemenway's belief that an analysis of 198 incidents in Atlanta provides a reliable estimate of anything.

Comments are closed.