More thoughts on that “What percent of Americans would you say are gay or lesbian?” survey

We had some discussion yesterday about this Gallup poll that asked respondents to guess the percentage of Americans who are gay. The average response was 23%—and this stunningly high number was not just driven by outliers: more than half the respondents estimated the proportion gay as 20% or more.

All this is in stark contrast to direct estimates from surveys that 3 or 4% of Americans are gay.

One thing that came up in comments is that survey respondents with minority sexual orientations might not want to admit it. So maybe that 3-4% is an underestimate. Here’s an informative news article by Samantha Allen which suggests that the traditionally-cited 10% number might not be so far off.

But even if the real rate is 10% (including lots of closeted people), that’s still much less than the 23% from the survey.

Or, to look at it another way, it’s no surprise that respondents grossly overestimate the rate of gays, given that they also grossly overestimate the rate of African Americans, Hispanic Americans, Asian Americans, and immigrants in this and other countries. Or similar overestimates if you ask people what fraction of the Federal budget goes to various well-known but relatively tiny programs.

From this perspective, the problem doesn’t seem to have anything to do with gays but rather a more general lack of numeracy, a lack of understanding about small proportions.

Suppose we wanted to ask the survey question in a way to elicit more accurate responses. One way would be to break up the majority category into subgroups. For example, what percentage of American adults are: Straight and married; Straight, unmarried, in a committed relationship; Straight, never married, and single; Straight and divorced; Straight and widowed; Gay. I’m sure these categories could be phrased better, also you have to figure out what to do with various intermediate categories. The point is that it could make a difference if you set up enough alternative categories.

Similarly for the ethnicities. What if, instead of asking black/Hispanic/Asian/white/other, you ask something like this: black, Hispanic, Asian, American Indian, English, Irish, German, Italian, Polish, Russian, etc.? I don’t know how this would go, but I could see it making a difference.

On the other hand, you shouldn’t have to do it that way to get a reasonable answer.

Another way would be to take a more direct approach: Ask the survey respondent about his or her 10 or 20 closest family members and friends: How many are gay, etc.? (You’d want them to identify the 10 or 20 people first, before saying why you’re asking, otherwise it would be too easy to recall one or more gay person.) Then you could ask about the rest of the population: Do you think you know many more gay people, more gay people, about the same, fewer gay people, or many fewer gay people, compared to the average American.

The point here is not to trick respondents into giving more accurate answers but rather to connect population questions of interest to respondents’ direct experience.

I wonder what Gerd Gigerenzer thinks about all this.

P.S. David Landy writes:

I saw your post today on overestimations of LGBT individuals, and its supposed relationship to innumeracy. Funnily enough, I’ve been meaning to write you on that specific topic (which you’ve posted on before)! I think your interpretation, which is the common one, is either wrong or simply too vague: My colleagues and I published a paper on this topic this year, and have one more in the works*. The short version is that we argue that these overestimations are a direct result of individual-level psychophysical response curves: they are a part of how people estimate proportions of any kind, and have little or nothing to do with people’s perceptions of particular subgroups.

* Here’s a dropbox link to a PDF version of the individual-level data and analyses, that confirm the pattern of responses within single individuals, for a wider variety of demographic items, including LGBT populations. It’s a lot of slides (but just a 20 minute talk)—but the most important/convincing are the data and models on pages 58-62).

Second *: Also in a rank ordering task we conducted, Mechanical Turkers actually seem to underestimate the LGBT population, specifically. This is important because rank ordering distinguishes between general psychophysical biases and specific population-specific misconceptions. Again, we’re prepping for publication, but here’s the key figure:

6 thoughts on “More thoughts on that “What percent of Americans would you say are gay or lesbian?” survey

  1. Maybe more nuetral questions like “what percentage of cars are Japanese?” could be asked to see if there is in fact an overestimation due to lack of numeracy.

  2. “… Or similar overestimates if you ask people what fraction of the Federal budget goes to various well-known but relatively tiny programs.

    From this perspective, the problem doesn’t seem to have anything to do with gays but rather a more general lack of numeracy, a lack of understanding about small proportions.”

    I would guess that the “availability bias” comes into play: If you hear a lot in the news about LGBT people, or certain ethnic groups, or certain government programs, then you are likely to believe that they are more common than if you didn’t hear about them.

  3. Stephen Ansolabehere, Marc Meredith, and Erik Snowberg have a 2013 paper in Political Analysis showing that the use of benchmarks can improve responses to numerical questions (unemployment rate). This might be an approach to improving responses to the demographic estimates too.

    Pay walled link:

    https://www.cambridge.org/core/journals/political-analysis/article/div-classtitleasking-about-numbers-why-and-howdiv/34A5F416016B86460DB34ADC7B1C9803

    Free link:

    http://www.sas.upenn.edu/~marcmere/workingpapers/AskingNumbers.pdf

Leave a Reply to Kyle MacDonald Cancel reply

Your email address will not be published. Required fields are marked *