Whipsaw

Kevin Lewis points to a research article by Lawton Swan, John Chambers, Martin Heesacker, and Sondre Nero, “How should we measure Americans’ perceptions of socio-economic mobility,” which reports effects of question wording on surveys on an important topic in economics. They replicated two studies:

Each (independent) research team had prompted similar groups of respondents to estimate the percentage of Americans born into the bottom of the income distribution who improved their socio-economic standing by adulthood, yet the two teams reached ostensibly irreconcilable conclusions: that Americans tend to underestimate (Chambers et al.) and overestimate (Davidai & Gilovich) the true rate of upward social mobility in the U.S.

There are a few challenges here, and I think the biggest is that the questions being asked of survey respondents are so abstract. We’re talking about people who might not be able to name their own representative in Congress—hey, actually I might not be able to do that particular task myself!—and who are misinformed about basic demographics, and then asking tricky questions about the distribution of income of children of parents in different income groups.

Consider the survey question pictured above, from Chambers, Swan & Heesacker (2015). First off, the diagram could be misleading in that the ladder kinda makes it look like they’re talking about people in the middle of the “Bottom 3rd” category, but they’re asking about the average for this group. Also they ask, “what percentage of them do you think stayed in the bottom third of the income distribution (i.e., lower class), like their parents,” but doesn’t that presuppose that the parents stayed in this category? Lots of grad students are in the bottom third of U.S. income, and some of them have kids, but I assume that most of these parents, as well as their kids, will end up in the middle or even upper third once they graduate and get jobs. It’s also not clear, when they ask about “the time those children have grown up to be young adults, in their mid-20’s,” are they talking about income terciles compared to the entire U.S., or just people in their 20s? Also is it really true that the upper third of income is “upper class”? I thought that in the U.S. context you had to do better than than the top third to be upper class. Upper class people make more than $100,000 a year, right? And that’s something like the 90th percentile.

I’m not trying to be picky here, and I’m not trying to diss Swan et al. who are raising some important questions about survey data that regularly get reported uncritically in the news media (see, for example, here and here). I just think this whole way of getting at people’s understanding may be close to hopeless, as the questions are so ill-defined and the truth so hard to know. Attitudes on inequality, social mobility, redistribution, taxation, etc., are important, and maybe there’s a more direct way to study people’s thoughts in this area.

31 thoughts on “Whipsaw

  1. >”It’s also not clear, when they ask about “the time those children have grown up to be young adults, in their mid-20’s,” are they talking about income terciles compared to the entire U.S., or just people in their 20s?”

    This was my confusion.

    >”There are a few challenges here, and I think the biggest is that the questions being asked of survey respondents are so abstract. We’re talking about people who might not be able to name their own representative in Congress—hey, actually I might not be able to do that particular task myself!—and who are misinformed about basic demographics, and then asking tricky questions about the distribution of income of children of parents in different income groups.”

    I’m not sure I follow this complaint.

    1. Are you intentionally suggesting that knowing the name of your congressman is somehow more fundamental knowledge than having a sense of the society you live in?

    2. Sure, people are likely not knowledgable about social mobility, but it’s still important to know in what way they’re not knowledgable. So I don’t see why that’s a “challenge” for the research. Seems like it’s actually the motivation for the research.

  2. Let’s not forget that the real income distribution of the parents is different from the real income distribution of the children 20 years later even if the percentiles stay the same. To illustrate this point more clearly, let’s expand the time-frame… Compare say someone born in 1900 to a family at the 67%tile income… compared to someone born in 2010 to a family at the 33%tile in income (today in the US that’s something like $30k/yr?). The modern “lower class” family has electricity, washing machines or at least laundromats, the ability to call across the country and talk to family and friends at any time for say around a penny a minute, drives a car, and said car has seat belts, air-bags, easily travels 60 miles per hour, and you can survive a crash at that speed. If they get an infection they have antibiotics, and they probably haven’t ever had measles, mumps, rubella, polio, are at no great risk for tetanus, typhoid, tuberculosis, limited risk for pertussis….

    So, which would you rather be on the basis of these example comparisons, an “upper third” person in 1900 or a “lower third” person in 2010?

    The concept of real income is something we need to hammer home in elementary school, because the dollar illusion is pernicious and leads to a lot of bad policy and thinking.

  3. That conditional probability question turns out to be very difficult to answer, even for experts. I’m building a lot of Bayes nets, and we are constantly asking our experts questions like that. For example, I remember one case where we were asking an expert in Math education a very similar question: for students who are low, medium and high on understanding of geometric series how many of them would be low, medium and high on their ability to find a common ratio for a series. She was the same expert who helped us define the variables, and she was comfortable with the probability behind the Bayes nets, but she still didn’t feel comfortable giving those numbers. (We wound up building regression models for the conditional probability tables.)

    If a person with expert training was having so much difficulty, I can’t imagine this class of question would be easy for a lay person. Furthermore, most introductory stat classes don’t cover conditional probability. I think that is a mistake, but that is a topic for another post.

    • ” for students who are low, medium and high on understanding of geometric series how many of them would be low, medium and high on their ability to find a common ratio for a series.”

      I don’t think any reasonable person would hazard an answer to that question unless they had specifically studied that particular question with questions specifically designed to evaluate “understanding of geometric series” and “ability to find the common ratio for a geometric series”.

  4. It seems to me that it might be possible to get more useful results if the question provided some numerical context. For example, if you told respondents something like “If every person were equally likely to grow up rich or poor, regardless of how rich or poor their parents were, then the numbers on each line would be 33.3, 33.3, and 33.3. If people born to rich parents were twice as likely to be rich as adults than people born to poor parents, then then numbers would be [well, I am not sure what the numbers would be, but you get the point]. Given that, what do you think the actual numbers are in the US?”, you might get responses that better reflect what people think about income mobility.

    • I’m not convinced that would be a good question, either.
      But this brings up the general question of choice of questions to be asked in research. This is not an easy task — getting good questions requires testing out the questions on real people, and then revising and repeating the process. Entire books have been written on this topic –e.g.,

      Converse JM and Presser S. (1986). Survey Questions: Handcrafting the Standardized Questionnaire, Sage

      Fowler FJ Jr and Fowler FJ (1995). Improving Survey Questions: Design and Evaluation, Sage.

      Also, in connection with yesterday’s post, see

      Schroeder YC (1987). The procedural and ethical ramifications of pretesting survey questions, Amer. J. of Trial Advocacy, vol 11, pp. 195 – 201.

      and

      Beimer P and L Lyberg (2003), Introduction to Survey Quality, Wiley

  5. In my experience with juries, it’s an heroic assumption to presume that the general public can even understand percentages well enough to answer questions about them, even with the aid of pictures of ladders.

  6. What is the point of the research? If it’s to develop the art of surveys, I can see that. But I have trouble finding value in the estimations of things like perceptions of mobility because those estimations are only important if you’re using them for some other purpose, like lower perceptions of mobility mean we should do this or that. That means the estimations are one dimension – really an inexact summary of some dimension we associate with that estimate – and then it’s applied in at least one other dimension, and likely in many without unbundling what those are. I think there’s a misunderstanding rooted in the idea that all questions should be encouraged, that questioning is good: this is true but it doesn’t mean all questions have value beyond the fact they’re questions. Mathematically we can construct long lines of relations that only mean what they claim to mean if you follow the thread that exact way and the contexts along the way remain the same, which is rarely all that true.

  7. On the rare occasions when surveys ask questions about public awareness of social statistics, the results are pretty dire. If you ask the public to guess what percent are black or Jewish or gay, you soon get minorities adding up to close to 100%.

    Income questions are particularly difficult for non-social scientists to generalize about. Heck, it’s a complicated subject for social scientists.

    • I doubt most people CAN correctly state their current income. Perhaps if we operationally define this as line X on a 1040, we might get a majority. But that excludes things that are income (401k contributions for example), and includes things people don’t think of as income because they never show up in the checking account (e.g. reinvested dividends in retirement plans). Should government payments such as SNAP count as income?

      This is before we get to whether people WILL correctly state their own current income.

      • Is income even well defined as a single thing? ;-)

        If you grow really awesome basil, heirloom tomatoes, squashes, lettuce, and corn, and have chickens that lay fresh eggs in your backyard so that you daily eat fresh food that would at a restaurant cost you say $100 for a couple, and at a grocery store the raw ingredients would cost $30, How *should* this affect our calculation of your income? By the restaurant replacement cost method that alone is $36,500 tax free which is something like the 33rd percentile in total income as calculated using typical measures. To consume $36,500 in restaurant food, while having a house with a garden, you’d need to say pay $1500/mo minimum towards your house, and then $3000/mo towards food. This is $54,000 year after taxes, which is median before-tax income for a family or about that. For a person making $54k post-tax, you’d have to have income something like $75k pre tax, which is something like 70%tile in total income.

        So, if you inherit a home and grow vegetables and cook fresh healthy meals, you make more in real income than most families in the country by far, and economists wonder why labor force participation rate is so low.

        • Another way to look at this is what I call Second Earner Syndrome (I actually spent quite a few months putting together information on this earlier this year. Need to go back to it and blog it in depth).

          If you have a family of 2 living in a house, with one person in the couple working making $55,000 a year (about median before tax cash income for a family), and the other is growing vegetables (or even just shopping at the cheaper grocery stores), cooking, doing laundry, etc in terms of consumption, the largest source of real income is the stay at home member, by FAR if you add one child into the mix (this is real income measured by the before-tax cash income required to replace the consumption, which is I think the relevant measure for the family decision).

          You might think that this would change as the cash income increases, but it doesn’t. As the cash income increases, so too does the marginal tax on any dollar earned by the second member of the household. Using replacement cost to calculate the value of the second earner’s production, to replace what the second earner is creating tax free at home, the second earner has to earn more and more pre-tax as the first earner’s cash income increases.

          For a very large range of incomes (say up to around $150k for the first earner) the second earner has to earn MORE than the first earner to replace his or more likely her self-created real wealth from staying at home. This is because marginal federal+state+socsec+medicare+salestax is around 50% on the second earner when primary income is $100-150k, and the value of the second earner’s directly produced goods and services is around $50-70k post-tax dollars as this is the cost to hire a full time nanny/housekeeper/cook and to purchase food at the convenient/expensive grocery instead of grow it, hire people to do odd jobs that could be done by the stay at home person like fixing toilets or painting a room or sitting around at the mechanic with the car while they change the oil or fix the air conditioner etc… just a wide range of stuff you can’t do easily if you’re 8-6 working and commuting at an upper middle class desk job.

          So, basically as women poured into the labor market in the 1970’s and 1980’s and the cost of real estate skyrocketed in the 1990’s-2000s, the real take-home after-tax value of working for cash has declined relative to the alternative of one member staying at home being relatively more self-sufficient creating wealth directly that they don’t pay taxes on. The result is that labor force participation peaked in about 1999 and men have steadily continued to withdraw from the labor market as women have leveled off their participation and even declined slightly relative to the peak during say 1995 to 2010

          https://fred.stlouisfed.org/graph/?g=fA6g

  8. By the way, Raj Chetty of Stanford has gotten his hands on actual IRS 1040 form data on income across two generations. Here’s my summary of his 2015 write-up, which shows a little more regression toward the mean than I would have expected:

    One of his tables focuses on families that averaged in the lower half in 1996-2000 and the other in the upper half. The median family in the bottom half was, of course, at the 25th percentile in income in the 1990s, while, due to regression toward the mean, their typical offspring is by 2011-2012 at the 41st percentile for his or her age group.

    The teenage dependents of families in the top half (a.k.a., 75th percentile) in the later 1990s have typically regressed as 30somethings by 2011-2012 to the 56th percentile.

    http://takimag.com/article/moneyball_for_real_estate_steve_sailer/print#ixzz4x3eiBhNT

    Chetty has since released a redo in 2017 of this 2015 analysis, but I haven’t looked at it closely yet.

  9. … but doesn’t that presuppose that the parents stayed in this category? Lots of grad students are in the bottom third of U.S. income, and some of them have kids, but I assume that most of these parents, as well as their kids, will end up in the middle or even upper third once they graduate and get jobs. It’s also not clear, when they ask about “the time those children have grown up to be young adults, in their mid-20’s,” are they talking about income terciles compared to the entire U.S., or just people in their 20s?

    Spot on!

    There are all sorts of ambiguities about how income is measured here and what it means to transition from one income group to another. The ones Andrew mentions are some of the important ones, but only some.

    The authors are making the heroic assumption that all respondents understand income and mobility to be exactly as defined in the study to which their responses are compared. This is like asking random civilians how many electrons a cobalt atom should have and then criticizing cobalt for having too few electrons.

    BTW, the studies doing surveys about wealth distribution are even worse (e.g., Norton and Ariely).

  10. A 2011 article by Fareed Zakaria in The Washington Post, “The downward path of upward mobility,” concludes that “over the past decade, growing evidence shows pretty conclusively that social mobility has stalled in this country.”

  11. To follow up on the inability of the public to understand something as advanced as “percentages,” (much less provide percentages that add to 100) a much better technique would be a conjoint analysis in which people are given a set of changes from parent income to child income and ask to order them in terms of most likely to least likely. Ask them then to order several sets of such changes and then calculate the mobility probabilities that best explain the choices. People can rank going from $15,000 to $360,000 as more unlikely than going from $30,000 to $82,000 without knowing anything about probability. In addition, these questions can cover both upward and downward mobility.

    • I think the pbm is deeper than just whether one understands percentages or not. The pbm is in the idea that having an opinion on how society is organized, is a natural thing.

      • Well, sure, but that just addresses whether the answer you get, even if sincerely expressed, is meaningful. I’m addressing the situation where you have an opinion but you lack the mathematical ability to get the answer out in a sensible (even y your own lights) way.

        • I understood your post. What I tried to say is that that type of question presupposes the mathematical ability, no matter how you phrase it (cf. your suggestion). To say something about society in abstract terms (ie. income, educational attainment, or what have you) implies statistical thinking. You could argue that one can have (in fact, if I read your comment well, must have) a sense of percentages without necessarily having the means to express it in the way expected by soc sci surveys. This seems weird though. Now if you presented people with options like “mobility from this type of housing (with an illustration), or this type of job, to that one” you’d get meaningful responses but also a bunch of different questions (sorry I hope my english isn’t to unclear)

        • I think what Jonathan is saying is actually that huge numbers of people can’t even grasp the idea that if you break down some group by percentages that the percentages need to all add up to 100 or the like. Working with fractions/percentages in even the most basic way is often challenging for the median person in the US.

          My very limited experience on jury duty supports this idea to some extent.

        • abdulh said, “Now if you presented people with options like “mobility from this type of housing (with an illustration), or this type of job, to that one” you’d get meaningful responses”

          Would most people understand what you mean by “mobility from …” means? I’m skeptical that they would. (For example, some might thing you are talking about ability to travel easily from one type of housing to another.)

Leave a Reply to Daniel Lakeland Cancel reply

Your email address will not be published. Required fields are marked *