Research connects overpublication during national sporting events to science-journalism problems

[cat picture]

Ivan Oransky pointed me to a delightful science-based press release, “One’s ability to make money develops before birth”:

Researchers from the Higher School of Economics have shown how the level of perinatal testosterone, the sex hormone, impacts a person’s earnings in life. Prior research confirms that many skills and successes are linked to the widely known 2D:4D ratio, also knows as the digit ratio. This is the ratio of the index and ring fingers . . . research conducted by a team of scientists from HSE’s Centre for Institutional Studies (John Nye, Maria Yudkevich, Maxym Bruhanov, Ekaterina Kochergina, Ekaterina Orel, and Sergei Polyachenko) became the first study to use Russian data to show the link between the 2D:4D ratio and a person’s income. The study was published in the journal Economics and Human Biology. . . . The number of observations in the base regressions totalled nearly 700 for men and 900 for women, and the age of the subjects varied between 25 and 60. A 2D:4D ratio was made for each participant using a specialised apparatus. In addition, the respondents, whose identities remained anonymous, were asked a number of questions concerning income and salaries.

The results of the regression analysis showed a negative correlation between the income and 2D:4D ratios of women. In other words, the higher the salary, the lower the ratio. The effect was negative even when taking into account salary predictors such as gender, age, education level, job position, and the position’s economic sector. What is interesting is that this quantitative association is seen in men as well, though only after taking into account respondents’ level of education.

Savvy researchers will (a) note the challenge of taking gender into account when the analysis was performed separately for each sex, and (b) the forking-paths and difference-betweeen-significant-and-not-significant aspects of the last sentence above. Other fun things that you can see by following the link to the original paper is that the researchers looked at three different outcome measures and also tried everything with left and right hands. Also, despite the first sentence of the press release, and despite the title of the paper (“The effects of prenatal testosterone . . .”), there are actually no measurements of prenatal testosterone involved in this research (let alone any causal identification).

Bad form to put something in the title of the paper that’s not actually being measured.

I should perhaps emphasize that I have no objection to people researching such things and publishing their results; it’s just that it’s an absolute disaster to rummage around in a pile of data, pulling out things based on their statistical significance level. To put it another way: lots of their apparently statistically significant comparisons will be noise, and lots of the things that they dismiss as not statistically significant can actually correspond to real correlations. All this is made worse by rampant selection (for example, on the three different outcome measures) and the casual slipping from finger length to (unmeasured) hormone levels to (nonidentified) causal effects. What they have is some data, and that’s fine, and I think they’d be better off just publishing their dataset and being more aware of how little they can learn from it.

And of course you don’t have to be John D. Rockefeller IV to realize that “One’s ability to make money develops before birth.” No finger measurements are necessary to discover this obvious social fact.

P.S. I also love the very last paragraph of the press release:

Disclaimer: AAAS and EurekAlert! are not responsible for the accuracy of news releases posted to EurekAlert! by contributing institutions or for the use of any information through the EurekAlert system.

All right, then.

I went to the main Eureka Alert site and found this gem: “Research connects overeating during national sporting events to medical problems,” which begins:

People who overeat during national holidays and national sporting events – like this weekend’s Super Bowl – are 10 times more likely to need emergency medical attention for food obstruction than any at other time of the year, according to a new study led by a University of Florida researcher.

Following along, I see this:

Most of the problems affected men, and most of the cases came during or just after the Thanksgiving holiday. . . . Over the study period, from 2001 to 2012, 38 people underwent an emergency procedure on the esophagus during or just after the holiday or sporting event time period (within three days of the event). Nearly 37 percent of those were due to a food impaction. Comparatively, of the 81 who had the same procedure two weeks before and two weeks after the event during the “control period,” just under 4 percent were due to food impaction. During holidays and national sporting events, the most common impacted food was turkey (50 percent), followed by chicken (29 percent) and beef (21 percent).

I just loooove how “people stuff their faces on Thanksgiving” transmutes to “overeating during national sporting events.” Tie-in to the Super Bowl, perhaps?

This last one indeed got picked up by some bottom-feeding news organizations, for example NBC Chicago which took the bait and led with the headline,

You’re Up to 10 Times More Likely to Choke on Snacks During Super Bowl, Health Officials Say

Suckers! Hey, NBC Chicago: you got played! Pwned by the University of Florida public relations department. Pretty embarrassing, huh?

But it’s all OK, because:

Disclaimer: AAAS and EurekAlert! are not responsible for the accuracy of news releases posted to EurekAlert! by contributing institutions or for the use of any information through the EurekAlert system.

On the plus side, NPR doesn’t seem to have fallen for either of these stories. So there’s some hope yet.

Why bother?

As always, the question arises, why bother even drawing attention to this sort of sloppiness? The quick answer is that someone sent me a link and I found it amusing. The long answer is that that the same research and reporting problems discussed above, also arise in more consequential areas. Remember that dude who was crunching numbers and said Hillary Clinton had a 98% chance of winning the election? Or those guys who said that early childhood intervention increased future earnings by 40%? Selection bias, confusions about causality, leaps that ignore differences between data and the underlying constructs of interest, and credulous reporting that treats every published paper as a Eureka discovery: These occur in problems big and small, and when studying statistical practice, statistical reasoning, and statistical communication, it can be helpful to study the many small cases as well as the few large ones.

18 thoughts on “Research connects overpublication during national sporting events to science-journalism problems

  1. “I just loooove how “people stuff their faces on Thanksgiving” transmutes to “overeating during national sporting events.””

    Paging Prof. Wansink. Maybe he can show that if you have tiny hands you’d have a lower risk of choking on a pretzel or other foodstuff.

  2. In around 2011, I once wrote, https://www.causeweb.org/wiki/chance/index.php/Chance_News_44#2D:4D,

    “If you put 2D:4D into a Google search, you will obtain 11,700,000 hits so you know that something is missing in your life if 2D:4D doesn’t ring a bell.”

    “the 2D:4D ratio is able to explain such disparate entities as sex and population difference, assertiveness, status, aggression, attractiveness, the wearing of rings, reproductive success, hand preference, verbal fluency, autism, depression, birth weight, breast cancer, sex dependent diseases, mate choice, sporting ability, running speed, spatial perception, homosexuality and more.”

    From http://www.pnas.org/content/106/2/623.full.pdf

    [Another] “startling conclusion is “We found that 2D:4D predicted the traders’ long-term profitability as well as the number of years they remained in the business.” More specifically, “traders with a lower 2D:4D would make greater long-term profits and would remain in the business for a longer period of time.” Numerically, “a trader in the lowest tertile of the 2D:4D range makes 11 times the P&L of a trader in the highest tertile.” With regard to experienced traders only, “low 2D:4D traders make, on average 5.4 times the P&L of high 2D:4D traders.”

    • Paul:

      PPNAS! That said, Table 1 in the linked paper looks pretty damn impressive. I do worry about the usual issues of selection, though. I don’t think we’d expect such large differences in a replication.

      • If that PNAS Table 1 is right, this effect if HUGE! Like, I’ve never seen anything that would be that good a predictor of earnings that isn’t, say, comparing high school dropouts with MBAs.

        I’m going to have to figure out how it is possible that relative finger length predicts almost an order of magnitude income differential within the same profession. Because my first thought is that this is just complete nonsense and something is measured wrong. It is just too big.

        • Yeah – agree on the small-N and right tail. But as for selection – I’d imagine there is a lot of selection into experience (meaning only successful traders stay around), but I don’t see how, conditional on being around that long, there is different selection between big/small finger ratios.

          But I think the first two things are enough. N=44 total, 22ish are less than 2 years experience, 7-ish people in each of 3 digit-ratio cells within experience bin. 7 people. 7 people with very high variance earnings (one of them seems to have lost money, and one made $8 million!)

          Here is a paper with much more data, graphed nicely at the bottom.

          http://bit.ly/2kwbmf2

          Relatively small effect, though the authors never, as best as I can tell, interpret it clearly (a 0.01 change in digit ratio is associated with a XX% change in earnings). And they go way over the top in their interpretation of gender differences. But at least they clearly show the raw data and best-fit lines/curves.

          I feel like this is a Brian Nosek situation. A real researcher who really believed these results would have pushed on it – this is easy, go measure a bunch more traders and see if it holds up. I mean, all they have to do is photocopy their hand*! This would be the finding of the century in the economics of human biology (or the biology of human economics?). But I guess that if PNAS wants it, you don’t f*** that up by actually checking whether it is true or whether your effects sizes make any sense at all.

          * “To determine 2D:4D, we obtained photocopies of traders’ right hands and measured digit length from the metacarpopha- langeal crease to the finger tip”

        • Jrc:

          By selection, I didn’t mean selection in who becomes a trader, I meant selection in what findings get reported in the research paper. The researchers could’ve seen all sorts of patterns that didn’t seem to make sense or didn’t fit their story. In the published article, all we see are the dramatic patterns that make a clean story. Which of course is what PPNAS is looking for.

        • Oh, selective reporting. Yeah, that makes much more sense. I mean, why would any “scientist” ever show results that didn’t perfectly align with their “story”.

    • Z:

      Indeed, this Rockefeller thing is part of the point. In addition to being shaky science, this digit thing seems to me to have a political angle of justifying inequality, with the idea that differences in income are, in some sense, biological or natural.

      • Andrew:

        “this digit thing seems to me to have a political angle of justifying inequality, with the idea that differences in income are, in some sense, biological or natural.”

        From the British Journal of Cancer

        http://www.nature.com/bjc/journal/v104/n1/full/6605986a.html

        entitled, “Hand pattern indicates prostate cancer risk” which begins with

        “The ratio of digit lengths is fixed in utero, and may be a proxy indicator for prenatal testosterone levels.” And
        concludes with “Pattern of finger lengths may be a simple marker of prostate cancer risk, with length of 2D greater than 4D suggestive of lower risk.”

        The reaction from

        http://ecodevoevo.blogspot.com/2010/12/loving-hands-or-go-finger.html

        “Of course, on the positive side, a glance at the hand is less embarrassing than a real DRE (digital rectal exam), or a PSA test to look for prostate cancer. It may be as useful, at least in terms of risk. It’s a lot cheaper. Of course the PSA testing companies are likely to resist this current interpretation.”

        • Table 2 shows risk estimates for fingers reported to be of approximately equal length and index finger longer than ring finger length and as compared with index shorter than ring finger pattern

          What is the obsession in medical journals with taking a continuous measure and mutilating it to fit some categories?

        • It’s not entire biological. Looking now at my left and right hands, I see the 2D:4D ratio is measurably different. And this seems to be related to how many injuries of various kinds I’ve endured on each finger.

      • Nye was a professor of mine in grad school. I can confirm that he, at least, is right/libertarian.

        Not that that’s uncommon in economics or an indicator of poor research in general. But, yes, this looks uninformative.

  3. As a note, checking the veridicality of these and other ‘findings’ is made more difficult by the cost upfront to read the actual article. The link trail above invites me to part with actual NZD$41.95 for junk science. Not the offer of the day! I’m not sure if my reluctance to do so is because of my 2D:4D ratio, or not. Perhaps journos have the same issues… and so rely on abstract surfing or something.

  4. The Boys From Brazil.

    Takes the idea and extends it to recreation of “key” environmental effects so combining nature with nurture to develop the next …

Leave a Reply to jrc Cancel reply

Your email address will not be published. Required fields are marked *