Skip to content
 

Note to journalists: If there’s no report you can read, there’s no study

Blogger Echidne caught a study by the British organization Demos which was reported in Newsweek as “Half of Misogyny on Twitter Comes From Women.” But, as Echidne points out, there’s no study to report:

I [Echidne] then e-mailed Demos to ask for the url of the study. The rapid response I received (thanks, Demos!) told me — and here comes the fun part! — that there IS NO WRITTEN REPORT THAT PEOPLE COULD ANALYZE.

That is bullshit. Absolute bullshit. . . .

That there is no report does not imply that the results are incorrect, only that we cannot tell if they are correct or incorrect. But a written report is very important. The reason that researchers write their studies up is so that others can see what they did, how they did it, and also so that others can judge whether the study was done right or not.

I agree. I’m reminded of the gay gene tabloid hype, where results presented in a 10-minute conference talk were promoted all around the world, without there being any paper describing the data and methods.

Or the Wall Street Journal article that reported on a claimed survey of the super-rich for which no documentation was provided and which we have no reason to trust.

Hey, journalists: Don’t get fooled. Demand to see the study! I think it would work.

Next time someone sends you a press release and you’re thinking of running the story, first contact the organization and ask to see the written report. If they say they don’t have a report, it’s simple: Either don’t run the study, or run a report that is appropriately dripping with skepticism, including the phrase “for which the organization refused to supply a written report” as many times as possible.

15 Comments

  1. Dan Hicks says:

    It looks like Andrew wrote this post back in May, when the press release first came out. Demos now has a short methods and results writeup here (linked from the target of “this study” above): http://www.demos.co.uk/wp-content/uploads/2016/05/Misogyny-online.pdf

  2. Dale Lehman says:

    I don’t know if their matieral was added after the criticism was noted, but to be fair to Demos, their site does provide a writeup of the study that is being reported. The methodology is described in some detail – in fact, in enough detail to find things to critique (such as the likelihood that they overfit their models – basing their algorithms on a relatively small test data set which was used to adjust the algorithms). So, while I agree with Andrew’s major point here, it may not be totally fair to point to this study as a prime example.

  3. Thomas says:

    I have a bizarre version of this kind of thing. At the beginning of the year, a study sponsored by the Committee for the Status of Women in Astronomy made a lot of news. All inquiries I made about methodology & interpretation were ignored by the authors, but others told me (on Twitter) that a paper was going to be published in the spring (of this year). The rumor (spread by one the authors on Twitter) now is that it is being held up in the review process at PLOS ONE. The truly amazing thing is that errors that I and others pointed out, working on the news coverage and the published slides, were corrected by altering, without marking the correction, in the published slides and without acknowledging our efforts. There is still no paper or report one can read to better understand “results” that have been circulating since November of last year.

    • Thomas says:

      (Sorry, broken link to the tweet. Here is the right one.)

      • Andrew says:

        Thomas:

        I don’t understand the bit about being under review at Plos-One. Does that journal have a policy that you can’t post online material that is under review? I post almost all my papers online before submitting them to journals, so if someone asked me about such a paper, I’d just point them to where the preprint is posted online.

      • Andrew says:

        P.S. Like you, I find it annoying when people alter what they wrote in response to criticism, but without acknowledging where the correction comes from. A few years ago I screamed at Gregg Easterbrook’s editors at Reuters for this behavior.

        • Thomas says:

          Yes, that’s annoying. But remember that here not even the change was noted in the document that was changed (which people had linked to to support their numbers.) For me, that’s on the list of “the worst things you can do” along with plagiarism and making up your data.

    • Keith O'Rourke says:

      Maybe on a another topic – nicely put – “Let your passion to know, not to master, be the core of your identity. And choose your mentors by how they they stimulate your curiosity, not your ambition.”

      “Geoff Marcy was naive to think he could have friendships with his students.”
      There are always risks but benefits I believe mostly to the befriended.

      Now, when I was a student I had three (or more?) inappropriate befriendings – two by female tutors and one male professor (hey it was the 1980’s). It does make anything they said about you suspect – but not necessarily untrue. It is uncomfortable. I don’t think in general you can’t be made safe in relations with other people and the benefit of what others thought and meant should be with others. Actions under which motives are not unclear are needed to “blame”.

    • Thomas says:

      I’ve decided to just tell the whole story in its details on my blog for those who are interested.

  4. zbicyclist says:

    A press release that promises a juicy headline dangled in front of a reporter needing a story is like dangling a treat in front of a dog. Asking them to resist isn’t realistic.

    On the other hand, “for which the organization refused to supply a written report” makes them sound like careful journalists, and has a better chance of happening.

Leave a Reply