Skip to content
 

Quick statistical comment

A reporter pointed me to an article to be published in a scientific journal and asked if I thought the statistics were OK.

I took a quick look and replied:

I did not look at the paper in detail but it seemed reasonable to me. The only part of it that I would not take seriously is the paragraph which starts, “We considered two possible alternative explanations…” Here they’re playing the usual game of sifting through statistical significance to tell stories, and that’s a mug’s game.

But the person who you should really ask about this study is **, who is an expert on ** and has studied topics such as **.

I’m sharing this with you just to give a sense that it’s possible to offer partial criticism, expending a small amount of effort and helping a small amount.

20 Comments

  1. Jack PQ says:

    OK, but this is a drop in the ocean… As you argue elsewhere, the right answer is to tell journalists that no peer-reviewed paper is perfect, and no single finding–even peer-reviewed–is definitely true. But indeed as you write, it’s also crucial for the journalist to talk to an expert in field X and ask him or her if the finding is plausible or not. This way the journalist can write about finding XYZ being intriguing and supported/not supported by the broader field.

    • Andrew says:

      Jack:

      Yes, my main contribution was to point the journalist to the expert. But it’s a bit more than that, as the journalist himself wanted to know if there were any obvious statistical problems, and so I could take a quick look.

      • Nick says:

        I’ve sometimes wondered whether this could be a compulsory step in the grant/peer review process – before we fund/publish you, please walk over to the statistics department and get someone to glance at your paper…

        • Physicist says:

          Your average physicist probably does statistics better than your average statistician, so this is terrible advice.

          • Andrew says:

            Physicist:

            We’ll make you guys go over to the psychology department, just to make you really suffer.

          • Keith O'Rourke says:

            The believe the training of statisticians to work in the wild (with others doing actual research in a high miss-incentivized setting) is improving, though the _average anyone_ likely does little more good that harm in (challenging) research.

            Conferring with a _statistician_ has become close to compulsory in some settings (clinical research grants) so one could try to discern if its done more good than harm in those settings.

            • Keith O'Rourke says:

              Opps, the believe -> I believe.

            • Clark says:

              There does indeed seem to be a lot more conferring with statisticians going on in the clinical research realm, both grants and journals often require it. Sometimes it seems that the researchers are playing lip service to the grant requirement, in that they’ll get the statistician to help prepare the methods in the grant, but once funded will go off and do whatever they like as opposed to following the methods. I’ve realized lately that I need to start having a conversation in which I say something like, “once you get funded, REALLY let me help you with the randomization and analysis…”. I know one big research group that seems to think that blinding is just something you say in a proposal, but can be ignored if someone needs a publication on their CV.

              Cynical as I am, I have seen real progress over the past decade. Things are genuinely getting better.

              • Keith O'Rourke says:

                I think they are (from second hand info) but one thing I recall was that the funds that were supposed to be set aside for statistical analysis and support very often were spent on something else.

          • Jonah says:

            > Your average physicist probably does statistics better than your average statistician, so this is terrible advice.

            That’s a joke I assume? I know quite a few physicists and the only ones I would trust to give statistical advice are now referring to themselves as statisticians (and _former_ physicists). Not that the quality of “your average statistician” is necessarily so high, but most of the practicing physicists I know would benefit from consulting with statisticians. That’s just anecdotal evidence but if anything my sample of physicists is probably biased towards the more statistically savvy.

            • Keith O'Rourke says:

              Some physicists probably do statistics better than most statisticians likely is not a joke.

              I don’t think much is known about the distribution of abilities (by field of application) of folks who could identify in good faith as statisticians to a granting body.

              By the way the group I found the most challenging to work in my career with were bio-physicists (even sometimes who were friends of the family)…

          • Aaron G says:

            Physicist:

            Cosma Shalizi (a physicist-turned-statistician, at Carnegie Mellon) will most certainly strongly disagree with you.

            http://bactra.org/weblog/297.html

            http://bactra.org/weblog/361.html

      • Jack PQ says:

        What I meant is, you can’t win at this, and your time is too valuable: think of the opportunity cost. Give a journalist an answer and he’ll write up the article correctly, etc., but teach the journalist to ask the *right* questions and he can do his job right for years and not bother you anymore.

        • Andrew says:

          Jack:

          Sure—of course I guess we could say this about responding to blog comments, too!

          Seriously, though: The total amount of time I spend responding to journalists is pretty little, and when I do all these little things I get a sense of what to look for in reading a paper. This maybe will help me write the article/book that journalists in the future can use to figure out what questions to ask.

          • John Mashey says:

            It is well worth investing modest amounts of time to help a journalist who actually tries,
            but indeed, best thing was pointing at the most relevant expert.

            Journalists who try to check are not as numerous as they might be.

  2. Surely: “We considered two possible alternative explanations…”

    …looks like they were just trying to find an explanation for the observations?

  3. Ahmed says:

    What is “mug’s game”

  4. numeric says:

    It might be nice to put a link in to the actual paper (or if unpublished, the title and scientific journal and the approximate publication date). I’m sharing this with you just to give a sense that it’s possible to offer partial criticism, expending a small amount of effort and helping a small amount.

Leave a Reply