Skip to content
 

I’m skeptical of the claims made in this paper

Two different people pointed me to a recent research article, suggesting that the claims therein were implausible and the result of some combination of forking paths and spurious correlations—that is, there was doubt that the results would show up in a preregistered replication, and that, if they did show up, that they would mean what was claimed in the article.

One of my correspondents amusingly asked if we should see if the purported effect in this paper interacts with a certain notorious effect that has been claimed, and disputed, in this same subfield. I responded by listing three additional possible explanations from the discredited literature, thus suggesting a possible five-way interaction.

I followed up with, “I assume I can quote you if I blog this? (I’m not sure I will . . . do I really need to make N more enemies???)”

My correspondent replied: “*I* definitely don’t need N more enemies. I’m an untenured postdoc on a fixed contract. If you blog about it, could you just say the paper was sent your way?”

So that’s what I’m doing!

P.S. Although I find the paper in question a bit silly, I have no objection whatsoever to it being put out there. Speculative theory is fine, speculative data analysis is fine too. Indeed, one of the problems with the current system of scientific publication is that speculation generally isn’t enough: you have to gussy up your results with p-values and strong causal claims, or else you can have difficulty getting them published anywhere.

12 Comments

  1. Tyler Ransom says:

    I don’t see a link anywhere on here.

  2. Bill Jefferys says:

    Andrew, did you intend to link to the paper? I don’t see a link.

  3. Andrew says:

    Tyler, Bill, etc.:

    Yes, this was a real email I received and there was a real paper. The paper was pretty useless, though, so I didn’t see any reason to link to it.

    • Dan Simpson says:

      Given the publication lag, I occasionally wonder if you remember what your post is alluding to :p

      • Andrew says:

        Dan:

        I don’t remember this one, but I could search my email to find out. If I cared, which I don’t.

        • Anoneuoid says:

          I agree there is no point in even looking at the original paper most of the time. It is like focusing on one drop of water in a tidal wave of crappy studies, almost as if these things exist just to waste the time of productive people.

          But why not write more about the kind of stuff that has an influence on our lives? Eg, a new one to me was the origin of the 120/80 blood pressure rule, which is based completely on “forking paths”: https://jamanetwork.com/journals/jamainternalmedicine/article-abstract/545007

          Actually I like that paper but don’t really see why 120/80 was any better a choice than the earlier 140/90 or (seems to me to be guaranteed future) 110/70 followed by 100/60…

          • Clyde Schechter says:

            Well, really the underlying basis for this should be decision-theoretic. Basically, until we get down to blood pressures so low you can’t stand up, lower is better, or at least no worse. Lower blood pressure is associated with decreased risk of coronary artery disease, hypertensive heart disease, renal failure and stroke. There is no reasonable threshold below which this relationship ceases to hold. Lower is better (until we get to levels lower than anybody contemplates using as thresholds).

            So if there were some completely harmless drug that could be applied across the board to lower blood pressure, we would throw away our sphygmomanometers and put the drug in the drinking water. But there isn’t, and it’s not likely there ever will be. All existing and past blood-pressure lowering drugs have a fairly high associated incidence of minor side effects, and some of them have an appreciable incidence of serious ones. So it’s a trade off. At what level of blood pressure does the incremental harm of more intensive treatment outweigh the benefit? This trade off is complicated, in part because the harms and benefits are of different natures, and the harms occur quickly, whereas the benefits accrue later.

            It’s also made complicated by the fact that a person’s blood pressure is varies greatly over time, and the measurement process, at least as usually done clinically, has a great deal of random error.

            So, in the end, you need a pretty complicated utility function, and maximizing its expectation is not a trivial task. It is really quite amazing that doing it by the seat of our pants, as we have done for decades now, we seem to have actually achieved more good than harm (though, no doubt we could do somewhat better with better analysis).

            And I haven’t even raised the question of whether some sort of “general public” utility function should be used or if each patient should develop and use his or her own.

            • Martha (Smith) says:

              I would guess that blood pressure is a situation where lots of “N of 1″ studies are needed. I suspect that generalizations, including Clyde’s assertion, ” Basically, until we get down to blood pressures so low you can’t stand up, lower is better, or at least no worse” don’t take into account individual variation and secondary effects. For example, I don’t think I’ve ever had blood pressure so low that I couldn’t stand up, but I’ve often had blood pressure so low that there is brief blacking out when I stand up quickly (pulling in abs does help some, though), and I wonder if having this type of temporary low blood pressure in the brain often might lead to some brain impairment in the long run. But then, high blood pressure can also impair brain function, so Clyde’s last two paragraphs do make a lot of sense.

            • Anoneuoid says:

              In general I tend to think that if improving our understanding is making a phenomenon seem more complicated, something is wrong. Perhaps the concept of “a resting blood pressure” is a flawed base to being with. A few things:

              1) You can easily see minor changes in posture, prior activity, mental state, and whatever “unknowns” affect blood pressure +/- 20% in a few minutes, enough to span multiple clinical categories…

              2) The “optimal” blood pressure seems to be the lowest that still allows sufficient perfusion of all the tissue in a body.

              Perhaps what we should really be concerned with is “sufficient perfusion”, which depends on factors like amount of tissue, metabolic efficiency of that tissue, and probability the body will need to suddenly jump into action of one type or another (running, digestion, fighting infection, etc).

Leave a Reply