Science journalist recommends going easy on Bigfoot, says you should bash of mammograms instead

Paul Alper points us to this transcribed lecture by John Horgan. It’s a talk Horgan gave to a conference on Science and Skepticism, which began:

I [Horgan] am a science journalist. I don’t celebrate science, I criticize it, because science needs critics more than cheerleaders. I point out gaps between scientific hype and reality. That keeps me busy, because, as you know, most peer-reviewed scientific claims are wrong.

Following the links, I also came across this bit by Horgan, from a post entitled, “A Dig Through Old Files Reminds Me Why I’m So Critical of Science”:

I [Horgan] keep struggling to find the right balance between celebrating and challenging alleged advances in science. After all, I became a science writer because I love science, and so I have tried not to become too cynical and suspicious of researchers. I worry sometimes that I’m becoming a knee-jerk critic. But the lesson I keep learning over and over again is that I am, if anything, not critical enough. . . .

The vast majority of scientists and journalists who write about science—not to mention the legions of flaks working at universities, science-oriented corporations and other institutions—present science in a positive light. My own journalistic shortcomings aside, I believe science has been ill-served by all this positivity.

I agree. It doesn’t help when credentialed scientists, recipients of huge levels of public funds and publicity, issue pronouncements such as “the replication rate in psychology is quite high—indeed, it is statistically indistinguishable from 100%.” Or when the National Academy of Sciences puts its seal of approval on laughable papers on air rage, himmicanes, etc. Or when Lancet publishes with a straight face an unregularized regression with 50 data points and 39 predictors. Or when leading news organizations such as NPR hype this sort of work.

It’s hard not to be a skeptic when so many leading institutions keep trying to keep following business as usual.

The controversial parts of Horgan’s speech, though, were not where he criticized picayune bad science—he didn’t mention “power pose” and Ted talks, and he could’ve, but that all wouldn’t’ve been so relevant to his audience, who I think are (rightly) more interested in the big questions (How does the universe work? Why are we here? Etc.) than in the daily life of the scientific community.

Rather, controversy came from his recommendations of where skeptics should aim their fire. Here’s Horgan:

You don’t apply your skepticism equally. You are extremely critical of belief in God, ghosts, heaven, ESP, astrology, homeopathy and Bigfoot. You also attack disbelief in global warming, vaccines and genetically modified food.

These beliefs and disbeliefs deserve criticism, but they are what I call “soft targets.” . . .

Meanwhile, you neglect what I call hard targets. These are dubious and even harmful claims promoted by major scientists and institutions. In the rest of this talk, I’ll give you examples of hard targets from physics, medicine and biology. I’ll wrap up with a rant about war, the hardest target of all.

Rather than getting into the details of Horgan’s argument with Pinker and others (go to Horgan’s post and search on Pinker for lots of links, including this fascinating story involving Marc “Evilicious” Hauser, among others), I thought it could be helpful to attempt some sort of taxonomy of science, or pseudo-science, that could be criticized. Then we can se how Horgan and the organized skeptic community fit on this scale.

So, here are various scientific or pseudo-scientific beliefs that get criticized:

1. Crackpot ideas: Bigfoot, perpetual motion machines, spoon bending, etc. These are ridiculous notions that educated people talk about only because they really want to believe (Conan Doyle and those obviously faked photographs of fairies) or perhaps for some political or quasi-politcal reason (Freeman Dyson and Velikovsky).

2. Ideas with a religious connection: Jesus sightings, Dead Sea crossings, Noah’s ark, creationism, anonymous prayer, etc. These ideas make zero sense on their own, but can be difficult to dispute without offending millions of religious fundamentalists. Perhaps for some people, offending the fundamentalists is a goal in itself, but in any case these ideas have baggage that spoon bending, etc., do not.

3. Pathological science: ESP, N-rays, cold fusion, and any other phenomena that show up when studied by the proponents of the idea, but which outsiders can never see. The work of Marc Hauser is an extreme example here in that he actively tried to stop others in his lab from questioning his data coding.

4. Junk science: power pose, air rage, ovulation and voting, beauty and sex ratio, himmicanes, etc. These ideas don’t contradict any known science but have been studied in such a sloppy way that nothing can be learned from the research in question. This work is junk science in part because of the very weak connection between theory and measurement.

5. Politicized questions: effectiveness of charter schools, causes of global warming, effects of a higher minimum wage, etc. These are issues that, at least in the U.S., are so strongly tied to political affiliation that it seems that one can only have discussions on the specifics, not on the bigger questions. Sometimes this leads to actual junk science (for example, claims about negative effects of abortion, or Soviet-era claims about the effectiveness of planned economies) but lots of this is solid science that’s hard for people to handle because they don’t want to hear the conclusions.

6. Slightly different are theories that are not inherently political but which some people seem to feel very strongly about, ideas such as vaccines and autism, various theories about diet. Also somewhere in here are belief systems such as astrology and homeopathy that follow some of the forms of science but sort of live in their own bubbles.

7. Unfalsifiable theories (what I call frameworks): Marxism, racism, Freudianism, evolutionary psychology, neoclassical economics, etc. All these frameworks are based on real insights and observation (as well as hopes and biases) but can get taken way beyond any level of falsifiability: for their enthusiasts, these become universal explanations of human behavior.

8. Mathematical models: Here I’m thinking of string theory, which Horgan disses because of being undetectable by experiment. I put these in a different category than item 4 above because . . . ummmm, it’s hard to explain, but I think there’s a difference between a mathematical theory such as superstrings, and a catchall explanation machine such as racism or Freudianism or neoclassical economics or whatever.

9. Hype: This could be any of the categories above (for example, the “gay gene”), or simply good science that’s getting hyped, for example solid research on genetics that is being inappropriately sold as representing a cancer cure just around the corner.

10. Misdirection: Horgan and others are bothered by the scientific establishment spending zillions on Big Science without attacking real problems that face the world. As we used to say, How come they can put a man on the moon but they can’t cure the common cold? This line of reasoning might be wrong—maybe Big Science really is the way to go. I’m just listing this as one of the ways in which science gets criticized.

11. Scientific error. This can get controversial. Horgan criticized (and believes skeptics should criticize) “the deep-roots theory of war,” on the grounds that “the evidence is overwhelming that war was a cultural innovation—like agriculture, religion, or slavery—that emerged less than 12,000 years ago.” On the other hand, various other people think there is strong evidence for deep roots of war. It’s hard for me to judge this one, so all I’ll say is that this kind of thing is different than items 1-7 above.

Different skeptics have different tastes on what to criticize. Horgan things we should be slamming items 8, 9, 10, and 11. I spend a lot of time on items 3, 4, and 9. Items 1 and 2 don’t interest me so much, perhaps because they’re pretty much outside of science. Various commenters here think I talk too much about item 4, but those cases interest me in part because of the statistical insight I can draw from them. Item 5 is super-important and it comes up on this blog from time to time, but I’m never quite sure what to say on such issues because it seems that people have already decided what they’re going to think. Item 6 is kind of boring to me but it’s a traditional topic of skeptical inquiry. Finally, items 10 and 11 are huge; I don’t talk that much about them because I don’t have much particular expertise there. Except for some specific areas in political science where I’ve done the research: I really do get annoyed when political pundits such as Michael Barone garble the connections between income and voting, or when know-it-alls such as Steven Levitt go around saying it’s irrational to vote, or when people who are know just enough math to be dangerous but not enough to really understand things go around saying that voters in large states have an electoral college advantage. These items are pretty small potatoes though, so I don’t spend too much time screaming about them.

P.S. I doubt that anyone’s actually talking about bigfoot anymore. But I do think that skeptics target the items that get the hype. Back in the 1970s, we really were hearing a lot in the media about bigfoot, the Bermuda triangle, Biblical relics, and ESP, so it makes sense that these topics were investigated by skeptics. In the 2005-2015 era, major news media such as NPR, Gladwell, Freakonomics, Ted, etc., were talking about psychology pseudoscience, and this attracted skeptics’ attention. Horgan’s point, I suppose, is that by always following the bouncing ball of what’s in the news RIGHT NOW, we’re neglecting the big chronic issues.

83 thoughts on “Science journalist recommends going easy on Bigfoot, says you should bash of mammograms instead

  1. Great post! A tree might be useful to present the main branches, and then subspecies. It is interesting that the focus is shifting away from “pure” junk science (Martin Gardner’s classic books criticizing all that stuff) to “junky” science (badly done for various reasons, and not to be believed, even though it appears to be the real stuff).

    Why do you think neoclassical economics is unfalsifiable? We can test that demand curves slope down, that prices reach equilibrium, that people’s choices reflect indifference curves, that utility is marginally decreasing… indeed we can test this against an alternative framework such as one of the Psychology & Economics models (Kahneman & Tversky etc.).

    • Your examples are solely in Microeconomics, which are not limited to neoclassical. The distinctiveness of Neoclassical Economics is a simple extrapolation to Macroeconomics with a whole host of underlying assumptions about different effects in Economics, (wealth vs. income effects is a good example) where it becomes more of a moral framework rather than giving testable predictions.

      • “Inflation is always and everywhere a monetary phenomenon” is a testable predictions. As is the non-neutrality of money in the short run and the neutrality of money in the long run. Similarly, that a rising capital – labor ratio is responsible for a chunk in the cross-country differences in income is also testable.

        • When people think of neoclassical economics, they don’t think of these very narrow issues which may be testable. They think, “the market is self-correcting,” “higher tax rates are bad,” or “regulations are bad.” Those are the things that Horgan is likely thinking here.

        • The market is self-correcting is testable. There is work by Franklin Fisher that could be used to make testable predictions that could be used to argue that the market is self-correcting.

          Higher tax rates are bad, and regulations are bad, are not part of (positive) economics. These are normative statements that could just as well be part of religious doctrine. What economics can do is to tell proponents of higher taxes and more regulations that these measures are self-defeating or that these are measures that will not attain the end sought.

        • None of that is neoclassical economics. Those are economic policy positions or beliefs, and they are indeed testable if you frame it more precisely than that. So those examples in no way mean that neoclassical economics is not testable.

        • Neoclassical economics is the mainstream of economic theory after the marginal revolution. If you or Horgan think it’s something else, that’s just your own ignorance.

    • I didn’t understand that either. What I think that is referred to is that the core assumptions of economics are not testable (see Lakatos). This, however, holds for pretty much any theory that purports to explain the world. We have core assumptions and auxillary assumptions. The latter are testable, the former are not.

      I suppose that this is why Andrew refers to it as frameworks.

  2. Can’t contradict any of your points but personally a bit sad to see that you are classifying ESP as “pathological” :) Although I am deeply thankful for the creation of the most powerful weapon to fight with pseudo-scientists, sometimes I am wondering if wiping them out from realms of academia is what Popper really hoped for.

  3. John Horgan’s article is worth reading in full. Here is an example of what he writes:

    “Another hard target that needs your attention is behavioral genetics, which seeks the genes that make us tick. I call it gene-whiz science, because the media and the public love it.”

    Here is another:

    “Over the past half-century, physicians and hospitals have introduced increasingly sophisticated, expensive tests. They assure us that early detection of disease will lead to better health.
    But tests often do more harm than good. For every woman whose life is extended because a mammogram detected a tumor, up to 33 receive unnecessary treatment, including biopsies, surgery, radiation and chemotherapy. For men diagnosed with prostate cancer after a PSA test, the ratio is 47 to one.”

    And possibly to the surprise of many, “Similar data are emerging on colonoscopies and other tests.”

    By the way, if you go to the article today, on the right-hand side you will find a picture of the handsome Rick Perry with the headline

    “Rick Perry Tapped to Run the Energy Agency He Once Vowed to Kill”

    https://www.scientificamerican.com/article/rick-perry-tapped-to-run-the-energy-agency-he-once-vowed-to-kill/

  4. Andrew –

    Per Horgan’s “Criticize Bigfoot less, Mammograms more” mantra. He’s convinced he’s found some deep “leaks” (as we say in poker) in the skeptical community, but has he ever shown it? One point – it takes almost no expertise to debunk Bigfoot. Takes a lot more to understand the statistically literature behind mammograms.

    Ok, if you take take 100 skeptics at random without controlling for educational background, maybe there is a larger proportion of them focused on bigfoot. But I don’t know that he’s shown this. What would happen if we looked only at skeptics with an MD or PhD. Or we could normalize for education, and also look at what % of their output is focused on bigfoot vs mammograms-related topics. As representative examples – would Steven Novella and David Gorski confirm Horgan’s intuition? My guess is no.

    Has Horgan ever done anything like this or anything quantitative with this thesis? As long as the rate of skeptical articles on bigfoot is not zero, it seems he can always claim there’s a problem. Or maybe he is suggesting even one is too many? Either way, without some real analysis about the magnitude of the effect or harm, isn’t his thesis not that useful?

    • >”Or maybe he is suggesting even one is too many?”

      It kind of is. There is really no justification for skeptical communities to waste time/space on stuff like bigfoot and psychology when the biomedical community is still struggling to use basic tech like
      1) blinding
      2) reporting basic methodological info such as sample size (rather than n = 5-10, etc)
      3) citing each other accurately
      4) using scatter plots and box plots (rather than dynamite plots or the even worse table equivalent filled with means, sds, and p-values)
      5) independent replication

      It does not take an expert to point these things out. I am noticing that over time I have been lowering the bar for that community, just to give them a chance of making some progress in my eyes. It may be too soon to talk about more substantial issues like coming up with real predictions to test rather than a “null hypothesis”, so a skeptic with a more “lay” background can play an important role here.

      • OK. The internet is a big place, and people write about whatever they find interesting. .

        Of course all the basic methodologies of science can and should be refined. But I don’t follow the logic that we are making less progress than we should because anyone with a computer can call themselves a skeptic and write about bigfoot. By this logic, we can also blame these people for distracting us from the millions of infectious disease deaths each year. This is just a trivial way of saying what we already know is true – yes, if everyone focused on problem X, we could solve X faster.

        People have different roles and different expertise. Just because someone calls themselves a “skeptic” doesn’t mean they have some inherent responsibility to start making sense about blinding and scatter plots. Anyone can call themselves a skeptic and come to a conference. We shouldn’t not confuse the title “skeptic” as if it was analogous to “oncologist” with formal expectations and institutional barriers to becoming one.

        Now, if the MDs, PhDs, and experts populating our important institutions are spending lots of time on bigfoot, then yes, this is a problem. But I’m waiting for proof that it is a real problem, other than anecdotal evidence of a few self proclaimed skeptics who write about bigfoot on the internet. Are we worried there might be widespread oncology conferences where speakers are focused on bigfoot instead of mammograms? But the internet is a big place. And the problem with Horgan’s thesis is that it’s essentially unfalsifiable once you admit that the rate of skeptical articles on bigfoot will always be greater than zero.

        None of this is to deny the problem of bad science that gets through the system. These are systematic problems that will have some systemics solutions. But trying to get every self labelled skeptics to stop writing about bigfoot doesn’t seem like one of them.

        • I think you’re missing the point. The point is that every day non-science is committed by people like “MDs, PhDs, and experts populating our important institutions” but because this non-science is about topics like enocrinology, water pollution, or central banking policy it gets not only a pass but major funding. This article is a call for someone to debunk the BS that is called science these days.

        • Yes, there’s bad science done by smart people. The cure for this is better science, more awareness, better-designed institutions, more training, etc. And there’s a role for many people – experts, journalists, funding agencies, etc. Yes, we want good skepticism in general and good bullshit detectors.

          But what doesn’t logically follow is to suggest that we are not making enough progress because some skeptics write about things like bigfoot on the internet (part of Horgan’s thesis). By this reasoning, we should also call out people like sports-lovers who spend time crunching sports stats…because after all, they could be spending their time reviewing questionable scientific publications.

        • In a sense, isn’t this about optimum allocation of resources: e.g. Every hour Andrew spends on dissecting some silly power pose paper he could have instead debunked some dangerously bad clinical trial?

        • Andrew:

          Perhaps. But that reminds me of how people justify the everyday utility of NASA spending by pointing out inventions like freeze-dried food & shape memory foam that were spillover by-products of NASA research.

          God may very well be in every leaf of the tree, but when you really need Him, isn’t it easier to go to Church?!

        • Rahul: I think point is that you don’t have to trek off to the Church but simply touch the nearest leaf.

          Also, thinking hard about silly problems may be more productive for gaining insight than hard complicated problems with serious consequences.

  5. I’m not clear on who the audience of Horgan’s speech is. (I.e. who are “skeptics?”) If it’s scientists, then the complaint of aiming at the wrong targets can be explained (though not justified) by noting that all the “easy” targets are non-scientists, so one is bashing people outside of one’s tribe. If it’s science writers, it’s harder to explain, except that it seems (to me) that (i) science journalism is a dying field, (ii) science journalists seem, in my experience, to have a poor understanding of science, and (iii) science journalists want to be part of the science club, not the non-scientists club.

    • See Mary Mangan’s comment below. It’s an amateur community centered around publications like Skeptical Inquirer, and as you see, she considers herself part of it. To the extent Horgan is wrong about what “they” focus on, he just might not read all oftheir blogs. I think he’s right about Skeptical Inquirer itself.

  6. ” Mathematical models: Here I’m thinking of string theory, which Horgan disses because of being undetectable by experiment. I put these in a different category than item 4 above because . . . ummmm, it’s hard to explain”

    In these cases, typically the mathematics is all true. It’s not like the whole thing rests on logical errors in proofs or something. The thing that is wrong with string theory or various other mathy ideas is that there is no hard connection between the elaborate mathematical construct and actual observable facts about the world. It’s possible to talk about the volume of an 18 dimensional sphere even if there aren’t 18 dimensions in the universe, but it’s not possible to make claims about the universe based on the volumes of 18 dimensional spheres unless you can first show that there’s a relevant connection between the math and observable measurements of the universe.

    • There is also an important linkage between mathematical ideas and catch-all theory in the form of evolutionary explanations for human behavior. See evolutionary game theory and cultural evolutionary theory. Lots of math, very interesting results, very tenuous connection to reality.

      That said, mathy stuff is important in the sense that it helps to generate hypotheses. Some of these hypotheses are event testable! Unfortunately, garden-of-forking paths and inherent complexity and noisiness of social behavior, etc.

      • That’s one reason I think we ought to still distinguish between hypothesis generation versus hypothesis testing. Or exploratory versus confirmatory studies if you will.

        If a crappy theory accidentally or misguidedly generates a true hypothesis that shouldn’t matter so long as the hypothesis itself is true.

        • I’m slowly coming to the position that *everything* is always exploratory to some extent. Even when I do something that’s clearly confirmatory hypothesis testing, I am constantly exploring the data and/or the model. It has never happened to me that I loaded up my data and checked whether my hypothesis was confirmed or not. That’s not quite true: I’ve done that in the distant past and always regretted it because e.g. I failed to check that model assumptions were satisfied.

    • Well just to be clear string theory doesn’t “hide the dimensions” as some sort of awkward attachment to be buried. They’re actually *necessary* to model ordinary particle physics:

      In “ordinary fundamental physics” (i.e. quantum field theory) people specify a theory by dictating the type of particles and interactions it contains. String theory does not have this freedom: there is only one type of string, and the form of the interacting theory is uniquely implied by the free theory. The only thing the string can do is sit in space/vibrate differently. In order to create any kind of interesting particle physics, there needs to be a suitably complicated geometry on the string scale to provide many qualitatively different ways to “sit in space”, and so look like different particles when you zoom out. The extra dimensions provide this. They aren’t an *addition* to the freedom of investigation in particle physics, they’re a *replacement* for it. Even if 4D string theory *did* exist, a string propagating on flat 4D space wouldn’t stand any chance of being realistic and no one would study it. The extra dimensions are structurally important in reproducing known physics.

      The Standard Model equips particles with a 12 dimensional space as part of the basic (gauge) theory; this is the thing you hear about when people say the SM is based on U(1)xSU(2)xSU(3) (despite the names, SU(2) is 3D and SU(3) is 8D). In ordinary quantum field theory this ‘space’ isn’t *actual* space, but a collection of “internal dials” that specify the particle state. In string theory these degrees of freedom are merely being incorporated in actual space (and some other tricks to fit it in *only* six extra dimensions). This isn’t nearly as large of a change as people unfamiliar with particle physics like to claim. In some sense the change in string theory is not the existence of any extra geometry, but the change from many-dimensional “internal dials” to dimensions of external space. That the experimental distinction between these is subtle and not accessible to ordinary accelerators is not that surprising, nor is it an attempt at running away from falsifiability.

      ——-

      I can understand if someone is skeptical of the *entire* search for quantum gravity. “Yes, there’s a theoretical problem, but I doubt you can find any relevant insights without strong data”. Fair enough, though I’m not quite that pessimistic. What I don’t like is when people zero in on string theory, as if these problems somehow originate with it, or as if it’s a series of bizarre superstructures unconnected to prior physics. None of that is correct. The issues with falsifiability are inherent in the fact that quantum gravity is answering a purely theoretical question in the first place, and it’s very unlikely that taking a slightly different approach will make the problem go away! If someone whines about how string theory is unfalsifiable, and then tries to present you with *their* pet theory of quantum gravity (implying but never explaining how they avoid these problems), then they’re being dishonest. I would appreciate these discussions about potential problems of quantum gravity a lot more if string theory weren’t constantly the only example being taken to task. If someone tries to take that kind of narrow focus it strongly suggests that they don’t understand the actual problems being addressed.

      • Note, I was quoting the bit about string theory, and wasn’t referring to string theory specifically when I made my statements about 18 dimensional spheres or whatever. My point was more general, it’s possible to build a great mathematical model of an idealized thing that just doesn’t exist in the universe, whether that’s string theory, or a special kind of cyclic predator-prey model, or an explanation for the spatial distribution of earthquakes in terms of the mandelbrot set, or whatever.

  7. I think you’re trying to be way too fair to “left vs right” by including Neoclassical Economics and Evolutionary Psychology as “unfalsifiable” (both fields which are widely embraced & were created by lefty liberal types as well btw). There’s no equivalence with Marxism and Freudianism. The number of real insights you can gain from reading an evo psych (The Adapted Mind) or econ (Principle of Econ by Mankiw) textbook are massive compared to the negative utility of seriously reading Freud or Marx (Marx as economist not Marx as historian).

    • Vladimir:

      You seem a bit incoherent: (1) you’re criticizing me for balancing left and right; (2) you’re saying that all these theories were created by lefty liberals. If (2) is correct then your criticism (1) makes no sense.

      Anyway, I was not trying to say that Marxism, racism, Freudianism, evolutionary psychology, or neoclassical economics are empty. I think all these frameworks can give insights and can lead to falsifiable hypotheses as well.

      You’re pointing to another question, which is whether such theoretical frameworks can have negative utility. That’s an interesting issue but really it’s outside the scope of my post. After all, lots of real science can have negative utility too. Whether a particular field of endeavor has been good or bad for humanity . . . that’s a topic for a different post!

      • Maybe I was trying too hard to psychoanalyze your motives for juxtaposing those specific fields :P I just don’t think it’s fair to lump them in together, but I get what your WaPo article is trying to say about how they can overreach into being too totalizing.

        • Some day, we’ll look at modern theoretical economics (broadly “neoclassical”) and think about it the same way we think about Marxists and Freudians today – as people who were so deeply rooted in the truth of their “framework” that they stopped even looking at the world and turned completely inwards. If you don’t think neoclassical economics claims the same all-encompassing explanatory power over human behavior that Marxism or Psychoanalysis did… well, try to find any social phenomenon that hasn’t been modeled in terms of utility trade-offs. Just like every social phenomenon has been explained in terms of the sub-conscious or capitalism.

          Maybe one difference is that, in our times, trading historical or self-reflective reasoning for mathematical reasoning seems like becoming more “scientific.” But you can’t tell me that you learn more about human beings from complex set theory or functional analysis than you do from looking at the broad trajectory of human history or deeply interrogating your own motivations.

          Don’t get me wrong – I’m a proud economist, and I do very much see many decisions as being the result of internal, individual cost-benefit types of analyses. And I think there is value in that. But it is just a framework. And so is Evolutionary Biology, Marxism, or Psychoanalysis. And they will all fall by the wayside in the due course of human history, not because we’ve gotten “closer to the truth” but just because we’ll have new questions we want to answer and these approaches will fail to answer them well.

        • Utility tradeoff models answer questions where decisions are made “along the lines” of a utility tradeoff. What car to purchase, where to live, what educational program to pursue, which jobs to take, what kind of food to eat, where to go on vacation… all of these have very definite aspects of tradeoffs between multiple options with different monetary costs associated which are salient to the decision maker.

          People make other kinds of decisions every day, like what they want to queue up in their music list while folding laundry. You can shoe-horn that into a Utility tradeoff but, I don’t think it adds anything. So, I agree with you that utility tradeoff models overstep their bounds of applicability.

          But they do have a boundary of applicability that is pretty big and important…. It’s not as clear (to me) that Freudian Psychology *has* a boundary of applicability. I mean, what testable predictions does it make, and about what areas of human behavior? I don’t know, since I’m not an expert on the history of Freud (though I did read one of his books in high school so I’m not totally ignorant either…), but most of the stuff I think of about Freudian Psych is all about how people doing stuff have conscious motives they are aware of but really do those things because of “secret motives” deep inside their brains they’re unaware of. The existence of these deep motives is posited but unobservable, and there is no real theory of consistency of deep motives that could even allow us to show how some consistent unobserved thing is the best explanation for observable behavior. It has “Unfalsifiable” written all over it. Whereas for example we can see that price increases in fish result in more chicken dinners as predicted by utility tradeoffs… or whatever.

        • Evolutionary Biology will certainly never fall by wayside :P But if you meant Evolutionary Psychology, I don’t think that will either since it’s a scientific framework so it will evolve to be more and more accurate. Just like economics as econometric methods get better. Freudianism & Marxism were rotten to their core since the founders didn’t rely on the scientific method (doesn’t matter how much Freud pretended his case study generalizations were rigorous), but these other fields will remain even if their content changes. Like I get that all this NHST stuff is really bad, but I still don’t see how it’s comparable to just writing random words.

  8. I try not to think about this subject because it takes me to sad places. But to make this a taxonomy, you need to reduce it to categories to which you can link your descriptive list. So for example, to propose, you have perceptions and explanations. Many of your types are things we want to perceive or wish to perceive, whether fairies or Bigfoot or ESP. Whatever we all sense at the edge of our perceptions, it isn’t something that can be regularized into a formal and thus repeatable description. Others are explanations, such as the magical workings of the free market must be the right fix for education or healthcare or, with more animus, the Jews are responsible for all wars, etc. The third category is to me where nearly everything rational falls: science works by constructing ideas on observations that may be wrong, that are almost certainly materially limited in precision, in contextual meaning, etc. and then we revise. At least that’s the case when we do it right. More typically, we have an interplay in which perception mistakes generate and feed into explanation mistakes and these persist to deny progress. Or in law, mortmain, the dead hand of the past. It can be the persistence of “The Revolution” way past the sell by date for how its “principles” act in reality. It can be preventing change, from the Spanish Empire’s Inquisitional “Let no new thing arise” to the long running prohibition against dissection (or, perhaps more tellingly, the requirement that numbers be written out, as in twelve hundred fourteen florins and the mental model of the world that kind of idiocy represents).

    I think perception issues are inexcusable. If you argue for ESP without presenting some new model for how the bleeping thing works and how it can be regularly described, then you’re a fool. Explanation issues tend more to evil: those who seek to explain all the world’s working as they extend into future are a special kind of fool. Marx, for example, made a large number of interesting economic and social arguments but went ridiculously far off the rails when he became programmatic about the future. Tens of millions of people lost their lives because of that shit. But then there’s Newton: we can understand the interest in alchemy because next to nothing of chemistry was known and this was a way of exploring, and we can understand the historical interest in Biblical revisions and errors, but he certainly crossed the line when talking about prophecy. That is, we can predict where a celestial body may be but not what’s going to happen and part of him had a problem with that last part and I’d say it was because that is where his perception issue matched his explanation issue, that he could not perceive clearly what can’t be perceived clearly and yet he couldn’t recognize that – given Christianity of his time, that’s understandable to an extent – so he applied his powers manipulating stuff to no real purpose. In some cases, it isn’t that an idle mind lets the devil in, but that an over-active mind can spin off into nowhere land. (I think any intelligent person has experienced that.)

  9. Horgan, again? A terrible talk–which I heard live–followed by absolutely no effort to back it his claims, acknowledge his mistakes, or even efforts to begin to grasp what “skeptics” are as a community actually doing, and why.

    Bigfoot: no, people don’t spend a great deal of time on Bigfoot. But then out of the loony labs, the “Bigfoot genome sequence” suddenly appeared not long ago. Skeptics did use this as an opportunity to explain what was wrong with this”research”. It was a teachable moment about science and evidence. And it’s helpful to have some of the old-time Bigfoot skeptics and their institutional knowledge of the players in this nonsense. It gave context to the story, who was pushing it, and what their history was. A good science journalist might appreciate these things. Obviously Horgan is not in that class of humans.

    Mammograms: I have absolutely seen the medical skeptics take this on. His pretending that they didn’t (and refusal to acknowledge that they did) is just bizarre for someone who seems to think he’s evidence-based.

    Homeopathy: This is a real issue harming people now. Regularly people come to skeptic forums asking for help as their relatives spend money on useless treatments, sometimes in lieu of actual cancer treatments. It’s also all linked to dangerous other public health ideas like not vaccinating. I’m sorry if this isn’t on Horgan’s greatest hit list, but who is he to set the priorities?

    Religion: Yeah, sure, ignore religion. While the new US administration rolls into power with a woman who wants to give vouchers made of public tax dollars to teach creationism in classrooms. Sorry, but I think educating children on nonsense with tax dollars worth fighting.

    And many of these things are gateways to nonsense, just like we saw with Alex Jones’ legions and Pizzagate. Alex Jones sells herbal nonsense, chemtrail treatment potions, hates GMO and vaccines, and all of these things are linked to the other conspiracy nonsense. I don’t understand why Horgan isn’t capable of grasping this.

    Horgan’s hysterical attempts to get us to focus on his pet problem, rather than things he doesn’t find interesting, is really just bizarre and a giant waste of time. I can’t believe he was invited to give that talk. It was the least constructive thing I’ve ever seen at a skeptics conference. And the fact that it keeps popping up like a zombie is really just mystifying.

    • >>>to get us to focus on his pet problem, rather than things he doesn’t find interesting<<<

      As an aside, are we living in a world where any problem an academic may choose to study is axiomatically interesting?

      I think it is sad that we refuse to pass a value judgement. There really are areas that are over-hyped versus underfunded. But I think it has become politically incorrect to criticize a researcher's *choice* of the problem he selects to work on.

    • Horgan’s talk jibes very well with my own impression of “big-S Skeptics” being the boy scouts of science (girl scouts, too, of course). My experience is limited, though: it comes from reading some Skeptics’ blogs, watching videos of Skeptics’ conferences (e.g. NECSS, events in Switzerland) and from exposure to FB posts pointing to Skeptics’ articles.

      The sense I get from all this is that Skeptics are a group of eager science enthusiasts that want to do good by telling (sometimes lecturing) the unenlightened about the way science shows the world to be. I agree with Horgan that they focus on soft targets, where the science is clearer than in other areas. By choosing to do so, they give an overly optimistic picture of how well science works and how firm its knowledge is. That’s the excessive trust or “scientism” part.

      For me, then, the issue is not about telling others what they should study, or even trying to impose one’s one interests on them. The real issue is that Skeptics (again, as I encounter them) create a toy image of science. They mostly focus on science as an *ideal method*, and neglect science as it is actually practized. Since the gulf between them is wide, this is not just unhelpful, but counterproductive. If you think science is so great and people just need to f*** learn not to believe the bullshit that religions and snake oil salesmen tell them, all will be fine. We know that nothing is further from the truth.

      They are too sure. Excessive trust.
      They are a club that threatens the very independent thinking they should be promoting

      • Also, one troubling thing I noticed is that at least on FB, Skeptics have attracted a followership that uncritically re-posts anything that they blog. Paradoxically, this obstructs the very independent thinking that skepticism should stand for, and makes it into another herd movement (a bit like religion?).

        One example is the flood of very similar posts about “brain myths” that has been going around for a while, for example as the film “Lucy” came out and Skeptics bashed the idea that humans only use 10% of their brain’s capacity. I didn’t see any independent thinking on that one. In particular, the evolutionary arguments given against the idea were truly horrible. (I’ve blogged about this, follow the link in my name.)

  10. Very thought provoking post. I am concerned about your category 5. I know better than to talk about children rearing discipline or the gold standard at dinners with my family, and I will avoid these topics to prevent a shouted diatribe from Uncle Joe who thinks volume is evidence. However, if someone like you stands back from some politicized topics, doesn’t that cede the public fora to the Uncle Joes of the world.
    I did once tell Uncle Joe that I bought a Fiat 124 Spyder with my fiat money, but I could have gotten a Honda or a Toyota with it.

    • Slugger:

      Yes, category 5 is tough.

      One thing I will say is that partisan takes on politicized issues can be valuable, if the authors state their models clearly. So, for example, I value the writings of Paul Krugman, Dean Baker, Tyler Cowen, and Alex Tabarrok on the economy. Krugman and Baker are coming from one position, and Cowen and Tabarrok from another, but I can usually follow their arguments, so I can get something out of their reasoning even if I don’t buy all their assumptions and all their conclusions.

    • >However, if someone like you stands back from some politicized topics, doesn’t that cede the public fora to the Uncle Joes of the world.

      But Uncle Joe probably has expertise in some field that you have no knowledge in. Uncle Joe’s opinion on fiat currency doesn’t impact his skill in running a small business, valuing a car or diagnosing a patient. You could argue with Uncle Joe on politicized topics and prevent him from monopolizing the public fora, while at the same time, conceding Uncle Joe’s expertise in the other fields that you are not familiar with. This approach should preserve Uncle Joe’s ego, which is probably the main reason why the shouted diatribes even occur.

      This approach is flawed too, because it implies that Uncle Joe actually is *competent* at those fields. This assumption may be warranted and lead to the “Gell-Man Amnesia Effect” (http://www.goodreads.com/quotes/65213-briefly-stated-the-gell-mann-amnesia-effect-is-as-follows-you).

  11. > I doubt that anyone’s actually talking about bigfoot anymore.

    There was some talk about bigfoot in the financial press this summer:
    http://www.cnbc.com/2016/08/17/the-mystery-of-how-a-company-set-up-to-search-for-bigfoot-hit-a-10-billion-value.html
    http://www.bloombergquint.com/onweb/2016/08/18/bigfoot-riches-and-bank-activism
    https://www.sec.gov/Archives/edgar/data/1569568/000107878216003202/s1080516_s1.htm

    And it could have been a Nature’s cover story, if the journal was not so close-minded. How could they reject the following offer? “If there is still any doubt concerning the existence of these hominins, we are also willing to allow a representative from Nature to travel here and see these individuals personally, preferably during a full moon to increase lighting since they are primarily nocturnal. Though there is never 100% guarantee that a sighting will occur, if a few (about 3) days are allowed, we predict that the chances of seeing one of these hominins approach 99%.”
    http://sasquatchgenomeproject.org/sasquatch_genome_project_013.htm

  12. I’m with Mary’s assessment, Horgan’s article, as with most of his writing is self-aggrandising contrarianism and in this case demonstrated little engagement/awareness of the community he was lecturing too. I wrote a detailed (and probably too long) breakdown of the problems with his article here (http://www.patheos.com/blogs/cognitivedemons/2016/06/john-horgans-ideological-tribe/) and I think Steven Novella’s response at Neurologica is worth reading too (http://theness.com/neurologicablog/index.php/john-horgan-is-skeptical-of-skeptics/).

    For those not inclined to read the links a summary is that while Horgan does on occasion provide a valuable service in calling out bad science and over-hyped results, he tends to do so based on his desire to be contrarian and/or ideological preferences rather than a consistent commitment to good science. Hence, about just as often as he is criticising mainstream science, he is conversely misrepresenting research literature and overhyping particular researchers, cherry picked studies, and books to support his preferred conclusions which are almost always ideological/political rather than data driven (his position on the ‘ancient war’ literature is a very good illustration).

    He also displays a problematic tendency to be overly sympathetic with those he deems ‘establishment outsiders’ including conspiracy theorists and pseudoscientists, hence his generally positive assessment of the discredited biologist/pseudoscientist Rupert Sheldrake, famous for his psychic dog research (I kid you not!), which seems almost entirely derived from getting on well with him personally. Although Horgan stops short of fully endorsing Sheldrake’s views, on his blog and in interviews he has repeatedly provided glowing personal testimony to Sheldrake’s brilliance and implied that there might be something to his research but he hasn’t looked into it enough and is not quite convinced… yet. That is reflective of the double standards he operates, string theory and theoretical physics in general is bunk but morphic resonance… who knows?

    In regards the typology produced above. That’s really useful although I disagree with the implication that as the numbers increase the issues become bigger and potentially more important? (I know that wasn’t stated but it does seem vaguely insinuated). It’s true that bigfoot isn’t exactly a pressing social issue but the kind of thinking that supports the ‘theory’ (reliance on anecdotal evidence, anomaly hunting, belief in grand ‘establishment’ conspiracies to silence the truth) can have much broader implications. I also think it’s debatable whether health issues like the anti-vaccine movements should be lumped in with stuff fad diets. Vaccine denial can have pretty severe consequences and again the kind of thinking that underlies it can produce very serious societal impacts (see what happened in South Africa when it had an AIDs denialist health minister)- given Trump’s general receptivity to conspiracies I also suspect this issue will become more important in the US in the coming 4 years.

    • Serious question: how was Rupert Sheldrake discredited? Why is he a pseudoscientist?

      Have you looked at the data from the “psychic dogs” experiments (as you call them)? Why does it deserve an “I kid you not”?

      Is it supposed to be obvious that because Sheldrake has studied whether pets can anticipate when their owners return home that he’s not a real scientist? Do you think that we should be able to make such an assessment without actually having looked at the research?

      Bashing bad science is important (and fun), but there’s a danger that we uncritically lump together everything that just *seems* obvious to us to belong to that category. We therefore fall prey to our own prejudices and narrow world views. You can’t know that something is bogus unless you’ve checked the research for yourself. How many who bash ESP can say that they know the literature well enough?

      • Alex: He’s a pseudoscientist because he posits a quasi-mystical energy field that pervades the universe and enables things like telepathic dogs on the basis of a very low quality of evidence. And yes I have looked into his research and the various attempted replications- I would say if anything I’ve spent too much time reading such work. As to why it deserved an ‘I kid you not’ comment, because he is claiming dogs have a psychic ability rather than focusing on the myriad of more mundane and much more likely explanations as to why dogs might ‘know’ and get excited when they think their owner is likely to come home. His research is full of all the standard hallmarks of pseudoscience, e.g. grand theorising based on a collection of unimpressive studies that fail to replicate, claims that his important work is being repressed by mainstream ‘materialist’ scientists, poor controls in experiments, reliance on anecdotes, etc. etc.

        As far as ESP and a host of compadre topics (Near Death Experiences, Remote Viewing, etc.), I agree that many people who are critical have not looked into the literature in depth for themselves. However, it is also the case that people familiar with research, experimental methods and statistical shenanigans, can rather quickly get a sense of the field from either reading a few of the papers and/or reading some of the critiques. I don’t blame people for not digging deep into a research literature that is widely regarded to be significantly flawed, because when you do dig you find out such sentiment is justified. And most people are already busy enough dealing with the problems in their own field so unless someone has a personal interest or wants to engage in a properly informed debate with an advocate I’m not sure I would recommend reading into the research of people like Sheldrake.

        • I believe that the vast majority of people criticizing ESP are not familiar with the literature. And I think there’s no substitute for that. If you want to form a solid opinion, “people familiar with research, experimental methods and statistical shenanigans, can rather quickly get a sense of the field from either reading a few of the papers and/or reading some of the critiques” cuts no ice. Sure, you can get a “sense”, but your sense might be wrong, and particularly in a field like ESP, might be determined mostly by social prejudice.

          I’m afraid your characterization of Sheldrake does strike me as prejudiced. Sheldrake works within the scientific methodology and has a reasonable approach. He investigates phenomena that are commonly experienced in the population (such as pets that seem to be able to anticipate their owner’s return by unknown means) and whose explanation might have wide implications for understanding the world, he publishes his results, he addresses criticism and alternative explanations, and his scientific reporting is transparent (e.g. he gives the *raw data* in a paper on the dog Jaytee, how often do you find that?). Also, he invites critics of his research to try to replicate his findings (e.g. here).

          You call him a pseudoscientist because “he posits a quasi-mystical energy field that pervades the universe”. How can you tell it is “quasi-mystical” without begging the question? What he does is posit an “energy field” – not a *quasi-mystical* energy field – as an explanation of phenomena, and it’s meant to be a testable hypothesis. That’s normal science stuff.

          You say he should not be claiming that dogs are psychic and instead “focus on the myriad of more mundane and much more likely explanations”. He does consider and discuss these explanations. I don’t know how you could get the opposite impression if you had actually read his papers and commentaries (you could start here).

          I’ve looked at and analyzed some of the psychic dog data myself (due to being able to get them from his published report) and it looks interesting enough to be worth investigating further. For example, here is a graph showing how much time before his owner’s return the dog Jaytee went to wait by the window vs vs the length of his owner’s trip home. And here is the data.

          You say “I don’t blame people for not digging deep into a research literature that is widely regarded to be significantly flawed, because when you do dig you find out such sentiment is justified”. It means precisely nothing that this literature is widely regarded to be flawed. You can’t assess a field by relying on what others (even the majority) think about it. That, in some sense, is one of the lessons I draw from this blog.

        • I remember seeing Sheldrake do a “phantom arm” experiment on TV. The set up was a perfectly okay (he’d have one armed people stick their phantom arm through a door with squares marked on both sides, and then he’d have their spouses try to decide which square the arm was coming through). But Sheldrake seemed a bit too eager to focus on (lucky) “streaks” in the data. So if someone had “detected” the location of the arm no better than chance over a predetermined amount of attempts, Sheldrake would still point out that they’d guessed right five times in a row at once point. “You seemed to have a good connection here,” he might say. That sort of put me off his “science”.

        • No connection, but “phantom arm” reminded me of this experiment where amputees with phantom pain in a amputated limb could get relief by performing exercises in front of a trick mirror that made it seem they had both arms.

          I was amazed at this solution back then; especially because the study came up with an appealing causal explanation.

          But now, I merely wonder if that result was robust, or yet another crappy page in the annals of non-reproducible science?

        • AFAIK this was not pseudo-science but a brilliant discovery by VS Ramachandran? From his home page

          “On the practical clinical side, the center was the first to show that visual feedback (conveyed through viewing a reflection of one’s own painful body part) can powerfully reduce chronic pain, including phantom limb pain and chronic (complex) regional pain syndrome caused by nerve or tissue injury, maladies traditionally considered incurable even with surgery and powerful, side-effect-laden drugs. The technique even provides some relief from paralysis resulting from stroke. Remarkably, pain from osteo-arthritis has been shown to diminish substantially with this sort of visual feedback therapy. Most of these therapies have been validated in placebo-controlled clinical trials. But more research is needed to establish why some patients are helped more than others.”

          See here. Diana was very critical of one of his papers and thought it was a parody, but he works on very unusual things and I don’t think he does statistical analyses of the Amy Cuddy variety anyway.

        • I would start here http://www.sciencedirect.com/science/article/pii/S156918611530019X expecting to find problems as that’s most commonly what happens.

          The unfortunate reality is that these meta-analysis are seldom un-problematic and even if they are there is possibility of mis-reported and not reported results.

          Here “The literature search yielded 14 studies that satisfied the selection criteria, of which five (4 randomised controlled trials and 1 case study) were reviewed after screening” which suggests at least for this particular medical condition – the evidence is fairly sketchy.

          I am more with Thomas, except for some exceptions, we’ve got to redo the last 20 years.

        • @Shravan

          How could you differentiate so quickly between brilliant discovery vs pseudo science? Did you study the statistical justification? Just curious.

          I’ve no opinion on the specifics, just general skepticism & more so given the fantastic nature of the effect.

        • @Keith

          At least from this meta study that you linked to I’d think this finding about mirror-therapy is more spurious artifact than fantastic result by V S Ramachandran.

        • I disagree with your assessment of Sheldrake’s research and the main message you have taken from following the blog.

          Respectfully, if you find Sheldrake’s research to be impressive, well controlled, and his morphic resonance theory to be appropriately scientific then I think you must have much lower experimental, analytical, and general scientific standards than is appropriate. Sheldrake’s research is about on par with Daryl Bem’s and Dean Radin’s, all of which serve as cautionary tales for how statistics and experimental designs can be abused by researchers’ biases, regardless of their intentions, to produce ‘significant’ results.

          The energy field he posits also is quasi-mystical. It not only controls/guides development and broader processes of evolution but also enables telepathy and precognition. It resembles the force in Star Wars much more than a modern scientific theory. What’s more is that it is posited as the solution to ALL of the ‘problems/limitations’ with the theoretical explanations provided in mainstream research on genetics, evolution and biology; the fact that few scientists in those fields agree with Sheldrake’s perception of said problems/limitations makes the case directly parallel to those advocating intelligent design as a solution to the ‘problems/limitations’ of the evidence for evolution. This doesn’t mean all issues are settled in mainstream research (far from it) but rather that the burden of proof for those positing brand new energy fields that enable superhuman abilities which defy fundamental things like causality, require lots of good evidence to support them. Sheldrake doesn’t have this. He has a collection of poorly controlled studies primarily conducted by himself and his collaborators. Independent replications by reputable labs invariably fail and are then dismissed or explained away as being due to the bias of the researchers.

          Sheldrake publishing some of his ‘raw’ data is commendable, but it hardly makes his case ironclad. How much data is in the file drawer? And how much is down to researchers degrees of freedom, e.g. the flexibility in the criteria for coding ‘hits’, which are not captured in excel documents? Similarly, your references imply that Sheldrake has dealt adequately with all of the objections of his critics but that is not my reading of the literature at all. It seems we just have very different standards of evidence: I read Sheldrake’s original papers, the critiques, and his various rebuttals and I still see massive problems in his research. You on the other hand, seem to think he is answered the criticisms and found some intriguing effects. I think we will just have to agree to disagree because I don’t see much to be gained from going into this deeply on a blog comment section.

          Finally on the lessons to draw from the blog. While broadly I think you have a point, relying blindly on consensus is severely problematic. However, that’s not what I am advocating. I am suggesting that people who are appropriately critical and well informed about research practices/statistical analysis should be able to fairly quickly recognise a research literature that relies primarily on controversial low quality studies to draw massive conclusions about science shattering discoveries. Your approach would imply that no-one should respect the scientific consensus on climate change until they have personally checked all the calculations of every paper to satisfaction, which seems to be the wrong message to takeaway. I do think it is good for people to get informed on issues and read papers where possible, but I also think it is entirely appropriate to recognise a consensus when it is based on high quality evidence from multiple independent sources (as is the case with climate change and evolution). There are always ongoing areas of debate and dispute but the shortcomings of the scientific process does not mean that all consensus positions should be considered bunk.

  13. I don’t know if Ben Goldacre has written about what motivated him to move from criticising junk science (3, 4, 6) in “Bad Science” to criticising 9-11 and “science as a broken system” (I feel like that’s not really in your taxonomy) in “Bad Pharma”. He is a really effective critic of both, and I think the reason many skeptical people progress from doing mostly 3,4,6 to the 9-11 might be the realisation encapsulated in this text: http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/
    i.e. after analysing a lot of pathological and junk science, you realise that the problems you point out afflict “real” science almost just as much, but with even direr consequences.
    So, I agree with you that it is instructive to evaluate junk science, not only because the agreement that ESP isn’t real allows you to focus on how the “statistical evidence” was obtained despite that, instead of also arguing with believers, but also because it kind of works well enough as the control group for Science with a capital S (the whole system with all the incentives, biases, job worries, hype in it).

  14. whoever claims being a critical journalist and doesn’t have a clue
    that string theory is *not* a mathematical matter but is related to physics
    shall get back all his schooling money. Total failure, sorry.
    Second, every scientist worth a penny should be critical, even to his own findings.
    If not, no good scientist. Cheers!

  15. Although it’s somewhat tangential, since I know something about mammograms, I’ll write somethng about them here.

    Since 2000, the National Cancer Institute has sponsored a consortium of investigators to model the impact of both mammography and innovations in treatment on breast cancer mortality in the US. I’m one member of that consortium. It’s been very interesting work. Any model necessarily incorporates several parameters that represent unobservable states of the world. Our consortium runs 6 different models, which not only make different assumptions about those unobservable parameters (and use common estimates of the observed ones) but also rely on different underlying structural models of breast cancer epidemiology. We have published heavily over this time period and our models usually produce reasonably similar results for most questions of interest, suggesting that those findings are robust to what one assumes about these things, at least within the range of variation of our models’ assumptions. But one area where our models have never achieved agreement is the rate of overdiagnosis. More recently, we have delved deeper into these questions and have been able to demonstrate that the rate of overdiagnosis in each model depends jointly on several parameters and that observational data are not sufficient to identify those parameters singly. Available observational data do identify a subset of parameter space in which those unobservables necessarily fall, but the range of compatible overdiagnosis rates that follow is very wide, ranging from about 1% to about 50%.

    Needless to say, this is all rather distressing, because overdiagnosis undoubtedly represents the greatest harm of mammography. What could be a worse outcome than having a mastectomy, radiation, and chemotherapy for a “lesion” that never would have become symptomatic had you not had a mammogram? Yet we cannot quantify the incidence of this problem.

    Anyway, my point is that, yes, one should be skeptical of mammograms. But one should also be skeptical of mammogram skeptics. Anyone who commits to any sharp statement about the frequency of mammographic overdiagnosis is implicitly committed to sharp beliefs about unobservable states of the world. That or they simply have no idea what they’re talking about.

  16. I don’t see how it can be so difficult to narrow the range, something is wrong (possibly with my understanding). What percentage of people screening positive on mammograms are also diagnosed with breast cancer after a biopsy?

    • First, don’t confuse false positives with overdiagnoses. A false positive mammogram is one which appears to show a cancer, but on biopsy no cancer is found. These occur with fair frequency, but they are not a serious problem: the only harm done is the biopsy, which is, relatively speaking at least, not a big deal. Overdiagnosis is far worse: the biopsy shows cancer. The problem is that not all cancers have a similar prognosis. Some breast cancers, if left alone, will either just sit there and do nothing or even go away spontaneously. Some will progress rapidly towards fatality. Most will progress at a moderate rate: breast cancer survival rates are rather good nowadays. The problem is that we are not able to tell which kind is which based on the biopsy. Some biomarkers and genetic tests shed a bit of light, but most leave things pretty uncertain. In particular, we have no useful way to distinguish a small breast cancer that will progress slowly from one that will not progress at all. So all of these cancers get treated: at a minimum there is a lumpectomy, at worst there is mastectomy + radiation + chemotherapy/hormonal therapy. That is the serious problem that we wish we could avoid: some women get treated for breast cancers that are so innoccuous that they would live a normal life with them and never even know they had them if they didn’t expose these cancers with mammograms.

      Here’s a very simplified explanation of why we can’t identify the rate of overdiagnosis. Remember that, by definition, an overdiagnosis is a cancer that never would have become symptomatic if not unmasked by a mammogram. So in the absence of mammography, we would have a certain incidence rate for breast cancer. When we implement a mammography program, the incidence of breast cancer goes up. This consists of two components: there is an initial burst of breast cancers that would have eventually surfaced symptomatically but now are found a few years earlier and are added to those that were destined to become symptomatic now. But over time that burst subsides, as (nearly) _all_ cancers are now being detected, on average, a few years earlier than they would have surfaced symptomatically. The key observation is that the new equilibrium incidence rate is not a return to the original rate: it is higher. There are two components to the increase in the new equilibrium incidence rate over the original incidence rate. One part is overdiagnosis: cancers that never would have surfaced at all are now being diagnosed. The other part is that the incidence of breast cancer tends not to be stable over time and may well be increasing over and above any changes in ascertainment. For example, in the two decades before mammography was introduced in the United States, the incidence rate of breast cancer was increasing steadily and pretty rapidly. This was in an era when the never-become-symptomatic cancers never surfaced.

      So, the question then becomes how to distinguish the part of the increase due to overdiagnosis from the part of the increase due to a secular increase in breast cancer incidence. There are some constraints imposed: overdiagnosis has no impact on breast cancer mortality (apart from the very small number of deaths associated with treatment), whereas a secular increase in breast cancer incidence would be associated with an increase in mortality. But it gets more complicated than that, because during the same era that we have been doing mammograms and improving mammographic technology, we have also introduced new, more effective treatments for breast cancer. While we have some decent data on how much these new treatments impact breast cancer mortality, even factoring this into the calculations leaves appreciable uncertainty about the overdiagnosis/secular trend partitioning of the incidence increase. Part of that uncertainty arises because we do not have adequate information about what treatments are actually being used in practice–this part of the uncertainty could, in principle, be eliminated through research–it is, in principle, observable. But even if we knew that exactly, the effectiveness estimates of treatment still have some uncertainty.

      Moreover, there are other areas of uncertainty. I referred above to breast cancers being detected “a few years earlier” (by which I mean about 2 years for women in their 40’s, 3 years for women in their 50’s and 4 years for women 60 and older) on mammography. That’s a fairly well established average lead time based on data from early randomized controlled trials of mammography, modeled assuming that the lead time distribution has an exponential form. But nobody knows if that asssumption is correct, and it turns out that if you make the tail of that distribution a bit lighter or heavier (and preserve the mean) the consequent calculation of the overdiagnosis fraction changes appreciably. If you also say that the mean estimate is more uncertain than typically assumed because it was based on an unprovable assumption, then you can expand still farther the plausible range of the overdiagnosis fraction.

      So we have at least four unobservable parameters that affect the overdiagnosis fraction: the proportion of breast cancers which are inherently non-progressive, the secular trend of breast cancer incidence, the shape of the lead time distribution, the mean of the lead time distribution, and one that is in principle observable but has not been well observed: the usage patterns of treatment. There are actually some others that I haven’t mentioned here, but their effects are much smaller. So, if you take extreme values for all of these, you end up predicting incidence and mortality rates that don’t match reality. But if you stay somewhat near the centers of these distributions, or even allow one of the parameters to be extreme, the modeled incidence and mortality rates match observation well–but the overdiagnosis fraction varies widely.

      • Thanks for another informative comment. Another concern that might or might not be based on good evidence: I recall reading a few years ago that some researchers believe that performing a lumpectomy may prompt a non-malignant tumor to become malignant, thus creating a risk from the lumpectomy vs “watchful waiting.” Is this substantiated by evidence?

      • One problem I have seen with a colleague who was very focused on mammogram issues is that he could not seem to understand that someone whose mother and aunts all had died of breast cancer should think about it differently than someone with no family history. It was as though having understood the basic concepts of specificity and sensitivity he could not grasp that the base rates for different populations differed.

  17. Andrew, This is exactly the sort of thoughtful response I hoped my admittedly shoot-from-the-hip critique of captail-S Skepticism would provoke. Your taxonomy of science-y things that are or should be scrutinized is terrific. Just one point re Bigfoot, since there’s so much discussion of him (it?). I added the Bigfoot jibe to my talk at the Skeptics shindig because just before I went on the MC, Jamy Swiss, ranted to me about how important it is to debunk Bigfoot.

  18. What Horgon describes is exactly what got me to stop attending Skeptics conferences and stop reading publications like Skeptical Inquirer and stop listening to more arguments about why God doesn’t exist and how awful religion is. The Skeptics do good work, yes; someone needs to go after the homeopaths and the “vaccines cause autism” crowd. But it’s just too boring and easy to spend your time criticizing obvious nonsense. Skepticism is more usefully applied to targets that aren’t obvious nonsense; claims that may or may not be true and warrant deeper investigation.

    An example that hit me a few years back… there have been scares over the state of Texas setting standards for what should and should not be covered in textbooks used in public schools. The argument goes along the lines of “Texas is such a large buyer that textbook companies will just put whatever Texas wants into their flagship books that most other states use, and so Texas sets the de facto guidelines for the rest of the country.” I saw this hypothesis repeated over and over again in news stories, but I could never track down a real source. Is it true? I don’t know. It sounds plausible, and worth subjecting to skeptical inquiry!

    Contrast this with chemtrails and haunted houses and miracles and statues of the Virgin Mary that cry. I suppose someone should be out there debunking this stuff, but it doesn’t seem like an interesting use of one’s time and brain power. And there is a lot of it in Skeptic culture, almost to the point where this is how skepticism is implicitly defined: a disbelief in obvious nonsense that unfortunately large chunks of the public fall for.

    • I don’t know if the Texas school board has that much power over textbooks in the entire country or not. But the principle that a large state can unilaterally impose its will on the nation as a whole is quite true and well documented. Here are two examples:

      1. California set more stringent emissions standards for automobiles than either the Federal government or any other state a long time back. Carmakers felt they could not turn their backs on the California market, and the cost of producing a special line of cars just for California was viewed as prohibitive. Consequently all cars sold in the US have met California emissions standards since shortly after that. (Mind you, I think that’s a good thing.)

      2. Among medical specialties, Family Medicine has long been something of a stepchild in the world of academic medicine. Until the mid 1990’s very few medical schools offered courses in the discipline. Then California enacted a law that required a four week rotation in Family Medicine, under the direction of a board-certified Family Medicine specialist to be eligible for licensure in California. (Applicants who received their medical degrees before the law went into effect were exempt.) Again, medical schools decided they could not turn their back on students who hoped to one day practice there. Now pretty much every medical school offers a course meeting the California requirement. As far as I know there are only one, or possibly two, exceptions. (Their graduates can take a course later if they choose to practice in California, but it delays their license and costs them time and money.)

      • Thanks Clyde, those are the sorts of examples I had hoped would come up in the “Texas sets the textbook standards” newspaper articles. I picked the example not because I find it really interesting, but just because it’s a kind of mundane claim that seems plausible but got repeated a lot without journalists or pundits (that I saw at least) bothering to back it up. I feel like this kind of thing happens all the time in popular discourse, but it doesn’t have the flavor of the sort of topic that interests the Skeptic community.

  19. P.S. I doubt that anyone’s actually talking about bigfoot anymore. But I do think that skeptics target the items that get the hype. Back in the 1970s, we really were hearing a lot in the media about bigfoot, the Bermuda triangle, Biblical relics, and ESP,

    Do we understand why they went away? That could be valuable information.

    • Victor:

      Interesting question. Here’s what I think was the view of generally educated laypeople in the 1970s:

      – Bigfoot: Maybe real, probably not.
      – Bermuda triangle: Not real.
      – Biblical relics: I think there really were people who thought that Noah’s ark was out there somewhere.
      – ESP: People were open to the idea.

      How have things changed?

      – Bigfoot: Globalism and all, it’s harder to believe that there are mysterious creatures out there near the edges of the map.
      – Bermuda triangle: Nobody really cared about this one. Once they stopped selling books on it, it disappeared.
      – Biblical relics: Still a big business, I’m sure, but somehow dropped off the mainstream. Maybe that’s because there are various specialized religious TV networks etc? Back in the 70s these things got more mainstream exposure?
      – ESP: Nobody cares because now there’s cooler things going on with real science: chess-playing computers, Google, MRI scans, etc. Even if ESP were real, it’s not half as impressive as modern technology.

      • With regard to Bigfoot: I think another reason is that every human being on the planet is now carrying a camera with them 24×7 (and trail cams are cheap and easily available). Absence of high-quality photographic evidence is starting to look like pretty good evidence of absence. The same can be said of UFOs.

  20. “Even if ESP were real, it’s not half as impressive as modern technology.”

    This is especially true because “scientific” parapsychology has gotten reasonably humble (if that’s the right word) about effect sizes. Once it was accepted that precognition, whatever it is, would be much less reliable than a known mental ability like, say, memory, and given how only moderately impressive our knowledge of the past is, actual predictions based on data, like actual records of the past, just became more interesting.

    I always have to make deliberate efforts to distinguish between “remote viewing” and “remote sensing”. It’s sort of reasonable to try to find an ability to gain information about distant object using occult forces in a world without surveillance aircraft and satellites and mobile phones with cameras. Reportedly the last serious attempts to demonstrate clairvoyance in the lab were made in the 1970s. Even by then it was becoming unnecessary.

    Again, this is because even the dubiously demonstrated effects that were being talked about weren’t very informative about the targets. At some point you’re as well off hiring a person with “good hunches”.

    To connect to the SETI discussion, people have been making a similar point on that thread. Even the scientists aren’t promising very much from the “message” any longer. The best we’d be able to do is establish a vague fact of the existence of aliens. Likewise, maybe some people’s guesses about coin tosses ultimately aren’t right 50% of the time but 52% of the time. So what?

Leave a Reply to Alex Gamma Cancel reply

Your email address will not be published. Required fields are marked *