Two reviews of Nate Silver’s new book, from Kaiser Fung and Cathy O’Neil

People keep asking me what I think of Nate’s book, and I keep replying that, as a blogger, I’m spoiled. I’m so used to getting books for free that I wouldn’t go out and buy a book just for the purpose of reviewing it. (That reminds me that I should post reviews of some of those books I’ve received in the mail over the past few months.)

I have, however, encountered a couple of reviews of The Signal and the Noise so I thought I’d pass them on to you. Both these reviews are by statisticians / data scientists who work here in NYC in the non-academic “real world” so in that sense they are perhaps better situated than me to review the book (also, they have not collaborated with Nate so they have no conflict of interest).

Kaiser Fung gives a positive review:

It is in the subtitle—“why so many predictions fail – but some don’t”—that one learns the core philosophy of Silver: he is most concerned with the honest evaluation of the performance of predictive models. The failure to look into one’s mirror is what I [Kaiser] often describe as the elephant in the data analyst’s room. Science reporters and authors keep bombarding us with stories of success in data mining, when in fact most statistical models in the social sciences have high rates of error. . . .

In 450 briskly-moving pages, Silver takes readers through case studies on polling, baseball, the weather, earthquakes, GDP, pandemic flu, chess, poker, stock market, global warming, and terrorism. I appreciate the refreshing modesty in discussing the limitation of various successful prediction systems. For example, one of the subheads in the chapter about a baseball player performance forecasting system he developed prior to entering the world of political polls reads: “PECOTA Versus Scouts: Scouts Win.” Unlike many popular science authors, Silver does not portray his protagonists as uncomplicated heroes, he does not draw overly general conclusions, and he does not flip from one anecdote to another but instead provides details for readers to gain a fuller understanding of each case study. In other words, we can trust his conclusions, even if his book contains little Freakonomics-style counter-intuition.

If you are thinking the evalution methods listed [in the book] seem numerous and arbitrary, you’d be right. After reading Silver’s book, you should be thinking critically about how predictions are evaluated (and in some cases, how they may be impossible to verify). Probabilistic forecasts that Silver advocates are even harder to validate. Silver tells it like it is: this is difficult but crucial work; and one must look out for forecasters who don’t report their errors, as well as those who hide their errors by using inappropriate measurement.

Throughout the book, Silver makes many practical recommendations that reveal his practitioner’s perspective on forecasting. As an applied statistician, I endorse without hesitation specific pieces of advice, such as use probability models, more data could make predictions worse, mix art and science, try hard to find the right data, don’t just use readily available data, and avoid too much precision.

The only exaggeration in the book is his elevation of “Bayesian” statistics as the solution to predictive inaccuracy. What he packages as Bayesian has been part of statistical science even before the recent rise of modern Bayesian statistics. [This is a point that Larry Wasserman and I discussed recently — AG]. . . .

In spite of the minor semantic issue, I am confident my readers will enjoy reading Silver’s book. It is one of the more balanced, practical books on statistical thinking on the market today by a prominent public advocate of the data-driven mindset.

Cathy O’Neil is more critical:

As a modeler myself, I am extremely concerned about how models affect the public, so the book’s success is wonderful news. The first step to get people to think critically about something is to get them to think about it at all. . . . Silver has a knack for explaining things in plain English. . . .

Having said all that, I [O’Neil] have major problems with this book and what it claims to explain. In fact, I’m angry.

It would be reasonable for Silver to tell us about his baseball models . . . [and] political polling . . . He also interviews a bunch of people who model in other fields, like meteorology and earthquake prediction, which is fine, albeit superficial.

What is not reasonable, however, is for Silver to claim to understand how the financial crisis was a result of a few inaccurate models, and how medical research need only switch from being frequentist to being Bayesian to become more accurate. . . .

The ratings agencies, which famously put AAA ratings on terrible loans, and spoke among themselves as being willing to rate things that were structured by cows, did not accidentally have bad underlying models. . . . Rather, the entire industry crucially depended on the false models. Indeed they changed the data to conform with the models . . .

Silver gives four examples what he considers to be failed models at the end of his first chapter, all related to economics and finance. But each example is actually a success (for the insiders) if you look at a slightly larger picture and understand the incentives inside the system. . . .

We didn’t have a financial crisis because of a bad model or a few bad models. We had bad models because of a corrupt and criminally fraudulent financial system. That’s an important distinction, because we could fix a few bad models with a few good mathematicians, but we can’t fix the entire system so easily. There’s no math band-aid that will cure these boo-boos. . . .

Silver has an unswerving assumption, which he repeats several times, that the only goal of a modeler is to produce an accurate model. (Actually, he made an exception for stock analysts.) This assumption generally holds in his experience: poker, baseball, and polling are all arenas in which one’s incentive is to be as accurate as possible. But he falls prey to some of the very mistakes he warns about in his book, namely over-confidence and over-generalization. . . .

Silver discusses both in the Introduction and in Chapter 8 to John Ioannadis’s work which reveals that most medical research is wrong. . . . [But] the flaws in these medical models will be hard to combat, because they advance the interests of the insiders: competition among academic researchers to publish and get tenure is fierce, and there are enormous financial incentives for pharmaceutical companies. Everyone in this system benefits from methods that allow one to claim statistically significant results, whether or not that’s valid science, and even though there are lives on the line. . . . there’s massive incentive to claim statistically significant findings, and not much push-back when that’s done erroneously, so the field never self-examines and improves their methodology. The bad models are a consequence of misaligned incentives. . . .

Silver chooses to focus on individuals working in a tight competition and their motives and individual biases, which he understands and explains well. For him, modeling is a man versus wild type thing, working with your wits in a finite universe to win the chess game. He spends very little time on the question of how people act inside larger systems, where a given modeler might be more interested in keeping their job or getting a big bonus than in making their model as accurate as possible. In other words, Silver crafts an argument which ignores politics. . . . Nate Silver is a man who deeply believes in experts, even when the evidence is not good that they have aligned incentives with the public. . . .

My [O’Neil’s] complaint about Silver is naivete, and to a lesser extent, authority-worship. I’m not criticizing Silver for not understanding the financial system. . . . But at the very least he should know that he is not an authority and should not act like one. . . . Silver is selling a story we all want to hear, and a story we all want to be true. Unfortunately for us and for the world, it’s not.

Putting the two reviews together

1. Nate’s a good writer, he gets right to the point and is willing to acknowledge his own uncertainty. Unlike Gladwell, Levitt, etc., he doesn’t present scientific inquiry in terms of heroes but rather captures the back-and-forth among data, models, and theories. This comports with my impression that Nate is a hard worker and an excellent analyst who can get right to the point of whatever he is studying. (And, by the way, in the world in which I live, “hard worker” is one of the best compliments out there; I’m not using letter-of-recommendation-talk in which “hard worker” is a euphemism for “weak student.” It is by working hard that we learn.)

2. Nate is excellent when writing about areas he’s worked on directly (baseball, poker, poll analysis), solid when acting as a reporter on non-politically-loaded topics (weather forecasting), and weak where delving into academic subjects (Kaiser, Cathy, and Larry all discuss where Nate oversells Bayesian inference, something that presumably wouldn’t have happened had he run a prepublication draft of his book by any of us for comments) and the more technical areas of finance where he doesn’t have a helpful expert to keep him from getting lost.

3. Kaiser’s review is positive because he’s treating The Signal and the Noise as a pop-statistics book along the lines of Freakonomics, and he (Kaiser) is happy to see Nate’s openness and questioning spirit, allied with solid practical recommendations. Cathy’s review is negative because she’s comparing the book to other treatments of technical failure in our society and she is worried that Nate is implicitly sending the message that the problems with the financial system arose from errors rather than corruption and that everything could be fixed with some Moneyball-type analysis.

4. Putting all this together, I think the two reviews given above are essentially in agreement. (Again, I say this without actually having seen the book itself; I respect both Kaiser and Cathy and it makes sense for me to try to synthesize their views.) Nate does a good job—perhaps the best popular writing ever on what it’s like to be a statistical analyst—but he doesn’t often leave the analyst’s box to look at the big picture. Nate considers non-statistical issues in some small cases (for example, when writing about the varying incentives of different groups of weather forecasters) but is, according to Cathy, too accepting of face-value motivations when discussing finance, medicine, and politics. But this should not, I think, diminish the value of Nate’s contributions in providing a perhaps uniquely readable description of statistical thinking in some real-world settings. Even while we recognize that people often have strong motivations to make inaccurate predictions when money and reputations are on the line.

31 thoughts on “Two reviews of Nate Silver’s new book, from Kaiser Fung and Cathy O’Neil

  1. I think Cathy’s review was terrible. I actually question her reading comprehension skills after reading her review. Both the political chapter and finance chapters deal extensively with incentive issues. Silver lays out why political pundits have nearly no incentive to try to get predictions right, and therefore do not get them right. In the finance chapter, Silver talks about principal-agent issues, herding and more.

    My guess is O’Neill takes herself very very seriously and is upset some one she considers ‘superficial’ is getting so much attention. I don’t think her incentive were properly aligned to write a useful review.

  2. Thanks for the re-post. Great to see the other perspective.

    OGT: I’d say most of Cathy’s points are valid but in my view, she’s demanding too much from a popular science book. As someone who writes such books, I get why he chose topics like the financial crisis and climate change. Silver narrowly focuses on his topic at hand, which is predictive accuracy, thus opening himself up to the criticism that he didn’t offer a comprehensive explanation of these issues from a data perspective. Maybe he went overboard with dropping names of interviewees (not really that bad compared to other similar authors) but if he’s going to write about a topic, he has to present an expert point of view because otherwise why would we be interested?

    Andrew: I even think that it’s possible that he threw in the Bayesian reference as an afterthought because the editor wanted a unifying concept. I’m just speculating based on the fact that it really isn’t integrated throughout the book. But surely, you can help him with that argument.

  3. The part about Bayes is really bizarre. He even says that Fisher got things wrong on the smoking-causes-cancer debate because he didn’t use/like Bayesianism. And he presents no evidence to such a claim. And I agree with O’neal that he gets things wrongly in the financial crisis and seems to be naive.

    And yet, I think the book is really great. I think you would like the emphasis on checking models and the importance of modeling things right. I particularly liked how he discussed the relationship between quality of data and quality of theory. When the theory is good (like in weather forecast) the measurement problems are alleviated by the power of theory. When one field (like economics or earthquake prediction) lacks good data and good theory, then the models will suck. He didn’t say so, but I think that he would support statements like you use to say that economics use folk psychology on their theorizing.

    • I too, liked how he framed models from different fields (physics, epidemiology, statistics) as exemplifying different points on a spectrum of apriori structural knowledge. There are interesting relationships between prior knowledge of causal structure and ability to predict (or power to reject) that aren’t frequently discussed in this sort of holistic way.

      I’ve always thought that modeling should be more of a unified field than it is in practice, and he does a pretty good job piecing together a glimpse of what things would look like if it was.

  4. Silver’s book was waiting for me when I returned today. I’ve only read what he wrote in his chapters on “frequentism”, Fisher, and significance tests and it’s obvious that the man has no clue, NONE WHATSOEVER, about what a significance test is. Let me just say that I thought I’d read the absolute bottom of the barrel, the worst, most irresponsible, silly and absurd and utterly off the wall and ignorant claims made about Fisherian tests and “frequentism”, but nothing, not even the witch hunters and scapegoaters,* comes remotely near the level of obtuseness of this shallow and crass treatment. The most hysterical rehearsals of statistical howlers are measured and learned next to Silver’s treatment (if name-calling even counts as a “treatment” of a statistical account). Frequentists, Silver alleges, go around reporting hypotheses like toads predict earthquakes and other “manifestly ridiculous” findings that are licensed by significance testing and data dredged correlations. (253). But it is the frequentist who prevents such spurious correlations. (Being a popular book does not excuse such an ill-informed, hysterical diatribe against methods about which he appears clueless.)

    It would also be hard to find a more vapid example of All-You-Need is Bayesian cheerleading. The simple use of Bayes Theorem solves all problems (he seems not to realize they too require statistical models), and if it were up to him we would stop teaching frequentist methods in the schools, and ban significance tests immediately if not sooner. I cannot even imagine how people like Wasserman could even seriously review the statistical chapters, rather than just turn away in embarrassment, disgust and nausea (though I give Wasserman credit for his effort, sure to be misunderstood by Silver). Silver gets the Statistical Badge of Shame for 2012. (Serious users of statistics: have plenty of Dramamine handy.)

    * http://errorstatistics.com/2011/09/26/whipping-boys-and-witch-hunters-comments-are-now-open/

    • Dear Deborah,

      I understand from your comment that you are displeased with Nate’s treatment of frequentist methods. I have very quickly browsed the book but I could not discover a particular “howler” — perhaps you care to give a concrete and specific example? I saw him mention that frequentist methods depend on the sampling plan, and that they do not routinely take a priori plausibility into account. The latter is clearly true, at least in my field. For instance, some journals in psychology publish the silliest findings simply because p < .05 [examples: people being able to look into the future; people being more creative when there is a box in the room (because you are "thinking outside the box"); and people judging relationships are more unstable when they sit on a wiggly chair.] Sellke, Bayarri, & Berger (2001) have demonstrated, using frequentists methods, that p-values dramatically overstate the evidence against the null.

      Cheers,
      E.J.

      • Deborah’s description is right about Chapter 8 which is the one chapter in which the Bayesian argument is made. It is largely one-sided, in compiling pretty much every pro-Bayesian argument out there, which would make readers wonder why anyone could ever use non-Bayesian statistical methods. The smoking/cancer example is used to bash significance testing, which I’m still trying to absorb. I also find the pro-Baynesian argument is weakened by the quantity of sub-arguments – would have been stronger if he had focused on the strongest points. It is further limited by his illustrative examples being all of the Bayesian updating variety.

        However, I think people should not throw away the book just because of Chapter 8. That’s why in my review, I say if you ignore the Bayesian argument, what remains is still valuable.

        From the perspective of books written for the mass market, Nate’s has many virtues:
        1. Use of contemporary examples (as opposed to the well-worn hypothetical examples like Birthday Problem, drunkyard’s walk, etc.)
        2. Use of long case studies, instead of little anecdotes that quickly fall apart if you think one level deeper. This necessarily retards the narrative because the author has to provide context, which some readers may find tedious rather than useful.
        3. Emphasis on model checking, error rates, etc.
        4. Belief in “all models are wrong”
        5. Restraint from making too-strong conclusions for the sake of getting attention

        For me, the most interesting follow-up question would be whether the differential predictive performance as observed by Silver is due to structural differences, differences in skill, or differences in the way we measure performance.

      • Anonymous: I’m not inclined to respond to “anonymous” comments on other people’s blogs (though I do on my own); I didn’t hide under anon or initials. But here’s a quick reply anyway: firstly, I said, not that he commits well-known howlers but that “the most hysterical rehearsals of statistical howlers are measured and learned next to Silver’s treatment (if name-calling even counts as a ‘treatment’ of a statistical account)”. Those critics point up fallacies of tests and would, at least the reformers among them, replace dichotomous uses of significance tests with other frequentist methods like confidence intervals. (For the most part, they are not espousing we give up frequentist methods or control of error probabilities.) Silver’s assertions here, by contrast (252-3), are just plain kooky. Anyone who could say that Fisher, the man who developed randomization and wrote books on experimental design, was searching for immaculate statistical procedures and that he excluded human error and bias, simply doesn’t know what in the world he’s talking about. One cannot even begin….Same for the assertion that frequentist methods are solely good for polling contexts. I wish he would actually read Fisher, he’s quite accessible. Anyone who thinks the possibility of finding “manifestly ridiculous” papers (in places like newsline.com) is evidence that these are licensed by or are representative of frequentist statistics is being irresponsible, and cannot be taken as mounting serious criticisms of the methods (unfortunately, as a rockstar pollster, people may believe him). As for your allegation that significance levels overstate evidence (not something raised by Silver), I and others have shown this to be false on several occasions. I’ve even responded to Berger on this.
        Mayo, D. G. (2003). “Could Fisher, Jeffreys and Neyman have Agreed on Testing? Commentary on J. Berger’s Fisher Address,” Statistical Science 18: 19-24.
        You can find discussion of this on my blog.
        http://errorstatistics.com/2012/10/21/mayo-section-6-statsci-and-philsci-part-2

        or in more comic form:
        http://errorstatistics.com/2012/04/28/3671/

        Clearly, there is nothing constructive to be gleaned from going further with this. I wasn’t reviewing his book, just making a blog comment on the statistics portion.

        • Dear Deborah,

          I apologize for posting anonymously — I have nothing to hide but I just forgot. It was difficult for me to get to the main point of your initial critique; I found myself quickly tangled up in a jungle of strongly negative words. I am afraid your reply is very much in the same style, and this just keeps making things hard for me to understand. All I get is that you are angry because Nate wants to euthanize the method you love. I can sympathize with your feelings: if I were to defend a method that everybody misunderstands and dislikes (a method that, moreover, violates the likelihood principle, yields misleading conclusions, and is taught without highlighting its core limitations) than I’d probably be a bit cranky myself.

          As far as the “allegation that significance levels overstate evidence” goes — the allegation is based on math and simulations, not on speculation. So the label “fact” is a more accurate description. Your counterargument focuses on that fact that the method has to assume some frequency with which effects are sampled from H0 instead of H1. This frequency is set to 50% and you find this a fatal flaw. To me, this reeks of desperation. First, Ioannidis has famously argued that in many fields the frequency of effects coming from H1 is much smaller then 50%, and this would compound the overestimation. Second, if you are truly worried about setting the frequency of sampling for H0 to 50%, you may vary it and do a robustness check. Or you may change the frequency of sampling until you feel the evidence is now in line with the p-value: you will then see that the frequency of sampling has to be highly biased toward H1. No applied researcher would accept such bias. None.

          At any rate, I very much feel you are hiding your head in the sand. But despite our disagreements on statistical inference it is good to have discussions like these and I wish you Happy Holidays!

          Cheers,
          E.J.

        • Dear E.J.
          Gelman and I have posted frank comments on each other’s blogs for a while now, and I had just read that chapter when I noticed Gelman was posting some reviews on his blog. I was not reviewing Silver’s book, just commenting on the chapter on frequentist statistics on a blog. Had I been reviewing it I would have read and discussed the other chapters. (I haven’t even discussed Silver on any of my blogs, though maybe now I will, having taken the time to write so much here.)

          I think one of the main reasons I was so disappointed with that Silver chapter is that I was expecting/hoping to like the book. Two months ago, an artist who was helping to paint a faux finish mural in my condo asked what I did. Philosophy of science is typically considered pretty esoteric, but when she heard I was interested in various statistical methods/knowledge, she asked, to my surprise, if I knew Nate Silver, and we discussed his 538 blog, and the day’s comments and dialogue. I thought it was great that he seemed to be bringing statistics into the popular culture, even if it was just polling and baseball.

          So I was dismayed at his negative remarks on frequentist statistics and his kooky, gratuitious ridicule of R.A.Fisher. I am not alone in thinking this (see Kaiser below). I guess I’m just sorry Silver comes off looking so clownish here, because I had thought he’d be interesting and insightful. I’ll look at the other chapters at some point, I’ve loaned out my copy…

          You are wrong to suppose that I am “angry because Nate wants to euthanize” methods that are widespread across the landscape of physical and social sciences, in experimental design, drug testing, model selection, law, etc., etc.

          These methods won’t die so long as science and inquiry remain.

          You see, science could never operate with a uniform algorithm (his “Bayes train”) where nothing brand new enters, and nothing old gets falsified and ousted. We’d never want to keep within such a bounded universe of possibilities: inquiry is open-ended. Getting beyond the Bayesian “catchall” hypothesis is an essential part of pushing the boundaries of current understanding and theorizing in science and in daily life.* Frequentist statistics, like science/learning in general, provides a cluster of tools, a storehouse by which the creative inquirer can invent, and construct, entirely new ways to ask questions, often circuitously and indirectly, while at the same time allowing (at time, ingenious) interconnected ways to control and distinguish patterns of error. As someone once said, statistics “the study of the embryology of knowledge, of the processes by means of which truth is extracted from its native ore in which it is fused with much error.”**
          They are methods for the critical scrutiny of the data and models entertained at a given time (not a hope for the indefinite long-run). The asymptotic convergence from a continual updating from more and more data (on the same query) that Silver happily touts, even in the cases where we imagine the special assumptions it requires are satisfied, is irrelevant to the day to day scrutiny/criticism of actual scientific results.

          I agree that foundational discussions are good, but much more useful when coupled with published material or at least blogposts.
          Good Luck.

          * An analogy made yesterday by Isaac Chatfield: frequentist statistical methods are a bit like surgical tools; they are designed to be used for a variety of robust probes which are open to being adaptively redesigned to solve new problems while in the midst of an operation/investigation.
          **Fisher 1935, DOE, 39.

        • “…science could never operate with a uniform algorithm (his “Bayes train”) where nothing brand new enters, and nothing old gets falsified and ousted”

          That is not what Nate Silver is arguing for. His version of Bayesian statistics is allied with Gelman’s, viewing model checking as essential. (Once your copy is returned, you can see this for yourself.)

    • Mayo’s critique is a grossly exaggerated and over-the-top.

      Most criticism of significance testing has to do with the misguided application and execution by folks attempting to use significance tests, which has a rich history in the scientific literature. That does not mean the theoretical foundation is flawed. Mayo seems to miss this point.

      • Dan: Are you saying Silver is merely pointing up some fallacious and misguided applications of significance tests, but sees the theoretical foundation as OK? I don’t see any evidence he construes his remarks this way, unfortunately. Thus his mention of that “frequentist statistics”–his term– should no longer be taught to undergraduates, and Fisherian tests banned (p. 260), even if he’s attributing these views to others. It is he who is over the top, but I think I’m most irked by his unwarranted ridicule. I give up.

        • Unwarranted? The backlash to significance testing is well deserved. That does not mean it should be done away with altogether, but the ridicule is mostly justified. Maybe Silver is being over the top but sometimes that is the best way to get people’s attention and make them think.

          If you were purposefully being over the top to achieve the same results, then so be it. But your backlash to the initial backlash seems to imply that significance testing should not be criticized. If your intended message is something different, I’d like to know what it is. I suppose I should read your material!

        • Dan: I think fallacious and unwarranted uses of statistical methods–Bayesian and non-Bayesian– should be open to criticism that is intellectually honest and reasonably knowledgable. It would be good, in criticizing “frequentist statistics”, if the critic actually said anything indicative of knowing what he was talking about. Have you read the relevant pages in Silver? For some well-known criticisms of significance tests, see my blogpost from today on errorstatistics.com

        • The ridicule or backlash seems to demonstrate a perpetual *lack of understanding* for significance testing, more than anything else. My question is: How is such a lack of understanding (mostly) justified? To me, the ridicule simply signals how much confusion continues to exist, in terms of conceiving the limits and proper usage/interpretation of significance tests. Hence, Mayo’s critique is *not* exaggerated nor over-the-top because Silver’s book does not help in any way towards reducing the already abundant levels of confusion surrounding significance testing, at least from what I have heard or read!

    • “The simple use of Bayes Theorem solves all problems (he seems not to realize they too require statistical models)”

      Yeah, they do. When Bayesian cheerleaders (and I am one!) do their thing, they often neglect to point this out, which is a serious issue. Somehow people (in general) have convinced themselves that likelihoods appear from thin air or by looking at the data, neither of which is true.

      “if it were up to him we would stop teaching frequentist methods in the schools, and ban significance tests immediately if not sooner.”

      People need to know about frequentist methods because other people use them, but I’m not a fan. I support their use for quick-and-dirty analyses but that’s it. When information is scant you need to use all the constraints you can and probability theory provides some constraints.

      • Brendon:
        Making these points and admissions, I think, disqualifies you from being “Bayesian cheerleader” as I was intending that term. I had in mind the kind of unsupported stances along the lines of “Hi Yo Silver, Everyone on the Bayes train!” (Down with immaculate Fisherian methods!)

  5. Andrew: Very nice post, I think Cathy made more important points than Kaiser, but I am biased because I have experienced the _damage_ caused by researchers uncritically buying into Bayesian hyper-promotion and ignoring incentives in research.

    Many years ago someone suggested that it was a good strategy to get clinical researchers to use Bayesian methods and only after they have been doing it for a while point out the limitations and challenges (the usual new technology ploy of exagerate the benefits, hide the costs/challenges and overstate the existing technology’s capacity to do essentially the same things.)

    I did not think was a good strategy, but it does seem to happen/work often. One perhaps could see Nate’s book as doing that for model based statistics and Bayes in particular?

    • O’Rourke: Do you remember who suggested this strategy? (I’ve heard something like this before*.) Such a conspiracy explains a few things. Did you mean “downplay” (or the like) as opposed to “overstate”, or am I reading this incorrectly?

      *It is in sync with my finding frequentist foundations being disinterred, at times. (Everything old is made new again.)

      • Mayo:

        It was suggested in a graduate biostatistics seminar by David Andrews at the University of Toronto. He did seem very serious about it, but he might have been pulling “us student’s legs”.

        What sometimes seems as obvious academic racketeering might well be folks not really understanding arguments that they hope dearly somehow are true (e.g. never being a reason or need to check priors or explicitly deal with multiplicities).

        I do recall someone (but not who) arguing that it was understandable that Bayesians exaggerate the benefits of their approach, hide the costs/challenges and overstate non-Bayesian technology’s INABILITY (typo last time) to do essentially the same things, as Bayesian methods have been misrepresented and maligned for so long. That might have been true when I wrote Two Cheers for Bayes. Controlled Clinical Trials 1996, but that is getting to be an awful long time ago.

        I do believe it is bad for the research community and even the Bayesian community in the very long run, but in the short run there are real advantages (as Cathy nicely put it “selling a story we all want to hear, and a story we all want to be true”). And I think people need to be on their toes about the possibility of negative push back if they seem to not be enthusiastically marching to that positive tribal drum beat.

        • O’Rourke: Thanks very much for this. I don’t know you, but I have been rereading blogposts lately and I’ve noticed some very informative remarks from you. This is another one.

          You say that you believe that misleadingly selling methods (or however you want to unpack “academic racketeering” here) “is bad for the research community and even the Bayesian community in the very long run, but in the short run there are real advantages”. First the former: I’m curious what bad thing you envisage for the long run day of reckoning: that one day (I don’t know, 2023?) people will say, “Wow, we sure were sold a bill of goods…these methods haven’t worked well at all…., and now we have a lot of questionable theories, models, medicines, and almost no one around who has been taught those older methods to redo the data analysis…what were they called again?”

          How perplexing, then, is your claim of the short-run advantages. Isn’t the task all about producing good science/warranted inferences today? About promoting intellectual honesty in today’s students/researchers?

          Your last sentence, which I like a lot by the way, is why it takes courage today for such intellectual honesty, even though it should not.*

          “And I think people need to be on their toes about the possibility of negative push back if they seem to not be enthusiastically marching to that positive tribal drum beat”.

          This is the reason I began my “frequentists in exile blog” a little over a year ago. Anyone ready to push back the push back—even a little bit–, please come over to errorstatistics.com.

          Tribal drum beat indeed!

          * I might refer to it on my blog (so it can earn you an Honorable Mention).

  6. Pingback: Metareviews: The Signal and the Noise | Stats Chat

  7. Pingback: Sunday data/statistics link roundup 12/23/12 | Simply Statistics

  8. Silver was taken apart by serious climate scientists who took exception to his “he said/she said” approach to what you refer to as “non-politically loaded topics,” climate scientists who have been put through legal and professional (and well reported) hell, including death threats, as a result of the “non-politics” of the topic. The people who through their more accurate interpretation of the “signal” of their more accurate models have gotten closest to right about what’s been happening were equated to those who have gamed models and obfuscated. This in a book that purports to be able to separate signal from noise. Which is the same criticism O’Neill makes about his work on finance, so apparently the climate scientists cannot comprehend either. (See http://www.huffingtonpost.com/michael-e-mann/nate-silver-climate-change_b_1909482.html.) It’s not just a matter of being naive about who’s telling you what. As O’Neill says, it feeds the narratives that support misperceptions of the causes and effects of both the financial crisis and climate change. That makes the book more than unfortunate, and rah-rahing the book while pooh-poohing the damage it will do in these fundamental policy areas is more than unfortunate as well.

  9. Pingback: Rejected Post: 3 Msc. Kvetches on the Blog Bagel Circuit « Rejected Posts of D. Mayo

  10. Pingback: Useful for referring–1-20-2013 « Honglang Wang's Blog

  11. Pingback: if you’re already using sophisticated non-Bayesian methods such as those of Tibshirani, Efron, and others, that Bayes is more of an option than a revolution. But if you’re coming out of a pure hypothesis testing training, then Bayes can be a t

Comments are closed.