A new definition of the nerd?

Jonathan Falk points to this book excerpt by Michael Lewis, who writes:

A lot of what people did and said when they “predicted” things, Morey now realized, was phony: pretending to know things rather than actually knowing things. There were a great many interesting questions in the world to which the only honest answer was, “It’s impossible to know for sure.” “What will the price of oil be in ten years?” was such a question. That didn’t mean you gave up trying to find an answer; you just couched that answer in probabilistic terms. . . . People who didn’t know Daryl Morey assumed that because he had set out to intellectualize basketball he must also be a know-it-all. In his approach to the world he was exactly the opposite. He had a diffidence about him—an understanding of how hard it is to know anything for sure. The closest he came to certainty was in his approach to making decisions. He never simply went with his first thought. He suggested a new definition of the nerd: a person who knows his own mind well enough to mistrust it.

I recommend reading Lewis’s entire article. There were a bunch of good stories, and some other bits got my attention too.

Like this:

It seemed to him that a big part of a consultant’s job was to feign total certainty about uncertain things. In a job inter­view with McKinsey, they told him that he was not certain enough in his opinions. “And I said it was because I wasn’t certain. And they said, ‘We’re billing clients five hundred grand a year, so you have to be sure of what you are saying.’” The consulting firm that eventually hired him was forever asking him to exhibit confidence when, in his view, confidence was a sign of fraudulence.

That was before there were Ted talks. But same idea: be confident, be strong, fake it till you make it, and, if anybody goes back and checks your predictions, just change the subject.

And this:

A lot of what the Houston Rockets did sounds simple and obvious now: In spirit, it is the same approach taken by algorith­mic Wall Street traders, U.S. presidential campaign managers, and every company trying to use what you do on the Internet to predict what you might buy or look at. There was nothing simple or obvious about it in 2006.

It’s funny, though, because these ideas were no secret back in 2006. Bill James became famous in the mid-80s, and by the end of the decade we were using hierarchical models to predict elections. It is true, though, that there wasn’t much general interest in quantitative prediction. It’s funny how long it all took. When it comes to polling analysis, we wrote our first paper on Mister P back in 1997 but only really started to hit the public consciousness twenty years later. Or, for an even more accessible example, people been complaining for decades about football coaches being too conservative on fourth down—and there it’s not hard to get some numbers!—but it’s only in the past few years that the default strategy has changed.

Why did it take so long? One reason is that the best way to improve predictions is not through better modeling but through more information. Lewis writes:

The Rockets began to gather their own original data by measuring things on a bas­ketball court that had previously gone unmeasured. Instead of knowing the number of rebounds a player had, for instance, they began to count the number of genuine opportunities for rebounds he’d had and, of those, how many he had snagged. They tracked the scoring in the game when a given player was on the court, compared to when he was on the bench. Points and rebounds and steals per game were not very useful; but points and rebounds and steals per minute had value.

This doesn’t explain the fourth-down-in-football problem—there, the standard explanation is asymmetric incentives, that, even after correcting for the probabilities, the reputational risk to the coach of going for it, not getting the first down, and then giving up a score, is greater than the gain from going for it, getting the first down, and continuing on to score. The idea is that nobody ever got fired for going by the book. But I don’t know how much I believe that story: Coaches do want to win, right?

Lewis’s article also features a bunch of interesting ideas on the work done, by the guys who evaluate prospective draft picks, to avoid getting fooled by misleading information—or, perhaps I should say, inappropriate inferences, as the problem wasn’t so much with the information as with what they were doing with it.

20 thoughts on “A new definition of the nerd?

  1. I love that definition.

    Expressing uncertainty to clients can elicit some fun reactions. Even better is suggesting that they pay for research in order to generate hypotheses rather than to validate them.

  2. “the best way to improve predictions is not through better modeling but through more information”

    Absolutely. Or as I like to put it, “information drives out analysis”. (You don’t need complex models to estimate something once it can be directly measured).

    But information demands work, and money, and experimentation. In teaching statistics, the demands of the semester mean students get unrealistically clean data sets, and relatively seldom have to determine what to collect in the first place. When I refereed youth soccer, I used to think about what measures one would want to develop around that sport, and how much work it would take to determine what measures and then to code, say, thousands of games.

  3. From the link:

    ‘In a job inter­view with McKinsey, they told him that he was not certain enough in his opinions. “And I said it was because I wasn’t certain. And they said, ‘We’re billing clients five hundred grand a year, so you have to be sure of what you are saying.’”’

    From Barry Ritholtz [http://ritholtz.com/2011/03/is-mckinsey-co-the-root-of-all-evil/]

    • Advocating side pockets and off balance sheet accounting to Enron, it became known as “the firm that built Enron” (Guardian, BusinessWeek)

    • Argued that NY was losing Derivative business to London, and should more aggressively pursue derivative underwriting (Investment Dealers’ Digest)

    • General Electric lost over $1 billion after following McKinsey’s advice in 2007 — just before the financial crisis hit. (The Ledger)

    • Advising AT&T (Bell Labs invented cellphones) that there wasn’t much future to mobile phones (WaPo)

    • Allstate* reduced legitimate Auto claims payouts in a McK&Co strategem (Bloomberg, CNN NLB)

    • Swissair went into bankruptcy after implementing a McKinsey strategy (BusinessWeek)

    • British railway company Railtrack was advised to “reduce spending on infrastructure” — leading to a number of fatal accidents, and a subsequent collapse of Railtrack. (Property Week, the Independent)

    McKinsey also played a huge role in education reform.
    http://observationalepidemiology.blogspot.com/2011/03/flashman-finds-work.html

    • After completing MBA school I did a number of internship like projects and was interviewed by a few of “McKinsey want to be” consulting firms. My sense was that they were looking for employees that could credibly create a sense of being sure about stuff no one could be sure about.

      My other sense was that any indication of an ability to grasp the real uncertainties was perceived as negative and on on my part likely lead to our parting of ways. They had some people who did understand but only in senior management positions.

      • “My other sense was that any indication of an ability to grasp the real uncertainties was perceived as negative”

        Sadly, all to often true; a preference for “positive thinking” over reality.

        • I’m not sure it’s a desire for positive thinking so much as it is for salability. In my experience clients usually expect certainty from the experts their paying; when you show signs of being uncertain they sometimes (more often then not) take that to mean deficient knowledge and move their business to the next person who will sell them certainty.

          It’s easier to sell to clients when your foot soldiers actually believe what they’re selling (i.e. they don’t fully understand the uncertainties involved).

        • +1.

          In many cases, consciously or not, clients are spending money to hire an “expert” to amplify a point of view that they lack the clout or credibility to advance. The last thing they want is some show-off with the ability to hold two opposed ideas in mind at the same time.

        • > It’s easier to sell to clients when your foot soldiers actually believe what they’re selling (i.e. they don’t fully understand the uncertainties involved).

          That was my sense of what was going on.

    • I have no particular affinity for McKinsey or any other consulting firm. But this list is downright disingenuous and could be easily written to have a much more positive light given the information provided in the link.

      For example (statements in brackets are a fictitious, but plausible addition to the statement of facts provided),

      “General Electric lost over $1 billion after following McKinsey’s advice in 2007 — just before the financial crisis hit.” [However, it has been estimated that GE would have lost $xx billion (>>$1 billion) if they had followed their previous path and did not follow McKinsey’s advice going into the crisis.]

      “Advising AT&T (Bell Labs invented cellphones) that there wasn’t much future to mobile phones” [However, they did so at a time when almost no one in the industry believed that the cell phone would ever be a popular item; and despite this one miss, albeit a big one, McKinsey’s track record of forecasting technological changes remains one of the best in the industry.]

      “Swissair went into bankruptcy after implementing a McKinsey strategy” [However, Swissair had very little hope of ever escaping bankruptcy given the obligations of the company prior to Swissair engaging McKinsey. It looks like the McKinsey strategy has given the company a good shot at restructuring the business post-bankruptcy protection]

      …Without relevant background information, which does not appear to be provided in the link you posted, each bullet item in the list could be made to show McKinsey in a different light…Not saying that’s likely, but I don’t think the original Author of the bullet point list is in a position to make that call (neither you nor I).

      BTW, the first point about GE is not at all substantiated by the article that your link references…in the actual article it says that GE lost 1 billion on the sale of WMC Mortage in 2007 after purchasing it in 2004….McKinsey was only involved to the extent that they were asked to comment on the risk of GE Capital heading into the crisis according to the article. I haven’t read the others in any detail; I stopped after the first article was completely misrepresented in the bullet point list.

      • First off, just to be clear, you referred to “the link,” but there are two links in the comment, one that gives the source of the list…

        “From Barry Ritholtz [http://ritholtz.com/2011/03/is-mckinsey-co-the-root-of-all-evil/]”

        And a postscript that connects it to the education reform thread. If you are looking for the supporting articles you should go to the Barry Ritholtz page, which provides links to for each point.

        As for the argument that the list is disingenuous and unfair. We should probably start with the three most egregious charges in the list, Enron, Allstate, and Railtrack, but even if we put those aside for the moment, how good does the company’s track record look? Let’s follow the links.

        http://www.washingtonpost.com/wp-dyn/content/article/2008/02/22/AR2008022202283_2.html?sid=ST2008022202336
        “Not as legendary or fateful a mistake as AT&T’s, however. In 1980, the company whose Bell Labs invented cellphones listened to McKinsey, the consulting company they’d hired. Its estimate of the market in the year 2000 was off by a factor of 120 — not even 1 percent of the real number. Based on that, AT&T decided there wasn’t much future to these toys. Not coincidentally, in 2005, it was swallowed up by SBC Communications Inc., originally a Baby Bell.”

        Even allowing for the qualifier “a big one,” defending the company’s record of technological forecasting (with no specific examples) definitely puts you in “other than that how did you like the play, Mrs. Lincoln?” territory, and the “nobody knew” defense doesn’t really hold either. A large segment of the telecommunication industry was betting heavily on mobile phones at the time:

        https://en.wikipedia.org/wiki/Mobile_phone
        “The first commercial automated cellular network was launched in Japan by Nippon Telegraph and Telephone in 1979. This was followed in 1981 by the simultaneous launch of the Nordic Mobile Telephone (NMT) system in Denmark, Finland, Norway, and Sweden. Several other countries then followed in the early to mid-1980s.”

        Knowing what we know now, the GE analysis also looks highly questionable.

        http://www.theledger.com/news/20101205/ge-goes-with-what-it-knows-making-stuff/3
        “Sixty days later, the consulting team, he says, told G.E. that money from nations with a trade surplus, like China, and sovereign wealth funds, among other investors, would provide enough liquidity in the financial system to fuel lending and leverage for the foreseeable future. (McKinsey declined to comment on the study.)”

        As do the recommendations to Swissair:

        https://www.bloomberg.com/news/articles/2002-07-07/inside-mckinsey
        “The bad news, however, is that Enron, which was paying McKinsey as much as $10 million in annual fees, is just one of an unusual number of embarrassing client failures for the elite consulting firm. Besides Enron, there’s Swiss-air, Kmart, and Global Crossing–all McKinsey clients that have filed for bankruptcy in relatively short order. And those are just the biggest. McKinsey also finds itself improbably lining up with other creditors to collect unpaid fees from recently bankrupt companies that soared during the late ’90s only to crash later. Battery maker Exide Technologies and NorthPoint Communications Group Inc., an upstart telecom provider, are two such examples.

        “At Swissair Group, McKinsey advised a major shift in strategy that led the once highly regarded airline to spend nearly $2 billion buying stakes in many small and troubled European airlines. The idea was for Swissair to expand into aviation services, providing everything from maintenance to food for other airlines as a way to increase revenues and profits. The strategy backfired, causing massive losses and a bankruptcy filing last October. McKinsey maintains it can’t be held responsible for the outcome because it wasn’t involved in the implementation of the strategy.”

        Add to that the company’s direct and indirect role in the botched implementation of common core and the statistically illiterate reworking of the SAT test.
        http://observationalepidemiology.blogspot.com/2014/04/being-management-consultant-who-does.html

        I’m with Ritholtz on this one.

  4. I should say that I work for a consultancy and have done so for my entire post-research career. Long before I did so, I remember reading a book on managing consulting (alas I no-longer remember what it was), which recounted that once upon a time Harvard Business School seriously considered restructuring their MBA ciriculum to be much less case study-centric. McKinsey lobbied hard and effectively against this change. They wanted the case-study structure to stay, because it trained students to have and express confident opinions even in circumstances when they had no claim whatsoever to relevant expertise.

    To this day, this is first thing I think of when I think of McKinsey (or Harvard Business School).

    If anyone has a reference for this anecdote, I would appreciate it.

    • “They wanted the case-study structure to stay, because it trained students to have and express confident opinions even in circumstances when they had no claim whatsoever to relevant expertise.”

      Aargh! Just what the world needs less of! (Kinda like power posing.)

    • Having experienced the case method at HBS and spent a few years in management consulting, I must disagree that the case method is intended to train students “to express confident opinions even in circumstances when they had no claim whatsoever to relevant expertise.” Students do become more confident in delivering their arguments but it does not have to come at the expense of substance. It’s a false claim that there is a trade-off between credible content and persuasive delivery.

      There are many non-HBS-trained management consultants, not products of the case method, who are equally guilty of making fluent arguments with scant evidence. The problems taken on by McKinsey-type consultants are usually vague and strategic, for which little relevant data can be obtained. That creates the condition for hot air.

      I have long held that the case method should be more widely used in teaching statistics, precisely because its Socratic-like questioning helps bring out the assumptions and uncertainties inherent in statistical problems.

  5. I can speak to the consulting bit since that’s my field–If consulting services and ‘business intelligence’ were ever actually peer-reviewed, then they would probably also be in a replication crisis right now, but who’s going to pay for the same analysis to be done twice with the risk of undermining the first ‘deliverable’? It’s basically the same incentive structure as academia: analysts have to churn out statistically significant results–and you usually get bonus points in presentations if your results are surprising or previously unknown (read: false positive).

Leave a Reply to Sean Matthews Cancel reply

Your email address will not be published. Required fields are marked *