Skip to content
 

“The following needs to be an immutable law of journalism: when someone with no track record comes into a field claiming to be able to do a job many times better for a fraction of the cost, the burden of proof needs to shift quickly and decisively onto the one making the claim. The reporter simply has to assume the claim is false until substantial evidence is presented to the contrary.”

Mark Palko writes:

The following needs to be an immutable law of journalism: when someone with no track record comes into a field claiming to be able to do a job many times better for a fraction of the cost, the burden of proof needs to shift quickly and decisively onto the one making the claim. The reporter simply has to assume the claim is false until substantial evidence is presented to the contrary.

Yup. This is related to advice I give to young researchers giving presentations or writing research papers:

1. Describe the problem you have that existing methods can’t solve.

2. Show how your new method solves the problem.

3. Explain how your method works.

4. Explain why, if your idea is so great, how come all the people who came before you were not already doing it.

There are lots of possibilities for step 4. Maybe your new idea is only now possible because of new technology, or network effects, or your idea flowed from earlier ideas that people only recently realized were effective, or maybe it was an idea taken from another area and introduced into your field. Whatever. The point is, if you don’t answer question 4, your responses to question 1, 2, and 3 are suspect.

Palko’s example concerned a newspaper story about “of entrepreneurs and Ivy League grads from the USA rescuing the poor children of Africa from poverty and ignorance.” Here’s Palko:

If you take down a few pages, there is some good, substantial reporting by Peg Tyre on this story. Unfortunately, as is all too often the case with New York Times Magazine articles, the first act is almost entirely credulous and laudatory. For example:

By 2015, Bridge was educating 100,000 students, and the founders claimed that they were providing a “world-class education” at “less than 30 percent” of what “the average developing country spends per child on primary education.” This would represent a remarkable achievement. None of the founders had traditional teaching experience. May had been an unpaid teacher at a school in China; Kimmelman worked with teachers and administrators developing an ed-tech company. How had they pulled it off? In interviews and speeches, they credited cutting-edge education technology and business strategies — the company monitors and stores a wide range of data on subjects including teacher absenteeism, student payment history and academic achievement — along with their concern for the well-being of the world’s poorest children. That potent mixture, they said, had allowed them to begin solving a complex and intractable problem: how to provide cheap, scalable, high-quality schooling for the most vulnerable, disadvantaged children on earth.

So, yes, question #4 is addressed in the news article—but the problem is the intro is entirely presented from the perspective of the boosters, the people who have a stake in selling their idea. It’s ok when giving a talk to sell your idea—there will be others out there to present their own, different views on the topic. But a newspaper should do better.

Palko continues:

Later on in the piece, Tyre actually does start digging into the business model, where the key drivers seem to be using cheap, substandard buildings, hiring undertrained and unqualified instructors, employing strong-arm collection tactics, and doing lots of marketing.

So Palko’a complaint is not with all of the article, just how it starts. Still, I think he’s making a good point. When we hear bold claims, our starting point should be disbelief. Assume the claim is false until substantial evidence is presented to the contrary.

P.S. Palko points here to another example of ridiculously credulous journalism.

23 Comments

  1. morton says:

    “Extraordinary claims require extraordinary evidence” … a phrase popularized by Carl Sagan, though originating with Laplace and David Hume.
    This concept is central to the scientific method and critical thinking.

    “Journalists are faced with the daily choice of painstakingly
    researching stories…. or writing whatever people tell them.
    Both approaches pay the same.”

    — Scott Adams

  2. Jeff says:

    The principle is hard to argue with, but the Peg Tyre passage above doesn’t read to me as overly credulous. It’s framed as “people are claiming all sorts of things, which, if true, would be a big deal.” This seems like a pretty reasonable way to get into the subject, since it helps to explain why you’re even writing the story.

    The Rolling Stone piece at the other link is another matter.

    • Alex says:

      This is literally a counter-example to the problem the post claims to illustrate! Even if you read only the offending passage, which would be unfair, all the claims are in scare quotes, which clearly signal skepticism. As we’re legislating “immutable laws,” we would be well-advised to think things a bit more carefully.

  3. Ema says:

    Don’t you casually dismiss the work of researchers in unfamiliar fields all the time?

    • Andrew says:

      Ema:

      The above quote was: “when someone with no track record comes into a field claiming to be able to do a job many times better for a fraction of the cost . . .”

      When I criticize work in areas outside my fields of research, I do have a track record, as my criticisms involve issues of statistical modeling, causal inference, and social science, where I’ve written a lot and done a lot of research.

      So, sure, I don’t know anything about ESP, but I’m still willing to say that I could learn just as much about ESP at a fraction of the cost of Daryl Bem’s work—but I make that claim based on my experience and understanding of statistics. Similarly with my criticism of the work of S. Kanazawa, etc.

    • Andrew says:

      Also, just to be clear, I don’t think there’s anything wrong with outsiders with no track record coming in and offering radical solutions. I just don’t think journalists should be reporting such claims uncritically. For that matter, if a journalist was reporting on my claim that I could do ESP research at a fraction of the cost, etc., then I think the journalist should report other perspectives too.

  4. ZC says:

    This applies to academics, too, as you of course know. As an economist, my eyes threaten to roll out of my head every time I see The Guardian (definitely the worst for this, although not the only one) promote some new crank who is seeking to ‘rethink economics.’

  5. Anonymous says:

    This is a very complex and important issue. The currency of improvement, in science or civilisation, is Good New ideas. These are worth making sacrifices for. The biggest threat to good new ideas is those who feel they are doing Good Work by destroying or denigrating anything new they come across, because they know they’ll get kudos through looking wise incredulous, and wonderfully skeptical.

    In fact we don’t need skeptics, we need Fair and Expert adjudicators. Thumper’s rule for sideline commentators might be: “If you can’t bring adequate expertise to an issue, don’t say nutt’n at all.” There are far too many out there who feel entitled to comment but who can only contribute inertia, which at best will be random. Also, it IS worth letting dubious stuff dawdle for a while, because almost no-one can quickly tell the difference between a really good new idea and rubbish. The immune system leaves some kind of chemical windows open at the end of the gut to get controlled “peeps” at new substances; this buys extra knowledge at some little risk. Most animals are vulnerable to viral attacks but the defenses could be much higher. Sharks I believe have raised those defenses and a much less vulnerable. I don’t want to denigrate sharks but much evolution has been contributed by viral inserts.

    Here’s what I said in chapter 2 about that dangerous old slogan “Extraordinary claims…”:

    “By the criterion of better explanation, we see that “Extraordinary claims require extraordinary evidence” as the otherwise sometimes wonderful Carl Sagan (1996, p. 213) is variously reported to have echoed, is unfortunately not right. Copernicus’ heliocentric solar system was already better at explaining the old evidence – it didn’t need any new observations. That’s not an unusual counter-example, and Sagan should have known it. As Popper said, extraordinary new theories that explain more and cannot be refuted, are better (not just because they explain more, but, as they’re “extraordinary”, they make more improvements in our belief structures). But their acceptance should still be solely on the basis of the standard of their prediction. Degree of unusualness need not alter our requirements for acceptance: the balance of our preference might still be tipped by one tiny observation, a reappraisal of the evidence, or even just a minor enhancement of our theorising skills. If “claims” refers to evidence, the saying is a tautology, so “claim” there means theory. In that case the slogan implies we’ve always already got the best possible theory to describe any current evidence, since it says we always need new evidence to change our minds. But to imply all our theories and theorists are and have been perfect is a long, long way from reality. It wasn’t true in Copernicus’ time, and in historical sciences like palaeontology it never will be a sound assumption. See my blog posting http://sciencepolice2010.wordpress.com/2011/12/01/even-the-good-guys-extraordinary-claims-require-extraordinary-evidence-really/
    …for more on this slogan and its history.”

    The worth of such strategies can be modeled and evaluated. When it is, I think people will be surprised.

  6. Eli Rabett says:

    Shorter: A consensus of experts has value

    • Andrew says:

      Eli:

      Yes. Also, it’s not necessarily true that a set of well-connected rich people represents a consensus of experts.

      • Mark Palko says:

        Exactly. This is one of those types of stories where you can almost guarantee the “well-connected rich people” aspect will show up. Another is the terrible business idea that’s raised 100 million in funding (see Soylent, Bodega, raw water).

        On a related note, whenever I read about someone particularly incompetent who has landed a plum assignment at a relatively young age, I’ve gotten in the habit of checking their biographies. A remarkable number of them come from elite private schools. Probably just a coincidence.

  7. Mark Palko says:

    I’d forgotten about that thread. When I saw the title in the list of your upcoming posts, I assumed it was something from this:

    http://observationalepidemiology.blogspot.com/2017/11/hyperloops-red-flags-deep-dark-tunnels.html

  8. Thanatos Savehn says:

    Andrew: “Science for $500, Alex”
    Alex: “We reliably sort the Michael Faradys from the Uri Gellers not by credentials, but by this.”

  9. Christian Hennig says:

    The current bulky six lines long posting titles look like some kind of experiment to me; what’s that about?
    If it’s for inciting feedback, I can say that I’d certainly prefer that space to be taken by a cute cat picture.

  10. Cody L Custis says:

    Did those instructors use power pose as the explanation in a TED talk?

  11. Kaiser says:

    Does anyone remember that NYTImes article trumpeting some economist who “solved the homeless problem” with a solution so simple that it was incredible that no one else has thought of it? A proper media company should have done a follow-up on that claim by now

Leave a Reply