Skip to content
 

Higher credence for the masses: From a Ted talk?

The Four Most Dangerous Words? A New Study Shows | Laura Arnold | TEDxPennsylvaniaAvenue

I brought this link forward in some comments but wanted to promote it to a post as I think its important and I know many folks just do not read comments.

As I once heard claimed in a talk on risk communication – “No one has as much credibility as a mother voicing concerns about her children – No one!” Now, academics have been voicing concerns about the quality of published studies and claims for a long time, me in 1989, others inspired by Fisher in 1959 and CS Peirce in 1879. Recently there has been an exponential explosion in the voicing of concerns about  the quality of published studies – but it does not seem to be reaching the masses. Academics must not have much credibility with the masses?

Now, in this TEDx talk we have someone that most folks will likely believe has the money to be a willing philanthropist and who does want that money to do some good, if not the maximum amount of good. They voice an inability to do that simply because they perceive the evidence in published studies is no where near adequate quality. In fact, that is where they now believe they need to spend their money (nuisance funding – what they wish they did not need to do) in order to have some hope in the future to fund ways to make the world better (interest funding – what they want to do). They cite some examples that have been discussed on this blog and even parody a TED talk on one of them.

Will it have high credence with the masses? So far less than 25,000 views – does not seem like the masses are there – yet.

p.s. Revised: In the original post I had put “washed up Wonder Woman” as the subtitle [which the editor replaced with “a Ted talk”] as it appeared as  a rude comment about the presenter’s appearance and I thought it would be a good example of why many people just don’t read comments. Originally writing here  – so I understand why many do not want to read comments – in 2017 we would hope a woman can give a talk and not have her appearance disparagingly commented on.  On this blog though many of the comments are worth reading and one way to slowly wade in is to start reading comments from some commenters and just keep with the ones that seem worthwhile.

p.s.2 Around 2005 I had a conversation with the president of a large disease based charity about how naivety in terms of running clinical trails (e.g. implementing/managing randomisation schedules, tracking trial progress, data entry quality control, engaging statistical consulting, reproducible analyses, etc.) of the clinicians they were funding was wasting a lot of their funds. I argued that this could be fixed rather inexpensively by funding some common resources for them to draw on.  They agreed this made a lot of sense but was sure it would not look very good to their donors as they would expect the money would only be directly spent on disease X. They did say they would see if other disease based charities might consider jointly pooling some resources.

Never heard back, but I think I started to understand why funding improvements in research quality was seen as unattractive.  (Only a very small percentage of grant submissions on reproducible research were being funded back then and I was perplexed about that. ) But just the appearance of high quality (if  credible enough to the masses) was more than adequate to make the funding system run smoothly (e.g. charities, funding agencies, research institutes, universities and governments). That is, poor research was likely to get as much positive publicity if not more than more expensive to fund higher quality research.

Its the really needing to depend on using the research that makes the real quality matter enough to be willing to risk investing in it in a world where research quality is poorly understood.

23 Comments

  1. Cody L Custis says:

    You lost me at TED talk.

    • Andrew says:

      Hey, Raghu, don’t blame me for this one!

    • Keith O'Rourke says:

      Raghu and Cody:

      In ideal inquiry (where one really wants to get at the truth above all) the exclusion of sources of engagement up front (with extreme prejudice) is a mistake. As it was explained to me when at Oxford, you cannot ignore comments from another faculty member but you can choose to dismiss them (apparently some slap on the wrist would occur). The point was dismissal requires some assessment of the comment.

      I’ll wait for some more comments, assess them and then try to be more concise and direct about what I found of value in this TED talk.

  2. She does look like Lynda Carter and mentioned Wonder Woman in her riff on Power Poses. And the mass media-bad epidemiology nexus is long discussed. Also people perceive scientific research as ‘at the bench’ when much of it, especially in clinical, is in the logistics. And educational and program research is very difficult because of all the sources of variability. She talks about a program where kids learn the alphabet but how many kids learned it without the program?

    She is right about anecdotes. Story-telling is a way of connecting and describing the topic but should not be a driver of decisions. For example there are all these documentaries about someone who starts a sports program in some poor urban area, anywhere in the world. And of course they become champions at the end. But nobody makes a movie about the sports program that doesn’t take off or the kids who get cut, get injured, or feel left out of the new ‘hot’ sport.

    • Brad Stiritz says:

      “Washed-up Wonder Woman” ..
      >in 2017 we would hope a woman can give a talk and not have her appearance disparagingly commented on.
      >>She does look like Lynda Carter and mentioned Wonder Woman in her riff on Power Poses.

      Agreed on the resemblance. The caption strikes me as highly complimentary, looks-wise. Yes, it’s back-handed and mean-spirited overall, but certainly not about her appearance.

  3. Anonymous says:

    John Oliver did a pretty good (if light) take on all of this last year; it’s at 9.5 million views: https://www.youtube.com/watch?v=0Rnq1NpHdmw

    • Keith O'Rourke says:

      Thanks – very entertaining!

      The main message I get here though is that the problem is mainly poor reporting and processing of apparent evidence along with real evidence with real evidence being discernible by well motivated experts.

      The message I get in The Four Most Dangerous Words is that real evidence is not being adequately produced or we can’t extract that even with well motivated experts – so we have to divert resources into fixing that before we can get real evidence to do what we really want to.

      • Ben Prytherch says:

        Keith, I (“Anonymous” above – forgot I was using a different computer when posting this link) completely agree. The Four Most Dangerous Words video raises issues that are more in line with what is emphasized on this blog. The John Oliver video makes it sound like if only it weren’t for these sensationalist, naive science journalists and unscrupulous researchers, there wouldn’t be much of a problem. And that doesn’t really capture the heart of the problem. But hey, 9.5 million views is impressive, and I don’t think that big a chunk of the general public would have the patience to absorb a video on Type M errors and forking paths and the consequences of misinterpreting p-values. Getting the public to be more aware and skeptical of sensationalist, naive science journalists and unscrupulous researchers is a worthy goal in itself.

  4. Corey Yanofsky says:

    Since the source is TED, a reasonable case can be made that the exclusion is with extreme judice ;-).

    • Keith O'Rourke says:

      I would say TED is the medium not the source (nor the message as McLuhan argued) but the TODD talks in anonymous’ link do make clear there likely will be confounding. (Some have expressed surprise that TED let Laura’s talk go through.)

      Especially when there is confounding I would not appoint a judex but rather a statistician ;-)

      • Martha (Smith) says:

        ” (Some have expressed surprise that TED let Laura’s talk go through.)”

        As I understand it, the talk was a TEDX talk, not a TED talk, and (if I am not mistaken), TED may not have as much say in the content of TEDX talks as in TED talks.

        • Keith O'Rourke says:

          Good point – TEDx talks are from the colonies and likely given more latitude* – though early in the talk some arguments are presented that suggest most TED talks if not misleading are unhelpful.

          * like when I was at Oxford and my cultural transgressions were overlooked.

  5. jim says:

    If people want to make a diff in the world and the social research isn’t effective they should: A) Use common sense intuition and do what they have to do or B) just start a business and hire the people you want to help. IMO most philanthropy doesn’t work in the long run. What works is a strong economy where people have opportunities

    • Peter says:

      Well, Jim you did not seem to have watched this Ted Talk. Invocations of common sense as a guide to what to do are silly (or just beg the question). Remember that common sense is the thing that tells us that the earth is flat. There are tons of claims that seem commonsensical but have been shown to be wrong (build more roads to alleviate congestion, hormone replacement therapy as the fountain of youth, rent control to make housing affordable, etc.).

      I suggest Duncan Watts’ book “Everything Is Obvious: How Common Sense Fails Us” (2012), also published under the title “Everything Is Obvious, once you know the answer”.

      • jim says:

        Well, Peter, there are tons more claims of “common sense” that have proven to be right. And there are claims of common sense proven wrong that are subsequently proven wrong.

        If common sense was wrong you’d have a hard time finding your way out of bed in the morning and humanity wouldn’t exist. In fact, we use commonsense to design experiments to learn about things we can’t sense directly.

        It would be fun to trash that book. Everything. **is** obvious once you know the answer. Common sense is what helps you find it.

      • jim says:

        A good example of “common sense” that was “proven” wrong and subsequently “unproven” wrong is the purportedly damaging effects of cholesterol in eggs. Generations of long-lived farm families knew that.

        A good example of science trying in vein to invert common sense is the attack on meat eating. (I knew several grad students who were evangelical veggies.)

        Another effort that I suspect will ultimately fail is the current attack on lecturing as a form of instruction. Humans have been learning from voice instruction for millenia, surely other approaches have been tried and ultimately abandoned many times.

        • Peter says:

          Jim, you are missing the context of this TedX Talk and you are misunderstanding me when you reply “If common sense was wrong you’d have a hard time finding your way out of bed in the morning and humanity wouldn’t exist.”

          Try to make fun of Watts’ book! You will find that difficult because Watts’ isn’t arguing that people are stupid, in general, because they rely on common sense which is faulty more often than not. Watts’ starts out by talking about natural scientists who imagine that they could easily solve the problems social scientists work on which Watts thinks is wrong because these social problems are often really hard.
          Laura Arnold, the TedX presenter, does not talk about how can we solve the problem of getting out of bed, putting food in our mouths, etc. She and her husbands are philanthropists who want to fund initiatives that solve difficult problems, problems a simple application of common sense cannot solve. Are we going to fight cancer or Alzheimers’ disease by simply letting us be guided by common sense?
          I recommend that you watch this TedX talk. It’s worth it. I did it, and I don’t regret spending 18 minutes on it. Also, Keith would not have posted about it if it was so simple!
          (Note: Watts has a Ph.D. from Cornell in physics and was a professor of sociology at Columbia University before becoming principal researcher at Microsoft. So, like Keith, he’s no dummy.)

  6. Trevor Butterworth says:

    The Laura and John Arnold Foundation funds the Metrics center at Stanford—and many other projects to improve scientific integrity, including the one I run. If you have any great ideas for a project, I’m happy to make an introduction to a program officer.

  7. Tom says:

    “I argued that this could be fixed rather inexpensively by funding some common resources for them to draw on. They agreed this made a lot of sense but was sure it would not look very good to their donors as they would expect the money would only be directly spent on disease X. “

    Charities are heavily scrutinized over their ‘overhead ratio’. People don’t want to fund lavish expenditures on routine administrative staff. Executive salaries, travel, etc and fund raising. If the common resources are part of the administrative budget, it would tend to fall into overhead. However, administration of research is research. This is simply accounting for costs. Of course, I don’t know if this is really a problem, but it isn’t a surprise that anything that would show up in the dreaded ‘overhead ratio’ would not be popular among charity administrators.

    Also — good Tedx talk.

  8. Platform Knowledge says:

    Talking causality: One should know the technical details of the platforms they are using to distribute content.

    The video is not listed. This means that one cannot find it through browsing or searching YT. Only by clicking a direct link. I just take the guess that this may explain some part of the difference between the funny cat videos and scientific critique.
    It is also not listed on the channel that published it.

    So, instead of wasting our time to rant about the lack of views due to unobserved measurement error (e.g., inability to set correct YT video settings), we could alternatively increase our knowledge to spread knowledge properly.

    • Keith O’Rourke says:

      Interesting (there are many things I should know but don’t).

      > increase our knowledge to spread knowledge properly
      Given we think the general public could help improve the quality of evidence (e.g. making demands on elected officials) that is something we need someone to do.

Leave a Reply