Grrr, this is annoying. A piece of exaggerated science reporting hit PPNAS and was promoted in Politico, then Kaiser Fung and I shot it down (“Could Google Rig the 2016 Election? Don’t Believe the Hype”) in our Daily Beast column last September.
Then it appeared again this week in a news article in the Christian Science Monitor.
I know Christian Scientists believe in a lot of goofy things but I didn’t know that they’d fall for silly psychology studies.
The Christian Science Monitor reporter did link to our column and did note that we don’t buy the Google-can-sway-the-election claim—so, in that sense, I can’t hope for much more. What I really think is that Rosen should’ve read what Kaiser and I wrote, realized our criticisms were valid, and then have not wasted time reporting on the silly claim based on a huge, unrealistic manipulation in a highly artificial setting. But that would’ve involved shelving a promising story idea, and what reporter wants to do that?
The Christian Science Monitor reporter did link to our column and did note that we don’t buy the Google-can-sway-the-election claim. So I can’t really get upset about the reporting: if the reporter is not an expert on politics, it can be hard for him to judge what to believe.
Nonetheless, even though it’s not really the reporter’s fault, the whole event saddens me, in that it illustrates how ridiculous hype pays off. The original researchers did a little study which has some value but then they hyped it well beyond any reasonable interpretation (as their results came from a huge, unrealistic manipulation in a highly artificial setting), resulting in a ridiculous claim that Google can sway the presidential election. The hypesters got rewarded for their hype with media coverage. Which of course motivates more hype in the future. It’s a moral hazard.
I talked about this general problem a couple years ago, under the heading, Selection bias in the reporting of shaky research. It goes like this. Someone does a silly study and hypes it up. Some reporters realize right away that it’s ridiculous, others ask around and learn that it makes no sense, and they don’t bother reporting on it. Other reporters don’t know any better—that’s just the way it is, nobody can be an expert on everything—and they report on it. Hence the selection bias: The skeptics don’t waste their time writing about a bogus or over-hyped study; the credulous do. The net result is that the hype continues.
P.S. I edited the above post (striking through some material and replacing with two new paragraphs) in response to comments.
A search giant influencing a presidential election is a plot line on House of Cards, season 4.
I liked that Kaiser and Andrew’s rebuttal said, “The question, though, is not whether there is some effect but rather how large it is.” That ties it neatly to the previous post, http://statmodeling.stat.columbia.edu/2016/07/18/29529/
I don’t think I can blame the reporter at all: I read the arguments in your Daily Beast column and although I’m convinced I cannot claim that *everyone* should be convinced.
It’s like asking me to decide which side of a debate in linguistics is right. If I have no expertise, the best I can do at short notice is to judge based on credentials. If fact, when two “experts” disagree, trying to decide who’s right based on my own naive, uneducated judgments might be a disservice to my readers. The better option might be to just present both experts after making sure they are authoritative and not quacks, crackpottery, fringe etc.
There’s Andrew on one side, Epstein on the other. Both authoritative. The work itself has been peer reviewed. Agreed, that peer review can be wrong but can a non-expert judge whether the peer-reviewer was right or a challenger expert?
I mean when the peer reviewers & editors thought Epstein *may* have something going in his hypothesis why should we assume that any lay reporter ought to be definitively convinced otherwise?
Christian Science Monitor is a media that has nothing to do with Christian Scientists.
Not exactly “nothing to do” with Christian Scientists, as it was started by Mary Baker Eddy. See here:
https://en.wikipedia.org/wiki/The_Christian_Science_Monitor#Concept_and_inception
But from what articles I’ve seen in there the reporting seems decent. Doesn’t seem like the usual Christian Science crackpottery.
Yes, the reporting in the Christian Science Monitor has always been considered to be of high quality. I was just pointing out that it is an exaggeration to say that it has “nothing to do” with Christian Scientists since it was founded by a Christian Scientist and every issue will have an article on a religious topic written by a Christian Scientist.
As far as its news coverage and other secular commentary is concerned, it has always had an excellent reputation for objectivity and accuracy.
> Other reporters don’t know any better—that’s just the way it is, nobody can be an expert on everything—and they report on it.
And yet other reporters believe it’s ridiculous but write the story anyway because to the media it’s about clicks and impressions and to the journalist it’s about career advancement.
Anon:
Yeah, it could well be like that horrible NYT article that those economists wrote last year about crowd noise in basketball. They should have and possibly did know better, but they probably figured, hey, it’s just a fun diversion, who cares? All they did by publishing that article was to degrade science communication in some very small way. The article about Google swaying the election bothers me more, in that they’re doing their part to degrade public trust in democracy.
Inflated reporting. I reached a similar conclusion independently when reading Epstein Aeon piece.
Epstein sometimes sounds like an activist on the subject which I’m not enthusiastic about
Interestingly, another study I followed assist found extremely high copying effects by unconscious priming.
(Flag priming study by Ran Hassin and friends. Later, a non-vetted replication failed (2014), with an argument ensuing
But the initial effect was huge (surprising the researchers themselves by far)
Voting might be vastly more fluid and heuristic based than rationality fantasises. Heuristic researchers have shown that voting is based mostly on very general heuristics (right left spectrum positioning) rather than detailed investigation
I think you’re being a bit harsh on Christian Science Monitor here.
That search engine study went all over the place, so it’s not like they’re raising it up out of obscurity. They literally include your quote that it is “poor science” as the last word on the matter, so I think the article is hinting that the reader should probably dismiss concerns that people have been raising based on the study.
I read the article as saying:
* Some people have said that Google might be influencing elections based on their search results
* but independent scientists say that these fears are overblown and bad science
That seems like a reasonable summary of the scientific argument and leaves readers with the impression that the study was overhyped.
+1
All said, I think Christian Science Monitor did a good job here.
The bit about “But that would’ve involved shelving a promising story idea, and what reporter wants to do that?” is also kinda unfair and mean to Rosen. I see no hint that he compromised anything here just to push a story.
Jon, Rahul:
I don’t think it was a good news story because I don’t think such a ridiculous claim should’ve been reported on at all. That said, what’s important to me here is the substance (there really is no evidence that Google could sway the election) not the reporting, so I altered my post accordingly.
If it is too ridiculous to get reported on how & why does it ever get published! Who are these ridiculous people sitting on editorial boards? Who are these ridiculous reviewers?
One might condone subtle errors sneaking through the peer review process but why are we getting obviously ridiculous material through?
In fact, why are we appointing ridiculous people to National Academies?
Lots of ridiculousness to pass around, eh! Well we cannot blame external stakeholders (e.g. journalists) when we are doing such a ridiculously bad job in the academic enterprise.
Andrew, I think that your post may be doing your own point a disservice. You state that Rosen should have put in the effort to consider the contents of Kaiser’s and your article, realized the criticisms were correct, and therefore discarded his story.
However, the second paragraph of your post clearly shows you have not put in the effort to learn what the Christian Science Monitor is: a newspaper respected for decades in journalism. If I were Rosen, reading your suggestion that I should put in more effort and thought when writing my science news articles, then I would likely be turned off by the observation that you have not put in the effort to learn what my newspaper is — even though the Christian Science Monitor itself is very widely known. Your post would appear somewhat hypocritical to me, as if you were criticizing others for not putting in the care and effort to consider ideas carefully, while at the same time exempting yourself from that standard by feeling free to judge organizations based on their names (“Christian Science…”) alone.
You then insult the religion associated with the newspaper (” …Christian Scientists believe in a lot of goofy things…”) for reasons that, regardless of whether they are correct, are irrelevant to the topic at hand.
All in all, trying to imagine things from Rosen’s perspective, this post may make him less likely to pay attention to your points and arguments in the future. For example, if he does not understand the details of the scientific arguments in your work, then the observation that you have been somewhat careless and unnecessarily insulting in a subject he does understand — this post — may make him suspect that you are also somewhat careless and unnecessarily critical in your scientific work. Other reporters may be affected similarly. I find this unfortunate because I agree with many of your criticisms.
I apologize for the length of this comment, but I have tried explain my points very carefully and politely, so they may be understood unambiguously. You may feel free to publish this comment or to choose not to publish it, as you wish — either is fine with me.
Not sure I agree with the argument in the “However, …” paragraph, but the points in the next two paragraphs are good ones.
Hi, yes, see P.S. above; I altered the post.
Given the incentives to report hyped results, maybe they *should* be reported – as a ridiculous claim or as hype unsubstantiated by the actual research.
You’re basically doing the same thing, but more indirectly, i.e. after the ridiculous claim or hyped result is reported ‘straight’ first by others.