Skip to content
 

Scientific communication by press release

Hector Cordero-Guzman writes:

I have a question for you about an ongoing congroversy\incident related to reporting of social science research.

Please see article linked below if you have a chance. I think this incident exposes real problems in the way social science research is presented and how it reaches the public…

http://www.latinorebels.com/2014/05/22/new-york-times-piece-on-hispanics-and-census-based-on-study-not-yet-finalized-or-public/

Essentially, we have a story in the NY Times based on a blogpost of a presentation made at a professional conference (that requires papers) that no one has seen and on a study no outsider has apparently reviewed. I asked the authors for the paper and they refused to share it. They did say they could talk to me on the phone about it…

Even after all of the concerns that were raised in the link below (including to the editor of the story at the NYT) and in other forums about the study\report not being public or available for outside academic experts and the authors not wanting to share it with me or others in the press to review, the New York Times went ahead and put this on Twitter on SUNDAY (see below)!!! To their over 11 million followers!!!

I told the authors that the way this has been handled has been very poor and is not and should not be best practice in terms of how scientific research is reviewed, released to the public, covered in blogs and the media, or discussed.

If they presented work that was not for discussion or attribution (at an academic conference that requires papers for presentations, BTW) then Pew should not have written about it and that should have been very clear to all…If they presented work at an academic conference at least their presentation should be immediately available for others to review. The more potentially damaging or stigmatizing or controversial the work the more authors have to be careful how they put it out in public.

The process broke down and my sense is that the research team (including, potentially, co-authors that work at the Census Bureau) has some responsibility for that.

Peer review and transparency are essential for science. If authors are presenting something, they should take responsibility for it and let others see the work. If it is not ready then it should not be presented or it should be discussed at a faculty seminar or workshop with clear ground rules…

The process was either not followed or broke down in this instance. Neither is good.

[And, I want to be clear that I have no objection to the study or the findings—whatever they are. I have only seen what is reported but have no way of judging accuracy, veracity, validity, reliability or anything else and that—to me—is the biggest problem right now…]

Do you agree? Is this the way to present social science research to the public?

I do not want to make a bigger public issue about this at the moment (though I have been quite vocal on twitter and facebook about this) but would realy appreciate your expert assesment, opinion, and any suggestions you may have.

I don’t want to feel that I am going after coleagues unfairly or making an issue where one does not really exist.

Cordero-Guzman also sends along this link and this one.

I have mixed feelings about this sort of story. On one hand I agree with him that there are problems when scientific communication is conducted via press release rather than peer review. And it’s not good when people won’t share the articles on which their conclusions are based. On the other hand, if the topic is important you can’t really wait until the article is accepted for publication. In some fields (such as statistics and political science) the publication process can be slow. Also, as we keep discussing, there’s nothing so special about publication. All that publication does is to fix a paper in some permanent form; it doesn’t mean the result is correct or even reasonable.

18 Comments

  1. Rahul says:

    If Cordero-Guzman is so sure this conference *requires* papers to be submitted, why not ask the Conference organizers for a copy? That seems the natural next step.

    Also, by presenting at a Meeting of this sort isn’t it generally acknowledged that most talks are typically about work in progress? I read the original sources (Pew & NYT) and happily in this case both seem to have made the provenance of the work (and the fact that it is as yet unpublished) quite clear at the outset of their articles.

    I’m not sure who to blame.

  2. av says:

    The paa conference requires either full papers or extended abstracts to be submitted about 6 months in advance of the meetings, and nominally expects authors to upload the full paper a few weeks before the conference. However, compliance with the latter expectation is typically low (I’d guess 60%). In part this owes to the long time to publication in sociology, econ and other fields that demographers call home – people don’t want to be scooped, which really does happen – part owes to worries about journal outlets considering the work ‘published’ if it appears in full somewhere like this – senior collaborators routinely advise me against posting papers on arxiv as well as places like the paa website for this reason, which I think is misguided but it’s a general sentiment in the field at least – and a part certainly owes to laziness or the paper not ending up too boring to write up. I don’t see anything too wrong with this state of affairs though, because it allows work that is a little more cutting edge to be presented, enables researchers to receive feedback before finalizing and submitting a paper, and enhances the meetings substantially (imagine if full papers only were required for submission six months ahead of time).

    Now, as to them not sharing the paper, I think this is generally poor form but it should be kept in mind that one of the authors works at the census bureau and the data comes from the census, so there are likely employer restrictions and various hoops that must be jumped before the results can be disseminated in manuscript form. I’d give them the benefit of the doubt on this one.

    However, I do think it sensible that news outlets should have a clearer policy about reporting on articles that are not published in peer reviewed journals or available in some other way (eg arxiv or ssrn) for mass consumption.

  3. MJ says:

    “On the other hand, if the topic is important you can’t really wait until the article is accepted for publication.”

    I have already said that, but I find that the economics profession (whatever one thinks about it in other regards) handles this very well: there are several depositories (CES, NBER etc.) where a paper is published as “working paper” or “discussion paper.” This is very useful in that it also show the intellectual trajectory of an idea, e.g. one can also see what has NOT worked out (sometimes in that it changes, sometimes in that the paper is simply abandoned) and where the author modified an idea following a suggestion from someone else (though this might not be acknowledged, unfortunately). Sometimes a working paper can also be more clear and detailed about a topic, because a journal restricts space to fully develop an idea.

  4. Rafael Garcia says:

    As I understand it, presenting unfinished work is common practice in my field (psychology), even when conferences “require” papers (many of the reasons have been expressed by previous commenters). Now those presentations ought not be snagged by popular press and disseminated, but one should not be surprised that it happens. The popular press is trying to make headlines, while the researchers may be trying to get feedback or (more likely) justify their use of department funds to attend the conference. As long as the popular articles (and all press that echos the original popular article) acknowledge that the paper is unpublished or work in progress, I see little problem with the practice.

    As av stated, not sharing the paper is in bad form – assuming a paper was/is/will be in existence. A far more worrisome practice, in my eyes, is citing work that was never published. One case (I cannot recall the details, though Lee Sechrest might be able to) had cited an unpublished study of the first author from the 1970s (the citing paper was from 201X). Obviously, being published in a journal is not the end-all, but, as near as my Google search could tell, the paper from the 1970s simply did not exist anywhere but in reference sections (of at least 3 articles).

  5. Jonathan Gilligan says:

    I agree that some results are too important to wait until publication, and that publication is not some magic status. But I cannot think of anything that would justify publicizing the results without making the methods and data available for people to scrutinize. There are plenty of places, such as arxiv, SSRN, and authors’ personal web sites, for making unpublished manuscripts and working papers easily accessible.

    The results are NOT important enough to share if the empirical and methodological basis of those results is not also important enough to share.

    If the justification for publicizing the results before peer review and publication is that they bear on issues of great concern to the public, then incorrect results can cause a lot of harm. Thus, it is important to quickly identify and correct errors and getting more eyeballs on the methods and data is essential.

    • There are obviously going to be a lot of issues when you are working at the Census Bureau and have access to data that has legal requirements for not being shared directly etc. I’d say cut the Census Bureau some slack in terms of allowing time to get that properly squared away before being too vehement about non-sharing. Ultimately, I agree with you, but a month or so in the case of legally sensitive Census results isn’t too much to expect.

      • Rahul says:

        Fine, but OTOH the researchers might have waited a few months for the next conference.

        I think the default expectation ought to change: Currently too many academics start scrambling to provide data, methods etc. after being asked for it. The expectation ought to be that once you speak about your results people *will* ask for supporting information. So the homework ought to start before such conferences and not after.

    • Rafael Garcia says:

      No disagreement here. The document can be unpublished (in the not-in-a-journal sense), but it ought to be somewhere.

  6. Steve Sailer says:

    The NYT piece was “More Hispanics Declaring Themselves White.”

    In general, articles in the American press about Latin Americans and race are hopelessly confused and obtuse because 21st Century American English lacks useful terms that can help make sense out of the subject like “mestizo” and “mulatto.”

    Thus, the media’s fiasco in calling George Zimmerman white.

    • Rahul says:

      Perhaps color identity itself is getting complicated? I know a kid whose two grandmothers are British & Chinese & grandfathers (Asian) Indian & Jamaican.

      What color do I call him? Light-brownish-black with a tint of yellow?

      • Martha Smith says:

        Yes, color identity is getting complicated. Some parents are responding to this by using the word “blended” or ad hoc words for their children of mixed ancestry — see, for example http://mixednation.com/.

        • Rahul says:

          Some people take the fact that “color identity is getting complicated” just as a factual observation. Others are positively happy that color identity is getting more complicated that a black-white simplicity.

          Steve Sailer OTOH I bet he’s sad that he can soon no longer weave his black-white simplistic stories. It’s much harder to spin those tales in a world with a RGB continuum as Fernando puts it.

          • Anonymous says:

            Exactly the opposite. The Sailer types love to study the varieties of man in all their details, like Darwin on the Galapagos Islands.

            It’s those who have baught into the ideology that we’re all essentially the same who are reluctant to notice differences. After all why waste your time noticing differences and details when dogma says they have to be inconsequential?

            It might shock you to learn that some in the latter category believe everyone in the former category are inherently racists and will denounce them so at the drop of a hat.

    • Fernando says:

      If you are really obsessed about color you might want to use a continuous scale, something like an RGB code. Then include it in their immigration application so you can fine tune the color of the nation via immigration policy. All in the name of “science”.

      PS I don’t know what language you are using but in Spanish it is spelled “mulato” with one t.

      PPS BTW you should review your ethics 101. An IS like a scientific fact never justifies an OUGHT. (Of course in this case “the fact” is up in the air.)

  7. We present work at various stages of development. if the authors are not prepared to make the work publicly available, in the conference proceedings or at SSRN or NBER, say, the work, including the presentation, should be identified as unavailable for citation or quotation. However, if it’s sexy and the press covers the presentation, that won’t stop them from reporting it. Their norms are not our norms. At the same time, I appreciate the desire to hold one’s data and codes closely while a work is under review. In most cases, serious review of data driven research cannot be carried out until the data and codes are made public. Nowadays, it is common practice to report that info online when the journal posts the paper.

  8. Hector Cordero says:

    Interestingly, the article was finally published this week…my main point, made a few weeks ago, is that if an author or team want findings covered in the press they must produce a paper or materials that others can see, review, and critique. It’s not really that complicated. Put up or shut up–but don’t put findings out there that no one else can see or review…that goes against the open scientific process…

    http://www.census.gov/srd/carra/Americas_Churning_Races.pdf

    • This report really suffers from lack of good graphical summaries of their data. The few graphs they have are ok, but there are a huge number of ways that they should have sliced up their data and displayed it compared to the 3 or 4 graphs they actually use.

      Examples:
      histograms of number of people who answered with n races in 2000 and 2010

      histograms or bar charts of number of people who answered as single race in 2000 that chose multiple races in 2010 split out by the race they chose in 2000 (panel) and the number who chose each additional race (bar)…

      Their big matrix with blue/black shading just doesn’t give the details.

      Finally, I’m with Rahul, race is not a very simple thing in the US.

Leave a Reply