Post-publication review succeeds again! (Two-lines edition.)

A couple months ago, Uri Simonsohn posted online a suggested statistical method for detecting nonmonotonicity in data. He called it: “Two-lines: The First Valid Test of U-Shaped Relationships.”

With a title like that, I guess you’re asking for it. And, indeed, awhile later I received an email from Yair Heller identifying some problems with Uri’s method. After checking with Yair, I forwarded his message to Simonsohn who found the problem and fixed it. Uri’s update is here.

Now, I don’t actually agree with Uri or Yair on this one: I don’t really buy the hypothesis-testing, type-1-error framework that they’re using. But that’s ok: it’s not my job to vet their methods. If these ideas are useful to others, great.

My real point here is that post-publication review really worked. Uri posted something, Yair had a criticism, Uri responded. And if Yair’s criticism had been fatal, I’m pretty sure Uri would’ve acknowledged it. Cos that’s how we roll. Open discussion, open data, open code. It’s what science is all about.

9 thoughts on “Post-publication review succeeds again! (Two-lines edition.)

  1. “Open discussion, open data, open code. It’s what science is all about.”

    That’s definitely not the case in certain parts in the field of publication ethics. My point of view is clearly demonstrated in a ‘Discussion Note’ with the title “Is partial behaviour a plausible explanation for the unavailability of the ICMJE disclosure form of an author in a BMJ journal?” which was published this week in the journal ‘Roars Transactions, a Journal on Research Policy and Evaluation’ see https://riviste.unimi.it/index.php/roars/article/view/9073 (open access).

    The findings in this ‘Discussion Note’ are mainly based on ‘no response’. The editors of ‘Roars Transactions, a Journal on Research Policy and Evaluation’ are welcoming comments and responses on my ‘Discussion Note’.

    • Sameera Daniels wrote: “Publication ethics is fascinating area.”

      Yup.

      All at COPE, the Committee on Publication Ethics, https://publicationethics.org/ , have decided already a very long time ago that they do not communicate with me, and that they thus also not communicate with me about the last sentence in the Abstract on my paper at https://riviste.unimi.it/index.php/roars/article/view/9073 in which I state:

      “This study complements other sources reporting ethical issues at COPE.”

      I received however from a friendly student an e-mail from Frits Rosendaal, one of the members of the council of COPE. Frits Rosendaal is a full professor at Leiden University / LUMC and chair of the Committee of Scientific Integrity at LUMC. Frits Rosendaal is also a fellow of KNAW ( https://www.knaw.nl/en/members/members/7961?set_language=en ). Frits Rosendaal is Editor-in-Chief of http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1538-7836 and a visiting professor at Università degli Studi di Milano ( http://www.unimi.it/ ). The journal in which I have published my paper is published by this university.

      This e-mail from Frits Rosendaal and dated 7 November 2017 contains non-scientific information information about myself and about my efforts to retract the fraudulent study on the breeding biology of the Basra Reed Warbler (my paper contains details about these efforts and about the links with COPE, the report at https://www.academia.edu/33827046 does not contain any details about COPE). This e-mail from Frits Rosendaal also contained a post-publication peer-review of my paper. This post-publication peer-review was very short:

      ” (…) bizarre claims, soms gepubliceerd in obscure blaadjes. Zie bijvoorbeeld https://riviste.unimi.it/index.php/roars/issue/view/1077 “.

      So Frits Rosendaal states that this journal is an “obscuur blaadje” ( = an obscure / doubtful magazine) and states that my paper contains “bizarre claims” ( = bizarre / grotesque claims). The e-mail of Frits Rosendaal does not contain details in which these statements are substantiated.

      So you are correct to state that “Publication ethics is fascinating area.”

      This post-publication peer-review of my paper by Frits Rosendaal implies towards my opinion as well that Frits Rosendaal, and likely also his allies at COPE, are unable to rebut / refute any of the findings / conclusions of my paper.

      I am looking forward to comments and rebuttals on this conclusion.

    • Jonathan (another one),

      it is stated in the section Acknowledgments of my paper at
      https://riviste.unimi.it/index.php/roars/article/view/9073 that “Legal representatives from various stakeholders, among them BioMed Central, COPE, TF,and the University of Pisa, have contacted the author in relation to the efforts to retract the fatally flawed study on the Basra Reed Warbler”.

      I send these legal representatives always a response in which I acknowledge the receipt of their letter / e-mail and I ask them in my response always the same 2 questions:
      (1): please provide me acccess to the full set of raw research data of the study on the breeding biology of the Basra Reed Warbler;
      (2): please provide me comments from a few experts / peers / reviewers who refute / rebut any of the findings of the report “Final investigation on serious allegations of fabricated and/or falsified data in Al-Sheikhly et al. (2013, 2015)” (at https://www.academia.edu/33827046 ).

      The responses on both questions are always similar:
      (1) no feedback on my query to get access to the raw research data, and
      (2) no feedback on my query for comments from experts / peers / reviewers who refute / rebut any of the findings of the report. The responses on the last question are mentioned in my paper.

  2. This might be the place to relate a story from long ago, when I took a course from the late Robley Evans, one of the finest teachers I’ve studied under. We were learning about detecting isotopes using mass spectrometers, which can distinguish charged atoms of different masses by deflecting them with a magnetic field. Prof. Evans told us about a time in the 1930s, when many new isotopes were being found because of improvements in mass spectrometers, that a particular researcher who had established his own mass spec. lab published a paper stating that he had looked for a particular isotope of lead (I forget which one), and it didn’t exist.

    Soon after, another researcher published a paper in which he had indeed detected the same isotope. This paper was considered to be reliable. Next, the first researcher reported that yes, he had also found that same isotope, and had gotten its values with ten times the precision of the previous report.

    Hmm. Dr Evans would never tell us the first researcher’s name. He did tell us that no one in the field paid any attention to later papers published by the first guy (I think he was already regarded skeptically in the field). Everybody who mattered knew his work wasn’t to be relied upon, and they didn’t have to tear him down publicly.

    Just a different take on the subject, a different kind pf post-publication review.

  3. This is also perhaps a good example of varying norms for what evidence is sufficient to introduce a new statistical test and encourage its widespread use.

    Note that Simonsohn’s paper does not include any theoretical results about the test or really any sketch of a proof. Rather, it offers heuristic arguments in favor of the test and some evidence from simulations, where it performs better than some other tests.

    Of course, there can be errors in proofs or the assumptions can be too strong and unrealistic (perhaps here assuming homoscedastic errors would be enough to get a proof, though actually I don’t know about that since if the true DGP is really U-shaped, then this V-shaped model with have heteroscedastic errors). So my point certainly isn’t to say that such simulation-based and other empirical evidence about methods has no place.

Leave a Reply to Martha (Smith) Cancel reply

Your email address will not be published. Required fields are marked *