Skip to content
 

Researchers.one: A souped-up Arxiv with pre- and post-publication review

Harry Crane writes:

I’m writing to call your attention to a new peer review and publication platform, called RESEARCHERS.ONE, that I have recently launched with Ryan Martin. The platform can be found at https://www.researchers.one.

Given past discussions I’ve seen on your website, I think this new platform might interest you and your readers. We’d also be interested to hear your thoughts about the platform, and would encourage you to submit any of your work that you think could benefit from this new outlet.

Some further information is included below for your reference.

First, the platform is meant to be entirely open, to all researchers, in all fields. The platform aims to realize the benefits of peer review without suffering its drawbacks by (a) making all communications non-anonymous and (b) putting all publication decisions (including peer review) in the hands of the authors. There is no editorial board or accepting/rejecting of papers.

Among other benefits, we believe RESEARCHERS.ONE will enhance transparency and remove publication bias.

There are a number of other aspects to the platform, including pre-publication public peer review and a commenting feature to allow for post-publication discussion and peer review.

We’ve written about the details of the platform in our mission statement here https://www.researchers.one/article/2018-07-1

And we’ve addressed some basic questions in these videos: https://www.youtube.com/watch?v=ZFAerOjIMGM and https://www.youtube.com/watch?v=JD3kd7duAdQ

They describe their new system:

There are no editors, there are no accept-reject decisions or other barriers to publication, and there are no barriers to access. In removing these barriers, the researchers.one au- tonomous publishing model gives authors total control over the publication process from start to finish, which includes selecting the mode of peer review (public access or traditional), choosing the invited reviewers (if traditional peer review is chosen), determining whether and how to address reviewer comments, and deciding whether or not to publish their work. Once published, articles are publicly accessible on the site, with options for other users to provide non-anonymous commentary and post-publication peer review. Ultimately, the quality of published work must stand on its own, without the crutch of impact factors, journal prestige, ‘likes’, ‘thumbs up’, or the artificial stamp of approval signaled by the label ‘peer review’.

Here are some of my many, many posts on related topics:

Post-publication peer review: How it (sometimes) really works

When does peer review make no damn sense?

An efficiency argument for post-publication review

When do we want evidence-based change? Not “after peer review”

Crane and Martin’s system, a kind of super-Arxiv with pre- and post-publication review, seems like a great idea to me. The challenge will be getting people to go to the trouble of submitting their manuscripts to it. As it is, I can’t even get around to submit things to Arxiv most of the time; it just seems like too much trouble. But if people get in the habit of submitting to Researchers.One, maybe it could catch on.

A cautionary tale

I remember, close to 20 years ago, an economist friend of mine was despairing of the inefficiencies of the traditional system of review, and he decided to do something about it: He created his own system of journals. They were all online (a relatively new thing at the time), with an innovative transactional system of reviewing (as I recall, every time you submitted an article you were implicitly agreeing to review three articles by others) and a multi-tier acceptance system, so that very few papers got rejected; instead they were just binned into four quality levels. And all the papers were open-access or something like that.

The system was pretty cool, but for some reason it didn’t catch on—I guess that, like many such systems, it relied a lot on continuing volunteer efforts of its founder, and perhaps he just got tired of running an online publishing empire, and the whole thing kinda fell apart. The journals lost all their innovative aspects and became just one more set of social science publishing outlets. My friend ended up selling his group of journals to a traditional for-profit company, they were no longer free, etc. It was like the whole thing never happened.

A noble experiment, but not self-sustaining. Which was too bad, given that he’d put so much effort into building a self-sustaining structure.

Perhaps one lesson from my friend’s unfortunate experience is that it’s not enough to build a structure; you also need to build a community.

Another lesson is that maybe it can help to lean on some existing institution. This guy built up his whole online publishing company from scratch, which was kinda cool, but then when he no longer felt like running it, it dissolved. Maybe would’ve been better to team up with an economics society, or with some university, governmental body, or public-interest organization.

Anyway, I wish Crane and Martin well in their endeavor. I’ll have to see if it makes sense for us to post our own manuscripts there.

53 Comments

  1. Harry Crane is an excellent debater, I must say.

  2. Harry Crane says:

    Thanks, Andrew. Ultimately, it is the hope that RESEARCHERS.ONE will develop into an open and transparent community of likeminded researchers. To develop that community it will be helpful in these early stages to have support and engagement from those who support the initiative.

    We’re glad Andrew thinks this is a “great idea”, and we hope you’ll support the cause by posting future papers to the platform.

  3. Ryan Martin says:

    Thanks, Andrew — Harry and I appreciate the shout-out and well-wishes.

    RESEARCHERS.ONE is sufficiently flexible that it can play the role of a “souped-up arXiv”, a traditional journal, or something else. The author is free to decide how to most effectively use its features. Currently, posting to arXiv might be “too much trouble” because authors still submit their paper to a journal. But Harry and I hope that, in time, RESEARCHERS.ONE will be the only place that authors choose to submit and publish their work.

  4. Overall, I’m totally behind open access reviewing and publication. But I’m a bit confused by the message here from the site and from Andrew.

    1. Isn’t arXiv* a pre-publication site? I thought the drill was to put things on arXiv, then submit them. RESEARCHERS.ONE** doesn’t seem like a souped-up arXiv in spirit.

    2. Why does the description conclude with “the artificial stamp of approval signaled by the label ‘peer review’.”? Earlier in the description, it said, “The platform aims to realize the benefits of peer review.” It also said, “There are a number of other aspects to the platform, including … peer review.” Which is it, an artificial stamp of approval or something worth pursuing? I’m actually confused about this point, not just trying to nitpick. My guess is that peer reviewing in very selective venues is what’s driving CVs these days, at least in computer science, where it’s de riguer to list the acceptance rate of every publication venue. I’m looking forward to the first hack, where journals get spammed with bad articles to up their selectivity. I guess it’s already happening organically in machine learning.

    3. I wouldn’t read too much into a 20 year old N = 1 cautionary tale. It’s like story time without the statistics! Most ventures based on good ideas don’t succeed due to bad execution or bad timing. It doesn’t say much about the idea, especially if we’re talking 20 years out. There’s no reason to sell out to a closed-access journal other than to make a buck—open source is sustainable and very inexpensive as evidenced by Transactions of the Association for Comptuational Linguistics and Computational Linguistics and Journal of Machine Learning Research, all of which operate with no submission fees and are completely open access (not the reviews or reviewers, though). TACL even operates like Bioinformatics with roughly 4 week turnaround—submitting there is a joy compared to other journals I’ve dealt with.

    4. I’m curious about Andrew’s concluding remark, “I’ll have to see if it makes sense for us to post our own manuscripts there.” What criteria are you going to use, Andrew? If you start submitting to open access journals rather than journals you think are “good” (I’ve never been sure what you mean by that, since you insist it’s not the prestige), I’ll stop opting out*** of co-authorship.


    * Andrew intentionally miscapitalizes words that are pronounced, hence his use of “Arxiv”, which looks wrong, but is intentional. I’m not sure if he does it for the pleasant typography (in which case, I’d suggest dropping the caps while he’s at it) or to annoy the people who originally named something.

    ** I’m curious how Andrew will spell “RESEARCHERS.ONE” if it becomes popular. Even I don’t like the all caps for two actual English words there.

    *** I have opted out of the journal game for anything closed or pay-to-open-access. I won’t submit or review. I try very hard not to cite closed sources (non-open access books, even Andrew’s) if there’s an even halfway decent free alternative. I’d urge everyone else to do the same, because journals can’t operate without free reviewing (or maybe they wouldn’t be hugely profitable if they paid us for our work). And here’s the pro tip—nothing bad happens if you say “no” to reviewing requests. Unfortunately, it’s the usual prisoner’s dilemma situation—me opting out is just less competition for prestigous publications for others, and I don’t think hiring or tenure and promotion committees share my concerns about open access.

    • My impression is the anonymous pre publication gatekeeping peer review is the type being called a stamp here. Technically in this traditional case there’s no way to verify that it even happened. Editors can ignore it with impunity etc

    • Andrew says:

      Bob:

      Thanks for the comments. In reply:

      1. To me, Arxiv is a publication site. To post something on Arxiv is a way to publish it.

      2. Peer review to improve the work is an unambiguous good. Peer review to give the stamp of approval, that’s the part is more controversial.

      3. I agree that my story is just an anecdote. The relevant part, I think, is not that open source failed, but that you need a team to do this; it didn’t work to have one guy try to set up a system with the hope that it would fly indefinitely on its own.

      4. I’d submit a bunch of papers to this new Researchers One site right now; the issue is that it takes effort to submit a paper. Even submitting to Arxiv takes a few minutes of unpleasant paperwork. This is time that I’d rather spend responding to blog comments.

      5. I just think it’s weird to call something arXiv. Arxiv looks more sensible as a name. I feel the same way about E. E. Cummings.

      6. You say, “nothing bad happens if you say ‘no’ to reviewing requests.” That’s not quite right. The (somewhat) bad thing that happens if you say No is that the journal editor has to find a new reviewer. By saying Yes, I’m helping out the journal editor. So for 10 minutes of work, I can help out the editor, and the author of the paper too (both by giving constructive comments and by expediting the review process). I recognize that by doing my reviews I’m contributing to a system that has major problems, and I do feel bad about that—but, in the immediate term, reviewing a paper is a way of helping people out.

      • “nothing bad happens if you say ‘no’ to reviewing requests”

        Andrew, I’m pretty sure Bob meant this in the sense of nothing bad happens *to him* there are no reviewer taxmen who come and extract a pound of his flesh or whatever.

        • Andrew says:

          Daniel:

          It could be that something bad happens to me when I say No, in that the journal editor can be annoyed that I’m not more cooperative. But then I’m damned if I do, damned if I don’t, as there’s no way I can avoid saying No to most of the review requests I get.

          • It “could be” but the point of Bob’s post is that in Bob’s experience although it could be, nothing bad in fact does happen. I think especially for early career PIs, the “could be” is a real fear and hearing “nope nothing bad happens” alleviates that fear. Apparently this works for you too, you say no to lots of things, and then … you seem ok. Though this might be attributable to your already high standing… so it doesn’t alleviate all of the concern.

            • Right—nobody’s even been annoyed at me for saying no. But then my friends are decent enough to edit and ask me to review for open-access journals :-).

              What I’m advocating is not reviewing for or submitting papers to proprietary journals. I don’t even feel bad not reviewing for them given that I’m not submitting to them.

      • Harry Crane says:

        Regarding #6: I believe something very bad does happen when you say ‘yes’ to review requests. By helping out the editor, you are directly contributing to the flawed system, as Andrew notes. Though no individual editor is personally responsible for this system, he/she does help to perpetuating it by assuming the position as editor. The career pressures that exist — being editor of a journal helps add a “line on the CV” and therefore helps with promotion, etc. — pressures people to accept these editorial positions, for their own benefit but to the detriment of scholarship as a whole.

        I’ve had awkward situations in which friends and colleagues have asked for an “informal opinion” about a manuscript. I’ve had to refuse. I’m not in the business of giving opinions to people who will use that opinion to accept and reject papers. I’d much rather provide that feedback privately and non-anonymously to the authors.

        • Keith O'Rourke says:

          A good metaphor for the challenge here might be a community of boatmen in the middle of the sea who realize the need for repairs but requests for repairs can be below the current water level for many and often it is unknown.

          For me personally my boat has most been in dry dock – but for those who just starting getting into deep water its hard not to help them not sink. Maybe at some point – we have to let that happen.

        • Indeed, my wife has said that she will have a hard time getting a grant for a particular bit of research related to using Agent Based Models to model bone healing because she doesn’t have someone on the grant with the appropriate CV. Apparently I don’t qualify even though I’ve built and published agent based models, because I’m not a professional academic with an academic CV of the type expected… (I do consulting work I’m not a PI). Instead she needs to get someone to be on the grant who has some kind of applied math academic CV even though that person has no idea about agent based models because … otherwise no funding.

          That line-on-the-CV stuff *really does make or break funding decisions* and so it’s got some kind of real world monetary exchange rate. It seems like the rate is something like $1M / yr / 20 lines in biology.

          Anyway, screw that. I think your whole system sounds great, and I’ll definitely consider publishing my paper on a logical foundation for Bayesian inference without Cox’s theorem and boolean true-false logical statements there if it takes off (the paper is nowhere near ready so I’ve got time to watch your scheme fledge). I really do hope you succeed. I especially like that it’s all-inclusive.

          • This isn’t surprising. The expected output of a grant is research in a particular area. Research is measured by publications. The best evidence of being able to make future publications at a given perceived quality level is to have made many such publications in the past.

            This adds a strong bias against junior people. I saw this on review panels where what I thought were lousy proposals by well known senior people were selected over the panels’ voting for what seemed like better proposals by junior people. I figured if they were just going to fund based on name, they didn’t need me to bust my ass reviewing these things for a week. I had the same experience applying to NSF, where program managers overrode their panels rating to not fund my proposals (they weren’t following Chomsky’s latest theories, so weren’t considered linguistics by the PM).

            The NIH officially recognizes the bias against younger applicants and says they’re trying to address it so less money gets concentrated on older PIs. Nevertheless, they continue to fund large centers which work in the opposite direction.

            Luckily for me, Andrew has enough publications for everyone, so I don’t need to worry about having publications to cite as output from our previous grants. It also helps that other people write things about Stan. In fact, that helps a lot. And also hurts, because Stan looks too much like engineering and not enough like research.

          • Clyde Schechter says:

            “Apparently I don’t qualify even though I’ve built and published agent based models, because I’m not a professional academic with an academic CV of the type expected… (I do consulting work I’m not a PI). Instead she needs to get someone to be on the grant who has some kind of applied math academic CV even though that person has no idea about agent based models because … otherwise no funding.”

            Has this been your wife’s actual experience, or just what she fears will happen. While it is certainly true in my area that the overwhelming majority of people who get funded have the standard academic CV, it is not true that a strong CV will get you funded even if it doesn’t demonstrate expertise in whatever you are slotted to do in the proposal. NIH study sections frequently shoot down proposals when a “big name” is listed on a proposal to do work that he or she doesn’t have a track record in.

            And I have certainly seen successful grants where non-academics have been funded to do work where they do have a track record of publication in the area.

            (I know this post is tangential to the thread, but…)

            • I think Bob’s comment gives insight into the concerns.

              There is plenty of erecting barriers to entry to protect funding to go around. “Grantsmanship” these days seems to be largely a game of chess in which you set up a series of multiple defenses against all the different kinds of attacks you can anticipate, and hope that these defenses allow your knight to stand just on the other side of the funding line.

              • Harry Crane says:

                To all those discussing grants on this thread, there are parallel initiatives that separate from, but working in close collaboration with, researchers.one that aims to address some of these funding concerns.

                What R.1 is trying to do for peer review/publication, there are projects trying to do the same thing for funding science. The combo of R.1 with these new grant funding projects could/would be a game-changer.

                There are reasons to be optimistic, but we’re at the very early stages. If you like the idea behind R.1, the best way to support would be to put your work on there. Building a community starts with 1 or 2 passionate people as Bob said.

                Thanks for all your comments and support. If you have specific suggestions for improvements and enhancements, please get in touch. We’re taking everything into account and trying our best to incorporate as much as we can. contact@researchers.one

  5. Harry Crane says:

    Responding to Bob:

    1. The key thing to understand about R.1 is that it is flexible in how researchers use it. It can serve as a kind of “souped-up arXiv with pre- and post-publication review” or it can serve the same purpose as a journal, in that it can be where you do peer review and final publication.

    It’s key feature is separation between what might be understood as “access” and “credibility” layers of the current publication process. We believe that peer review should be accessible to all, for the purpose of improving research quality, not for the purpose of conferring credibility, prestige, etc. The latter contributes to a number of problems, including publish-or-perish and even some of the issues with the replication crisis. R.1 offers a platform for those who care about quality scholarship, without wanting to subject themselves to the trouble of the journal peer review process.

    2. This was partially addressed in paragraph above. The “benefits of peer review” are the scholarly benefits of peer review. For example, it is not uncommon for me (and others) to email friends/colleagues my drafts before I submit to journals*. This provides helpful feedback (indeed, peer review) that improves my work. The peer review that goes on at journals serves this purpose only tangentially to its primary objective of giving the editorial board grounds to accept/reject the paper, and thus to signal the paper’s potential impact, prestige, etc. The latter is not necessary, and I believe is detrimental, to quality scholarship. I hope this clears up your confusion.

    3. The cautionary tale is worth bearing in mind, and worth considering how R.1 can avoid this fate. There are things working in our favor. In particular, we do not rely on the labor of any single person to keep the platform running. The platform is up and running. It is a feature of the site that we have no editorial board. We hope to build a community around these core principles, and that community would fuel interactions on the platform. But even without that, the maintenance and upkeep on the site requires a relatively small amount of funding (Ryan Martin and I have seeded the site with our funds for the time being).

    4. I think someone like Andrew may be in the best position to use the site at present. Given his presence, he would have no trouble to solicit reviews and feedback, to which he could reply as appropriate. The openness of the platform is also amenable to non-conventional and potentially controversial opinions that might have a hard time gaining acceptance in mainstream journals.

    * Except when my co-authors insist, I no longer submit papers to journals. I plan to conduct all peer review and publish my work on R.1 from now on.

    • Thanks for the thoughtful response to both (2) and (4). I really like having people give me comments on things, because it’s usually constructive. Peer reviews are usually not, or at least not in the sense that are helpful (e.g., add another graph, run another experiment, etc.—we can always make things better, but it’s a matter of judgement as to which is worth doing—rarely do you see a reviewer suggesting that something be removed).

      I still don’t make much of the cautionary tale as a cautionary tale. It’s not that it was started by one person—almost everything starts with one or two passionate people.

      Building community is obviously critical as something like this can’t work with just a few people. We’ve been very careful with Stan to try to build and maintain a helpful, constructive online and in-person community. Partly that’s because I prefer that kind of community and partly out of interest in the project continuing long term.

      I also try to avoid papers. I publish most of my work in the Stan manual or in Stan case studies or in grant proposals.

      • Bob I’d like to once again reiterate that the Stan manual, case studies, and some ArXive papers out of the Stan group (Betancourt’s HMC paper especially) is probably the single most important set of publications in modern applied statistics anywhere. You guys do a tremendous job of making real in-depth useful applied statistics stuff very very accessible and understandable. And I believe that a big part of that is *because* you avoid the “novel, transformative, intensely compressed, peer reviewed, Nature paper” type format. That’s all about chits and not at all about quality.

  6. Dikran Marsupial says:

    “choosing the invited reviewers (if traditional peer review is chosen)” I don’t think that is a good idea https://www.sciencedirect.com/science/article/pii/S0921818117306586

    • Ryan Martin says:

      Traditional journals rely on reviewers for gatekeeping, so there would be a huge conflict of interest if authors could select their reviewers. However, on R.1, the reviewers are not gatekeepers and their opinions about the paper are not used to make accept/reject decisions. Peer review on R.1 is strictly for the authors benefit, so if they choose to only solicit feedback from friends, then that’s a missed opportunity for them.

      And if the post-publication peer review reveals that an author published low-quality work on R.1, then he/she must take sole responsibility. For a traditional journal, like in the case you mentioned, since the journal accepts the role of gatekeeping, if low-quality work gets through, then the journal must also share some of what should be entirely the authors’ responsibility.

      R.1 gives authors the freedom to control the peer review process but also the sole responsibility for doing good work. The traditional journal’s role as middleman in this process isn’t helpful; arguably, it’s detrimental.

  7. Harry Crane says:

    Replying to Dikran Marsupial:

    It’s important to understand how the platform works, and in particular how it differs from what we’re used to, before projecting problems in the current model onto it.

    The problems with authors choosing reviewers in the standard publication model are understood. But those concerns aren’t relevant in the researchers.one model because the referee reports play no role in any accept/reject decision. And, unlike the traditional system in which the referee process is shrouded in mystery, this aspect of peer review on R.1 is known up front by all readers and authors on the site. Thus, it can and should be taken into account when engaging with literature published on researchers.one.

    • Let’s just take this a little more explicitly in this direction. On R.1 everyone knows that the reviewers are chosen by the author. So if the reviews are all happy talk and good-buddy stuff, you can ignore them, and engage the paper itself, which you may find not worth its salt, and then ignore. But at least it’s available, and not in the “file drawer” so to speak, inaccessible to anyone. If the reviews give useful real feedback, then they add to the content, and they are also attributed to the reviewer, so the reviewer can contribute to science without having their contribution file-drawered (as they are in the traditional model, where no-one sees the reviews except the author and editor pre-publication).

      Nothing about this whole scheme is designed to put a stamp of approval or a barrier to entry up, and that seems like a good thing. But, there may well be plenty of junk due to this.

      To the extent that R.1 makes this work, it will require some kind of way for readers to put up successful “barriers to attention”, because few people are planning to spend their whole life reading papers and filtering them into quality bins (ie. playing the role of Professional Movie Reviewer). I think this is where Harry and his crew need to spend some effort: coming up with a way to filter by topic, author, reviewer identity, quantity of reviews, reader ratings, whatever. To the extent that the “Movie Reviewer” role of someone like Roger Ebert is a worthwhile type of role, the non-anonymous nature of reviews is excellent, and a way to leverage a reputation as a good reviewer is needed.

      • Harry Crane says:

        Our goal is to let the use of the platform evolve as organically as possible, with minimal interference by us. Ryan and I have created a platform that we hope will provide the means necessary for publishing and peer review, we do not want to bias toward articles based on arbitrary metrics (which could and would devolve into similar problems we now face with emphasis on journal impact factor, etc.).

        To the extent that we can achieve the functions you suggest without bad side effects, we’d like to hear your input.

        One thing to realize is that we have built a more organic recommendation mechanism into the site through the “Recommended Content” feature on each user’s profile. For example, you can see this on my profile https://www.researchers.one/author/res47. Note that recommendations are not limited to R.1 content.

        The idea here is that you will not need to comb through every paper yourself in order to find something to read. Everyone has colleagues whose opinions they respect. If those colleagues come across papers that they think are worth reading, they can recommend them on their profile. So when you want to find new papers to read, you could go to the profile of people you trust and take their recommendation.

        This fine grains the same exact system that exists with journal filtering. If you read papers because a specific journal published it, then you are taking the recommendation of a relatively narrow editorial board. This feature gives everyone the chance to recommend content. Like journals, not everyone’s recommendations will have the same weight, simply because fewer or more people will consult the recommendations of different users.

        • I agree with the idea of not using gameable metrics, like citations, and I agree with the idea of using recommendations from people you trust. But I think that requires a bunch of engagement from authors: they need to keep their recommendations up to date. And that’s problematic, because it costs time and effort.

          In the end, a system for dumping huge quantities of research into a pile will wind up as that. For example ArXiv has lots of content. Suppose today I want to go read the best stuff on Hamiltonian Monte Carlo on ArXiv. How do I do it? If I search for “hamiltonian monte carlo” I get 1181 articles as of this moment. I have time to read say 10 and I am interested in articles specifically about applications to Bayesian statistics rather than say calculation of materials properties in solid state physics… what do I do?

          One thing I think would be helpful is the ability to create recommendation lists that stand on their own. Suppose I’m interested in a topic, say HMC, and I read a bunch about it. Now, I have a list of articles that I think are worthwhile and some opinions about what is worthwhile about them. So I should be able to publish that list, and hopefully bring some publicity to papers both *on* and *off* R.1

          It’s much more likely that a person who is reading up on a topic will organize their own thoughts into a topical list with commentary than that a person who is going about their research day will think to update their big list of recommended papers across all topics on a regular basis.

          • I have time to read say 10 and I am interested in articles specifically about applications to Bayesian statistics rather than say calculation of materials properties in solid state physics… what do I do?

            That’s easy. Read whatever Michal Betancourt writes. And if you have time, follow some of the citations.

            • Sure, but “just ask Bob Carpenter what he’d recommend” unfortunately doesn’t scale past HMC/Stan stuff and some linguistics topics that I don’t actually have much background to be interested in.

              Who would you suggest for models of unstable granular materials, or micro-physics of friction and its role in earthquake initiation, or economic models of couples’ labor market choices in the face of taxation and child care costs, or effective anti-poverty policy, or agent based models of cell-cell interaction during organogenesis, or fluid mechanics of sailboats?

      • I’ve never understood this filtering argument. I find nearly all of the papers I read either through word of mouth or through references in papers I’m currently reading. Many of those wind up being pre-publications or even pre-arXiv and many are just blog posts.

        Do other people read the tables of contents of prestigous journals for reading?

        • Yes.

          I do get a lot of papers from word of mouth, citations, and specific searches. But I also read the tables of contents of about 7 journals, both “prestigious” and more subject-specific, every week. If I didn’t do this, I would have a poorer grasp of things not immediately tied to topics I work on, and of (possible) connections between things I work on and the large variety of other topics out there. Of course, I skim these tables of contents pretty quickly, but still it takes a good amount of time. I probably should be spending *more* time on this, rather than less.

          This reminds me that I often get the sense that people who don’t “get” the filtering role of journals seem to work in small fields (small by nature or small by personal construction), in which one can know all the people worth keeping up with, and know all the papers worth looking at. For most of us, this isn’t the case — there’s a torrent of stuff constantly coming out, that we must somehow grapple with.

        • Rahul says:

          I use the filtering function often like this. Say, I see a great yield reported for a chemical reaction in an Unknown-Journal I’m skeptical. If I read that in Good-Reputation-Journal I’m more likely to believe it.

          So the Journal reputation serves as my prior about the reliability of a reported number.

        • I don’t read the TOCs because I don’t really trust journals to align their idea of important and interesting with my idea, which is why I really want a system to *carve out my own slice* of the available articles. Sure I can do lots of keyword search, but for example if I find an interesting article can I find out who cites it? Can I limit the list of who cites it to papers with certain keywords? Can I give known authors higher weight in my search (Let’s say Bob Carpenter is a 10 and random joe is a 1 and Andrew Gelman is a 20…) so now show me all the papers that cite any of the papers in some given set and ordered by descending sum of author score with keywords “Bayesian Decision Theory” or “Hamiltonian Monte Carlo”…

          Let’s also make it easy for me to quickly add author scores to my database of authors, so that while I’m searching above, I find a paper, and I recognize that the author has made a good contribution, I can easily click and add that author with rating say 15 to my personal database of author scores, and my search will re-sort.

          Furthermore, give me the option to propagate trust through the network of citations. If a paper is cited by a trusted author add 1/2 the trusted author’s score to that paper, and then everything that paper cites, add 1/4 the author score to those… do this for several levels of indirection, and then sort those papers…

          Basically what it comes down to is, I *don’t* want journals telling me what’s important, but that doesn’t mean I don’t want to impose some personal ranking and prioritization onto the enormous body of work out there, and the more tools I have at my disposal to personalize my search, the better I can align the numerical score with my actual level of interest.

          • Daniel: I like the recommendation score idea. It isn’t perfect, but it would be great to try! I’m often amazed at how good a job Netflix does at predicting what movies I will like, or YouTube for recommending music, and we haven’t really adopted these approaches (at least not widely) in academia.

            • Yes, but I’d also add that I HATE searching for stuff on Netflix because beyond the first page of recommendations, they are completely useless search wise. There is *no* ability to tune your search to your current *conditional* interest. Like for example, good luck to you if you want:

              “Find me a family comedy either animated or live action suitable for kids age 6+ and adults, not made by Disney, not a sequel, increase the ranking if it stars an actor I like, increase the ranking if it involves nature or animals, made after the year 2000, include titles not currently available so I can add them to my watch list”

              Good luck. Netflix tries to decide what you are interested in, in that sense, although they personalize to you, they are still the “traditional journal model”. Instead I want comprehensive methods to build personal scoring systems for myself so that I can tune the scores to the current conditional interests.

              • That’s one of my biggest beefs with not just Netflix, but other search engines. I used to really like the Excite search engine, because it used TF/IDF, so I could dump in a really big search with all sorts of associations and it’d find related stuff and drill down into the topic I cared about. Then along came Google with pure boolean search and page ranking—it’s more like what Netflix gives you.

                Diversity of search results and downrating things that have been recommended before would also help. Google got a lot better in search results diversity over the past decade. It used to be a joke that to find the Michael Jordan in machine learning you had to scroll past a thousand pages of hits about the basketball player.

                Maybe all this chatter about human-centerer AI and machine-assisted search will come to fruition.

        • Andrew says:

          Bob:

          I don’t know exactly how this works. I guess it depends on the field. When I hear about research that’s far from my field of expertise, it’s typically because someone emails me about it, and in turn they have heard about it through a media report, which in turn often happens with papers that appear in tabloid journals (Science, Nature, PNAS) or other journals that are connected to powerful publicity machines. Not always—sometimes a paper in Plos-One or Arxiv can get lots of attention too—but I think that publishing in a tabloid is more likely to lead to publicity and attention and readership. Whether that works with other journals, I don’t know. The #1 journal in political science is APSR, and I almost never see APSR papers getting publicity in the news media. Within political science, though, I’d guess it’s more likely your paper gets noticed if it appears in the APSR.

          • Nobody emails me about articles they see in traditional news coverage other than the occasional relative. And I never get people mailing about random topics, either. The occasional relative mailed me when speech recognition was first taking off.

            I was more talking about where you find technical papers to read. That’s the audience I want to reach.

        • AllanC says:

          But think about earlier in the chain in your social network. How do the people you trust or papers you read (and like/trust) know what other papers to read/cite? They would hear about it from someone they trust or a paper they read, or happened to stumble upon it by accident. And so on and so forth down the chain.

          There is nothing wrong with leveraging your social network for recommendations. I do this extensively and I am often very happy. Sometimes I think those I trusted were high on glue, but hey, it’s a good ratio of hits to misses!

          But realistically there is probably a lot more important, and interesting work on the topics you want to investigate beyond what your network has been exposed to. In retrieval terminology I suppose there is high precision, and low recall. If you’re going to build a new opportunity for article dissemination then it makes sense to try and tackle the low recall aspect; especially if your goal is to publish everything. It’s just too important to leave it to unexplained characteristics as to why some papers takeoff within your circle and others do not.

  8. Robert says:

    I can’t watch the video right now, so maybe this was addressed. But how does R.1 plan on avoiding the “vixra problem”? The arxiv has broadly good content, or at least similar quality to that of published journals. Some researchers post silly things that step outside their field and clearly wouldn’t get published, but quality is quite high considering there’s no review whatsoever.

    Now you make the single change of not requiring *any* filter on who can publish papers, and you get vixra. Which is almost entirely garbage. Any quality papers there will never be seen, because filtering through the crap is too much effort. In principle one can comment on vixra papers, but the signal-to-noise is too low for this to actually be worthwhile. Even if all of the thousands of papers ‘proving the Riemann hypothesis’ had quality reviews disproving them in the comments, finding quality content wouldn’t be any easier, and the thorough reviewing of the sea of terrible papers wouldn’t incentivize anyone to upload there. If the literature is going to be evaluable within a human lifetime then work that’s deemed very likely to be terrible needs to be easy to skip, and searches need to be able to prioritize by how well-received a paper (or review) is.

    Part of the issue with vixra was that arxiv already existed and was/is reputable, so that the *only* people who would bother with vixra when it was young were people who couldn’t post on arxiv, and its reputation cemented from there. But R.1 also has to deal with this effect somehow.

    It looks like anyone can establish themselves as a publisher and reviewer on R.1, and all concerns about likely quality are supposed to be decided after the paper or review is added. Are there any plans to combat potential noise problems caused by large numbers of underqualified people trying to publish and review without any background knowledge? Hopefully the small submission fee deters the worst of it, but I’m not sure it would be enough. I have yet to see a forum of nontrivial size that tries to discuss academics without an ‘expertise filter’ (a thorough moderation team, at the very least) and accumulates quality content.

    • This is a good detailed explanation of the concern I brought up above.

      I agree with the idea of allowing lots of submissions, but I also strongly strongly agree with the need to address the issue as you put it: “If the literature is going to be evaluable within a human lifetime then work that’s deemed very likely to be terrible needs to be easy to skip, and searches need to be able to prioritize by how well-received a paper (or review) is.”

      Thought needs to go into how to accumulate information about the quality of a particular contribution, and how to allow people to sift through this *quality* information via appropriate filters and search tools.

      For example, I might like to build up a weighted average of quality ratings, weighting by my own prior for the reviewer quality. Each reviewer could give a certain numerical rating to papers they’ve read, such as a scale of 0-10, but then *I* get to decide how to aggregate those by assigning a 0-10 rating to individual raters and a prior weight to be assigned to people I don’t know, so that I can get recommendations based on my own trust level.

  9. Harry Crane says:

    Thanks for comments by both Robert and Daniel.

    As Robert notes, there is a small submission fee, which should deter some level of vixra. But an ‘expertise filter’ is something that we have no intention to ever provide. ‘Expertise filtering’ is essentially the current peer review/publication model. It has many problems of its own, and we are trying to combat those issues.

    I would also point out that published journals also have a high proportion of “vixra”. They are just dressed up as peer-reviewed, accepted articles, but many published papers are still not worth reading.

    I mentioned above one possible feature of the site, which for sure can be enhanced as the platform evolves, in which users recommend content. For sure, not everyone will be very active in updating their recommendations. But some will, and those who do will build up a reputation and influence as a result.

    Beyond that, I will say that we have been contacted by a number of people who are involved in complementary initiatives, and who would like to develop some system of “reputation scores” and the like to go along with platforms such as ours. We encourage any and all such initiatives to build such features on top of the platform. However, we are not intending to implement such things ourselves, because we do believe it is important that the entity providing access to publication should be separate from any assessment or evaluation of that material.

    I believe this is something very fundamental that the current system gets very wrong, and it leads to a number of big problems. As you’ve noted above, our proposal at R.1 raises some new questions and is presented with new challenges. This is also true, but these challenges do not appear insurmountable by any stretch. There are a number of ways to address them. Those who have suggestions, and those who wish to help out, can contact us any time.

    • I agree with not trying to create a universal reputation scoring system, but that’s not the same thing at all as a personalized searching system.

      Perhaps the most important thing you need is an API for extracting metadata so that third parties can create add ons

    • Anonymous says:

      “Beyond that, I will say that we have been contacted by a number of people who are involved in complementary initiatives, and who would like to develop some system of “reputation scores” and the like to go along with platforms such as ours”

      How would that work? I reason it is hard to objectively “score” something to be used for “reputation” (it’s not like timing or measuring something).

      I fear this “reputation scoring” will therefore probably be largely “subjective” and the same things that probably went wrong in the last decades will happen again. People who are viewed as “hip”, or whatever characteristic is relevant here, will amass great “reputation scores”, folks will manipulate that, groups of people will influence things on grounds that have nothing to do with science, etc.

      I was very glad to read you saying “Ultimately, the quality of published work must stand on its own, without the crutch of impact factors, journal prestige, ‘likes’, ‘thumbs up’, or the artificial stamp of approval signaled by the label ‘peer review’.”, but bow i am worried you are already thinking about possibly “joining the dark side”…

      I fear reputation systems, and perhaps even also the “recommended articles” on your site, will only lead to a few folks influencing what everybody else believes and/or thinks is worthy of their attention (just look at twitter and facebook to see how that will probably work out).

      In my reasoning this is exactly what went wrong in the last few decades, where some topics seems to have gathered lots of attention. In combination with editorial/journal influence, publication bias, and questionable research practices, this has possibly resulted in decades worth of published nonsense and wasted resources.

      All these “reputation” things, and other possible “improvements”, to me are only reinforcing the exact problematic issues that have probably plagued science in the last decades…

      I reason if, and how, researchers use and cite papers is all the peer-review that is needed.

  10. Anonymous says:

    I still have trouble understanding why a reviewer would spend their time reviewing work on “Researcher.One”.

    In a recent discussion i wondered whether it would be beneficial for reviewers to earn possible co-authorship in case of a truly useful review (https://andrewgelman.com/2018/07/25/journals-refereeing-toward-new-equilibrium/#comment-809001). I reason if reviewers improved a paper, they should be credited from a purely scientific perspective.

    In the link above i described a system/format where reviewers could 1) possibly earn co-authorship (during a certain window/time period), and 2) in the process build a portfolio of reviews that they could possibly link to on their CV, etc. Both, in my reasoning, could be a reason for reviewers to join such a project.

    (Parts) of the format described in the link could be used by “Researcher.One”. For instance, i just checked the “profile page” of authors and commenters on the site, and it seems to me that only “papers under review” and “recommended content” are displayed there. I wonder if it would be useful to also list “uploaded reviews” on profile pages, so folks can link to their profile pages on their CV’s and others can easily find and look the review work they did (that would be reason 2: “building a review portfolio” of the format i linked to).

    (Side note 1: the paper linked to in this blog post about “Researcher.One” already received comments (reviews?), and the paper seems to have been updated based on those comments. I could not easily find the previous version of the paper however, and subsequently wondered if this is optimal. I am not a fan of constantly updated manuscripts, as i reason they make it hard and/or impossible to easily verify and compare citations, quotes, and comments/reviews. For instance, it seems to me to not really work if i were to cite or quote a piece of your paper that will later be deleted/re-written (?).

    Possibly compare this to the format i described where there is a window, of what has traditionally been called “pre-publication” review, where you can earn possible co-authorship. There would be a clear moment where the paper will be published, just like in traditional journals, which from that moment would not be changed, and could therefore be cited and quoted without any issues. After that, you could still have post-publication review but perhaps in the form of comments (also see next side note).

    (Side note 2: the comment section seems to be used for reviews, but i can also see a button that says “upload review”. I wonder if it’s confusing to have both options for reviews, also in relation to my comment above about possibly making sure reviews are displayed on the profile pages. Please also note that the format i described would fit perfectly with “uploaded reviews” (the “pre-publication review”), and with “comments” (the “post-publication review”), even if there is no real publication decision.

    • Ryan Martin says:

      Thanks for your feedback on the site!

      Re: why spend time reviewing for R.1? Reasons I can think of for reviewing for a traditional journal: I think the paper seems interesting; I feel some desire to serve the research community; or I feel like doing the editor a favor. The first two also apply for reviewing papers on R.1, only the third doesn’t. But why is an editor more deserving of a favor than an author?

      Re: profile page. At present, the profile pages on R.1 have three categories — Published Articles, Articles under Public Peer Review, and Recommended Content — it’s just that there are no published papers on the site yet. As part of our second phase, Harry and I are already planning to add a Comments/Reviews section as you suggested. I’m all for users getting credit for insightful reviews and comments, so this at the top of our list for phase 2.

      Re: co-authorship window. A nice feature of R.1 is that authors and reviewers can communicate directly. This creates an opportunity for reviewers who have helpful insights to potentially get involved in the research itself, either as a co-author of the paper under review or on some future developments. This, in my mind, was one of the main selling points of non-anonymous review. But as with everything on the site, we think it should be up to the authors to decide if and when a reviewer’s feedback warrants co-authorship.

      Re: Side note 1. We don’t expect that articles will be revised frequently. Harry and I had some special incentives to address the comments on our paper quickly that typical users wouldn’t share. So we don’t expect papers to get revised any more or less frequently than they do, say, on arXiv. And in the spirit of giving authors control, we have opted to leave the decision of whether previous versions of the article would be made available to the author. He/she is can choose to upload previous versions of the article as supplementary material if they like. But if a previous version has a stupid mistake that is corrected in a revision, then the author should be able to remove that.

      Re: Side note 2. Our intention was that “comments” would be for short comments/questions whereas “reviews” would be for more detailed feedback like referee reports. This is not clear on the site now, we’ll work on that.

      • Anonymous says:

        Thank you for your reply! Here are some additional thoughts should they be useful in some way or form.

        1) You wrote: “Re: why spend time reviewing for R.1? (…) than an author?”

        I am more and more baffled by traditional peer-review: why would i spend my time anonymously contributing to a paper and not receiving co-authorship if i substantially improve it, or receive any other credit whatsoever.

        Isn’t this also warping science in a major way: people who possibly wrote sub-par stuff get help from other people who fix things for them, only for the people who possibly wrote the sub-par stuff to get the rewards for the now decent or perhaps even excellent paper….This whole thing is just crazy to me!?

        Concerning your reply: doing the author a favor is just as unscientific as doing the editor a favor in my reasoning so i’ll leave that for what it is. As for your other 2 possible reasons: if i think the paper is interesting or i want to serve the research community i’d rather spend my time writing a paper myself (possibly building on/using the paper i found interesting). Like i wrote before: I reason if, and how, researchers use and cite papers is all the peer-review that is needed.

        If you want to use peer-review to improve papers, you should also reward that with co-authorship if someone truly improves it. Perhaps you could then also view a paper going through peer-review as going through some ultimate test: as an author you are saying here is my paper, see if you can spot mistakes and/or make it better. If can’t substantially improve it: perhaps my paper is decent enough to be cited/used/read in it’s current form. If you can substantially improve it: thank you for your contribution and you hereby deserve co-authorship.

        2) You wrote: “Re: profile page. At present, the profile pages (…)for phase 2.”

        This seems like a good idea. But please note, it seems highly problematic to me if the peer-reviewed version can be switched by the authors with a newer version without the possibility for readers to verify and check things in the older version. Without this possibility many problematic things can happen i reason.

        For instance, my review pointing to many (let’s assume correctly spotted) mistakes in the paper by tenured famous professor X, would make no sense when tenured famous professor X simply used my comments and corrected the mistakes and uploaded a new version without anyone being able to check the previous version with all the mistakes which i pointed out in my review. Next thing you know, tenured famous professor X is busting my b@lls in an editorial of his/her journal for writing a “nasty peer-review” accusing me of pointing out mistakes he/she obviously did not make (which nobody can now verify because there is only 1 new version available for the community to verify/check things).

        3) You wrote: “Re: co-authorship window. (…) warrants co-authorship.

        This could be a nice “selling point” for your project to further contemplate and/or possibly promote. Next to your option for listing peer-reviews on the profile pages, you would now have the 2 reasons i could come up with for reviewers to join your project/site (possibly see some additional thoughts on why i think that could be useful here https://andrewgelman.com/2018/07/25/journals-refereeing-toward-new-equilibrium/#comment-809001).

        Also note that possible co-authorship is something that (in my reasoning) can not easily be “gamed” or abused. For instance, if i have friends who want to help me out and reward my review on “Researcher.One” with co-authorship they could have just as well put me down as a co-author from the start.

  11. Dan F. says:

    There seems to be little evidence that peer review improves publication quality or helps facilitate distribution of ideas. To the extent that it does improve quality, the effect is likely swamped by the damage the process does by certifying garbage papers.

    In mathematics one generally reads the ArXived paper and cites the published version (often inaccessible behind a paywall). The process of publishing in a journal serves bureaucratic/administrative needs – the publication does not count for deans and funding agencies unless it is published in a (acredited = impact factored) journal – but the post to the ArXiv is publication in the traditional sense, does serve to assert priority (for those that care about such things), and does serve to distribute ideas (which is the goal, isn’t it?).

    One goal of post publication review seems to be to aid potential readers in the filtering process. In mathematics this is already partially achieved by MathReviews. Often one can infer from a well written review whether it is worth the bother to look at an article (although the reviews are not generally evaluative, when well written they summarize the contents in a way that aids in filtering). Note, however, that MathReviews is very professionally and carefully edited. For a post publication review process to be useful in filtering articles it would have to have the coverage (damn near everything) and quality of MathReviews. This seems impossible without a well paid and highly professional staff like the AMS has for MathReviews.

    It could be that the goal of post publication review is to help deans and funding agencies by eliminating the journal-impact factor mess, and replacing it with evaluation of individual articles. Ignoring the criticism that this is putting the horse (certification) before the cart (distribution of good ideas), it seems difficult to make it work without also guaranteeing coverage and quality of the reviewers. This is where the traditional peer review process is not broken – one trusts the expert editor to pick an expert reviewer – and while this does not always occur, nearly any alternative mechanism that does not count on the subjective use of the expertise of experts is likely to fare even worse.

    • Harry Crane says:

      R.1 is about harnessing the good parts of peer review without having to deal with the bad parts. If you want to skip peer review entirely, you can do that too. The decision is the authors.

      These are great points, especially the first — that peer review (the way it is done now) might do more harm than good. For me, I find real peer review valuable — the peer review I get when I ask people whose opinions I trust to read and comment on my paper. The traditional (“official”) peer review at journals is more of a rite of passage, and I generally have found myself arguing with editors as much as possible in order to avoid making the changes the referee requests, as many of these changes (add this simulation, etc) would make the paper worse in my opinion.

      Overall, I think giving the authors discretion over what goes in their own papers is the key. This cannot be fully achieved as long as articles are filtered through a hierarchical editorial system, no matter how benevolent those editors may be.

  12. Nat says:

    I think that the single biggest challenge to implementing a website like this is creating the community of users. I joined a similar website ScienceOpen a few months ago. I would imagine there are many websites offering their own twist on the publishing / peer review platform. I think the focus needs to be on how do you create a meaningful experience with a few hundred or a few thousand users across difference disciplines. If you can solve that problem, then maybe you can grow it.

    It is not clear to me that a single website is actually the best solution. Maybe in order to build the necessary community of users it might be better to start smaller with a specific domain focus similar to existing journals. Maybe these websites should be connected to existing professional organizations that also have a specialized focus.

Leave a Reply