Skip to content
 

Let’s get hysterical

Following up on our discussion of hysteresis in the scientific community, Nick Brown points us to this article from 2014, “Excellence by Nonsense: The Competition for Publications in Modern Science,” by Mathias Binswanger, who writes:

To ensure the efficient use of scarce funds, the government forces universities and professors, together with their academic staff, to permanently take part in artificially staged competitions. . . . how did this development occur? Why did successful and independent universities forget about their noble purpose of increasing knowledge and instead degenerated into “publication factories” and “project mills” which are only interested in their rankings?

Here we should distinguish between natural and artificial competitions. For example, if students get to choose what universities to attend and staff get to choose where to work, then universities will need to compete for both students and staff. But competition for government research grants, for example, could be considered as artificial in that an alternative would just be for the same amount of public funds to be distributed among universities according to some formula.

As Binswanger notes, what works for the top research universities might not make sense more generally:

How can you impress the research commissions responsible for the distribution of funds? This is mainly achieved by increasing measurable output such as publications, projects funded by third-party funds, and networks with other institutes and universities. In this way, “excellence” is demonstrated, in turn leading to easier access to further government research funds. Competitiveness has therefore become a priority for universities and their main goal is to perform as highly as possible in measurable indicators which play an important role in these artificially staged competitions.

One might say that there is no real alternative to this sort of competition—but fifty years ago, or maybe even thirty years ago, the above picture would not have reflected what was happening at many universities.

Binswanger continues:

Relevant publications are in professional journals, where submitted work is subjected to a “rigorous” and “objective” selection method: the so-called “peer-review process”. . . . However, among scientific journals strict hierarchies also exist which are supposed to represent the average “quality” of the accepted papers. In almost every scientific discipline there are a few awe-inspiring top-journals (A-journals), and then there are various groups of less highly respected journals (B- and C- journals), where it is easier to place an article, but where the publication does not have the same significance as an A-journal article. Publishing one’s work in an A-journal is therefore the most important and often also the only aim of modern scientists, thus allowing them to ascend to the “Champions’ League” of their discipline. Belonging to this illustrious club makes it easier to publish further articles in A-journals, to secure more research funds, to conduct even more expensive experiments, and, therefore, to become even more excellent. The “Taste for Science”, described by Merton (1973), which is based on intrinsic motivation and supposed to guide scientists was replaced by the extrinsically motivated “Taste for Publications.”

I’d like to stop here and issue a mild dissent. Yes, there is some extrinsic motivation to publish in top journals, a motivation which I don’t feel much right now but which was a big deal for my colleagues and myself when we were younger. Even now, though, I’d like to publish in top journals, not so much for the league standings or even to help out my younger colleagues, but because I feel that papers in such journals are more likely to be read and to make a difference. But I don’t really know how true that is anymore; it may just be habit that I retain a weak preference to publish in higher-ranked venues.

Binswanger continues:

At the end of the peer review process, the reviewers inform the editor in writing whether they plead for acceptance (very rare), revision, or rejection (most common) of the article submitted to the journal in question. Quite a few top journals pride themselves on high rejection rates, supposedly reflecting the high quality of these journals . . . For such journals the rejection rates amount to approximately 95%, which encourages the reviewers to reject manuscripts in almost all cases in order to defend this important “quality measure”. Solely manuscripts that find favor with their reviewers get published . . .

And thus:

The peer-review process is thus a kind of insider procedure . . . The already-established scientists of a discipline evaluate each other, especially newcomers, and decide what is worthy to be published. . . . Outside of the academic system, most people neither know what modern research is about, nor how to interpret the results and their potential importance to mankind. Although scientists often also do not know the latter, they are—in contrast to the layman—educated to conceal this lack of knowledge behind important sounding scientific jargon and formal models. In this way, even banalities and absurdities can be represented as A-journal worthy scientific excellence, a process laymen and politicians alike are not aware of. They are kept in the blissful belief that more competition in scientific publication leads to ever- increasing top performance and excellence.

Also this amusing bit:

Calculating published articles per capita, Switzerland becomes the world’s leading country . . . in no other country in the world are more research publications squeezed out of the average researcher than in Switzerland.

Are you listening, Bruno?

Binswanger lists a number of “modes of perverse behavior caused by the peer-review process,” most notably this one: “Form is more important than content.” I think about that all the time when I see papers backed up by “p less than 0.05.”

Nick Brown points us to this quote from Binswanger:

Cases of fraud such as the example of Jan Hendrik Schoen mainly affect the natural sciences, where the results of experiments are corrected or simply get invented. Social sciences often have gone already one step further. There, research is often of such a high degree of irrelevance that it does not matter anymore whether a result is faked or not. It does not matter one way or the other.

This reminds me of Clarke’s Law: Any sufficiently crappy research is indistinguishable from fraud. Just to clarify: I’m not saying that all, most, or even a large fraction of social science research is fraudulent, nor am I questioning the sincerity of most social science researchers. I’m just agreeing that in many cases the empirical evidence in published papers can be pretty much irrelevant, as we can see in the common retort of authors when problems are pointed out in their published work: “These mistakes and omissions do not change the general conclusion of the paper . . .”

That sort of attitude is consistent with the idea that publication, not research, has become the primary goal.

And this sounds familiar:

What scientists at universities and other research institutions are mostly doing are things such as writing applications for funding of research projects, looking for possible partners for a network and coordination of tasks, writing interim and final reports for existing projects, evaluating project proposals and articles written by other researchers, revising and resubmitting a rejected article, converting a previously published article into a research proposal so that it can be funded retrospectively, and so on.

I guess we could add “blogging” to that list of unproductive activities . . . Hey! I guess things could be worse. Imagine a world in which, in addition to everything else, productive researchers were expected to regularly blog their findings, participate in internet debates, answer questions posed by strangers in the comment sections of their blogs, etc.

In all seriousness, I’m glad that blogging is an option for researchers, but just an option, and not considered to be any sort of requirement. 15 years ago, when blogging was beginning to really catch on, one could’ve imagined an academic world in which blogging would’ve become expected behavior of junior and senior scholars alike, leading to a world of brown-nosing, back-stabbing, etc. I’m a little sad that blogging isn’t more popular—I hate twitter—but at least blogging hasn’t been sucked into the bureaucracy.

A few years ago, I wrote:

It’s easy to tell a story in which scientific journals are so civilized and online discussion is all about point scoring. But what I’ve seen here [in the case of a particular scientific dispute] is the opposite. The norms of peer reviewed journals such as PNAS encourage presenting work with a facade of certainty. It is the online critics such as myself who have continued to display a sprit of openness.

At the time, I was thinking of these positive qualities as a product of the medium, with online expression allowing more direct and less mediated discussion without the gatekeeping role that has created so many problem with PNAS, Lancet, and various other high-prestige journals. But maybe it’s just that blogs are a backwater, relatively well behaved because they haven’t been sucked into the incentive system.

Indeed, there are some blogs out there (none on our blogroll, though) that do seem to be “political,” not in the sense of being about politics but in being exercises in strategic communication, and I hate that sort of thing. In the alternative universe in which blogging had become an expected part of academic production, I guess we’d be seeing noxious politicking on blogs all the time.

27 Comments

  1. Anonymous says:

    “Following up on our discussion of hysteresis in the scientific community, Nick Brown points us to this article this article from 2014, “Excellence by Nonsense: The Competition for Publications in Modern Science,” by Mathias Binswanger, (…)”

    Yes, thank you! I was recently searching for this paper and couldn’t find it. This paper should become much more well-known, and cited in my opinion. It only seems to have been cited 13 times when i google it (???):

    https://andrewgelman.com/wp-content/uploads/2018/02/Binswanger-2013-Excellence-by-Nonsense.pdf

    (Side note: also possibly compare this to all the efforts concerning “improvements” and “changing of incentives” of recent years. To me it seems that a lot of these “improvements” are all working together to actually reinforce all the issues Binswanger wrote about…)

  2. Ed Hagen says:

    My analysis of the problem is that career advancement in science is based on outcomes, which have a large random element that is not under the scientist’s control, rather than study design and execution, which is entirely under the scientist’s control. Hence, there is a huge incentive to game the system to get the desired outcome:

    https://grasshoppermouse.github.io/2017/12/05/academic-success-is-either-a-crapshoot-or-a-scam/

    • Oncodoc says:

      I certainly have seen this in my field. Lots of compounds are proposed for the treatment of various malignancies, and the vast majority turn out to be of little value. However, if the molecule you are studying does work, your rewards will be huge. You will be invited to present at national meetings, you will be cited a lot, and pharma will send you around the country to give talks to community doctors, and you get generous honoraria. Thus if you are studying gamma methyl superdrug, you will be richly rewarded for success even if your study methodology is no better than the studies for alpha, beta, or delta methyl superdrug being done at the academic center across town.
      The biggest problem that I have seen result from this is not in stimulating bad data or bad methods since the community will recognize and criticize this. The definition of success does get pushed. Very modest improvements in median survival get presented with breathless praise. I always thought that if I can’t see the difference in survival curves from further than the twelfth row in the auditorium, it aint a success.
      Maybe all studies in an area should be done by a consortium of researchers and published anonymously.

  3. Cappella says:

    -> “To ensure the efficient use of scarce funds, the government forces universities…” & “Here we should distinguish between natural and artificial competitions”

    well, we’re primarily talking basic economics here; any hysteria arising is caused by erroneous economic premises. Economics is fundamentally about ‘efficient use of scarce resources’.

    Government bureaucrats, at their discretion, handing out large amounts of taxpayer money to universities/researchers/etc is NOT economically efficient and can never be so. That is in no way a ‘market competitive’ situation. Command-Economics is polar opposite to normal voluntary market economies. No such thing as “artificial competition” — that concept is total nonsense.

    Our massive government economic-interventions into the university “system” are outrageously counter-productive — even those getting the free money are unhappy about the amounts and the bureaucratic thrash to get it. America cannot afford such economic foolishness (note the Federal deficit will be 2 TRILLION in FY2019 –the Federal Government is bankrupt, but has a mystical printing press)

    For the American public, there’s a simple economic solution to all this — but you would see intense, genuine hysteria from the university-establishment … if it was even casually proposed.

  4. Anonymous says:

    >One might say that there is no real alternative to this sort of competition—but fifty years ago, or maybe even thirty years ago, the above picture would not have reflected what was happening at many universities.

    Andrew – if not that sort of competition, what was happening at many universities back then?

  5. James says:

    To the experienced researchers here: any advice for a just-starting-a-phd individual? I will work hard to be good at what I do and hence publish based on merit, but this takes a lifetime- how to fight this incentive system while trying to start your career?

  6. Anoneuoid says:

    How can you impress the research commissions responsible for the distribution of funds?

    Making accurate and surprising predictions or performing engineering feats. An intermediate step is generating the data or tools that allow this.

    This is mainly achieved by increasing measurable output such as publications, projects funded by third-party funds, and networks with other institutes and universities. In this way, “excellence” is demonstrated, in turn leading to easier access to further government research funds.

    This is bad.

  7. Neuronymous says:

    My goodness how time flies. There is a certain degree of naive anachronism in this thesis, albeit valid. Corporatization of elite universities is rapidly overtaking the artificiality of competition as the corrupting influence par excellence in academia. Elite universtities with inordinate influence over editorial decisions are using scn journals as glossy marketing platforms for their patented genetic technologies. Their pages are filled with optogenetic or crispr studies often highlighting how revolutionary they are in the first paragraph (except the latter remain in reality theoretical, with a dearth of approved clinical studies because of concern re on and off target effects). There may be a secondary publication bias or file drawer problem for any critical caveats to the general hysteria generated by gene editing headlines. Nonetheless resurrection of the woolly mammoth is imminent. Not to mention high publication rates of junk science including common variant GWAS in disease status. Thankfully the latter are not currently patentable but departments are constantly looking at gene products that may be. The problem has moved on with avengeance right under the noses of even the most astute critics.

  8. Rahul says:

    For funding projects (at least the smaller ones), what if we shifted focus from the “proposal”, which seems the centerpiece currently, to the individual researcher’s competence?

    Give intelligent, deserving people funds and trust them to produce output, on average. Don’t focus so much on the returns from individual projects since there’s such a large element of uncertainty and luck here.

    I mean, it’s not as if we are doing any better by demanding pages of budgets, and goals and timelines etc.

    • Anonymous says:

      You wrote: “For funding projects (at least the smaller ones), what if we shifted focus from the “proposal”, which seems the centerpiece currently, to the individual researcher’s competence? “”

      Yes, thank you!

      It seems to me that a lot of recent discussions about how to improve (psychological) science, seem to completely leave out 1) the role of the individual, and 2) how to reward the most competent individuals (so they can in turn produce the best science). Both to me seem crucial in (improving) science.

      Although it can perhaps be reasoned that it is sometimes a good thing to reward the proposal instead of the individual, this only holds when the proposal is actually “good”. Just like it only make sense to reward individuals that are “good”. Now, i reason this should all be assessed by Science and scientific principles (e.g. logic, evidence, etc.). If this makes any sense, the question for me becomes how to make sure:

      1) that Science will lead the scientists, and
      2) that the most competent scientists will be “rewarded” (e.g. via jobs)

      Some folks seem to me to think that (large-scale) groups and group-processes will produce the best science and reward the best scientists, which i largely disagree with. In fact, i reason groups and group-processes is what largely caused the mess in (psychological) science in the first place (e.g. groups of authors citing each others work, and defending it via peer-review and editorial roles, etc.).

      Anyway, the paper by Binswanger linked to above makes that all clear.

    • But how to keep this from being a “fund my best friends” scenario? My proposal was to score proposals and then fund them based on the sum of the score and a random number, partially debiasing the score.

      http://models.street-artists.org/2015/10/19/a-much-better-grant-evaluation-algorithm/

      • Rahul says:

        The fund-my-friends risk is common. I don’t think this proposal does anything to reduce that.

        The point I was making was about evaluating people vs proposals.

        The anti-nepotism and anti-cronyism aspects are present but orthogonal to this.

        • See, the thing is in large part what we have *already* is “fund people” rather than proposals. How do you think you get your proposal funded? In large part, it’s by being a recognized person in your field. So what we have right now is largely a “fund my friends” system in the guise of a “fund my proposal” system.

          Also, in biology my wife has a college who just got some kind of “fund this person” grant that exists already, I can’t remember what it was called something like mid-career something or other, but it was a big grant for quite a number of years. Off the top of my head, maybe like $8M over 10 years or something like that? The goal of the grant is to free people up from the rat-race of proposals… but how do they choose the people to get this grant? It’s people who have been very successful getting funding in the proposal rat-race… which happens because… publications and recognition and grants… It’s a nonlinear feedback effect.

          • Rahul says:

            What we need is a fund-the-best-researcher system with some controls to prevent it from becoming fund-my-friend.

            There’s two dimensions to the problem: proposal vs person & nepotism vs merit.

            Even with checks to improve merit, the question remains about whether to tie it to proposals or people. And, if, as you say, it is already people-centric, then why not get rid of the lip-service proposals?

            • Mikhail says:

              And surely, by best-researcher you mean the researcher who published the most papers in the journals that are considered prestigious?

              • Martha (Smith) says:

                This is probably not what Rahul has in mind, but is probably what his proposal would end up doing if there were an attempt to implement it.

              • I’m pretty sure Mickhail’s comment was tongue in cheek. But yes, that is the real issue.

                Still, Rahul, widely funding people to do research is basically a bet on an uncertain outcome. Is it a good bet? It depends a lot on what kinds of outcomes actually come about.

                One thing is if you can make funding be low cost (in terms of time, ie. not require a lot of overhead of constant rewriting proposals to the latest specifications etc) and not very dependent on faking up information about the project, then people will potentially engage more in actual research. If we get more preliminary research at low cost and can then fund more high quality research based on well established preliminary research… you might well get better results. The “UBI” commentary below sort of addresses this same idea.

              • Mikhail says:

                To expand on my comment more seriously:

                I think the main issues is how are we measuring the quality of the research/researcher and who is doing this measurement. not whatever we are going to fund people or proposal.

                I dont see that the system granting money to a person who “already wrote X papers on buzzwords” is better or worse than system granting money to proposal “related to buzzwords and promising to generate Y new papers”

  9. Mikhail says:

    I wonder if Universal Basic Income could improve the situation

    • Yes, it could, if combined with other useful policies. For me, the definition of a UBI is that it is a payment whose quantity depends on factors *other than income*. So factors like illness, age, or… participation in research, could be used to modify the quantity.

      For example you could dramatically increase the preliminary results required before getting project specific funding, and use some kind of Bayesian Decision Theory to decide on funding quantities, and then use UBI to allow people to afford to carry out the preliminary work. For example if you file a request for preliminary funding with a description of your project, you get a fixed dollar amount (same for everyone, or same for everyone in your field) to fund your preliminary research so long as you provide some kind of say twice-annual public report on progress. You’d also probably want to have a method for people to access public laboratory equipment, a kind of “Maker Space” for scientists. Overhead currently going to universities as part of project specific grants could be diverted to university run “maker spaces” where the qualification to use it is basically that you have taken some safety courses etc and pay hourly or daily for access to the space.

      It’s not a bad idea. If you took the existing say NIH budget and converted half of it to UBI type funding at say $50k/yr per person and half to project specific funding with a much higher bar for gaining that funding… you’d probably drive out many of the rent-seekers and be left with the people who were actually devoted to their particular idea and had enough confidence to work on it to the point that they could pass the higher bar for project specific funding.

      • There’s obviously plenty of gaming potential here too.. so you’d need to address that quite a bit. For example illiterate 90 year old dementia patients shouldn’t be getting research UBI increments just because their scheming neighbor signed them up… at the same time, restricting UBI payments only to PhDs employed by universities would be a terrible idea too… leading back to something like what we’ve got already. In the long term, it’s probably better to call this stuff something different than “UBI” because the usefulness of the true *universal* component of UBI is too important to be corrupted badly by associating it mostly with rent seeking…

        Whether a thing works in theory vs whether it works in practice with people scheming to get a piece of the “Free Lunch” are two different things. The more complicated the rules for the free lunch the more special schemes are possible.

    • Martha (Smith) says:

      “I wonder if Universal Basic Income could improve the situation”

      It might help somewhat, but there are always unintended consequences and unexpected problems in implementation, some of which Daniel discusses.

      But UBI possibly improving the research situation is (in some form) one more argument for UBI.

    • Mikhail says:

      I wonder if we can categorize major scientific discoveries by the source of income of their authors? People like Newton and Darwin did not actually HAD to do science for a living, have they?

  10. Mikhail says:

    Speaking about feminism…
    The word “Hysteria” have somehow problematic origins, I know some feminists arguing against using it.

Leave a Reply