Skip to content
 

Raghuveer Parthasarathy’s big idea for fixing science

Raghuveer Parthasarathy writes:

The U.S. National Science Foundation ran an interesting call for proposals recently called the “Idea Machine,” aiming to gather “Big Ideas” to shape the future of research. It was open not just to scientists, but to anyone interested in potentially identifying grand challenges and new directions.

He continues:

(i) There are non-obvious, or unpopular, ideas that are important. I’ll perhaps discuss this in a later post. (What might you come up with?)

(ii) There is a very big idea, perhaps bigger than all the others, that I’d bet isn’t one of the ~1000 other submissions: fixing science itself.

And then he presents his Big Idea: A Sustainable Scientific Enterprise:

The scientific enterprise has never been larger, or more precarious. Can we reshape publicly funded science, matching trainees to viable careers, fostering reproducibility, and encouraging risk?

Science has transformed civilization. This statement is so obviously true that it can come as a shock to learn of the gloomy view that many scientists have of the institutions, framework, and organizational structure of contemporary scientific research. Issues of reproducibility plague many fields, fueled in part by structural incentives for eye-catching but fragile results. . . . over 2 million scientific papers are published each year . . . representing both a steady increase in our understanding of the universe and a barrage of noise driven by pressures to generate output. All of these issues together limit the ability of scientists and of science to tackle important questions that humanity faces. A grand challenge for science, therefore, is to restructure the scientific enterprise to make it more sustainable, productive, and capable of driving innovation. . . .

Methods of scholarly communication that indicate progress in small communities can easily become simple tick-boxes of activity in large, impersonal systems. Continual training of students as new researchers, who then train more students, is very effective for exponentially expanding a small community, as was the goal in the U.S. after World War II, but is clearly incompatible with a sustainable, stable population. The present configuration is so well-suited to expansion, and so ill-suited to stability . . .

It is hard to understate the impact of science on society: every mobile phone, DNA test, detection of a distant planet, material phase transition, airborne jetliner, radio-tracked wolf, and in-vitro fertilized baby is a testament to the stunning ability of our species to explore, understand, and engineer the natural world. There are many challenges that remain unsolved . . .

In some fields, a lot of what’s published is wrong. More commonly, much of what’s published is correct but minor and unimportant. . . . Of course, most people don’t want to do boring work; the issue is one of structures and incentives. [The paradox is that funding agencies always want proposals to aim high, and journals always want exciting papers, but they typically want a particular sort of ambition, a particular sort of exciting result—the new phenomenon, the cure for cancer, etc., which paradoxically is often associated with boring, mechanistic, simplistic models of the world. — ed.] . . .

Ultimately, the real test of scientific reforms is the progress we make on “big questions.” We will hopefully look back on the post-reform era as the one in which challenges related to health, energy and the environment, materials, and more were tackled with unprecedented success. . . . science thrives by self-criticism and skepticism, which should be applied to the institutions of science as well as its subject matter if we are to maximize our chances of successfully tackling the many complex challenges our society, and our planet, face.

“Radio-tracked wolf” . . . I like that!

In all seriousness, I like this Big Idea a lot. It’s very appropriate for NSF, and I think it should and does have a good chance of being a winner in this competition. I submitted a Big Idea too—Truly Data-Based Decision Making—and I like it, I think it’s great stuff, indeed it’s highly compatible with Parthasarathy’s. He’s tackling the social and institutional side of the problem, while my proposal is more technical. They go together. Whether or not either of our ideas are selected in this particular competition, I hope NSF takes Parthasarathy’s ideas seriously and moves toward implementing some of them.

You are welcome in comments to discuss non-obvious, or unpopular, ideas that are important. (Please do better than this list; thank you.)

25 Comments

  1. Thanatos Savehn says:

    Ok, so long as we remember that science is the navigator and not the captain.

  2. Thanks, Andrew! Like I emailed: we should have joined forces…

    You should post your proposal — I’d love to read it, and I’m sure many others would as well!

  3. Terry says:

    How much of human effort is “wasted”.

    We see in academia that a huge amount of effort produces no useful end-product. But, when we look across society, we see enormous waste in many places. I have begun to think there is enormous waste almost everywhere. So, how does the waste in academia compare to waste elsewhere? Are advanced societies now so rich that most of what we do is a waste in some sense? What if almost all university education is a waste in the sense that students could learn just as much sitting at home with just textbooks and test packets? Is society so rich that students can afford to “waste” four years in what is basically an extended vacation in academia?

    Measuring waste is quite problematic though. One measure would be the amount of resources actually consumed relative to the resources that would be consumed if the end-product were produced more efficiently. This would be a low end estimate. A much higher estimate would come from categorizing as waste anything beyond the necessities for living. After all, we don’t NEED to put wallpaper on our walls, we could live perfectly well without wallpaper, or even paint. Do we really need 24-hour political chatter on multiple cable stations? King Lear nails this:

    O, reason not the need! Our basest beggars
    Are in the poorest thing superfluous.
    Allow not nature more than nature needs,
    Man’s life is cheap as beast’s. Thou art a lady:
    If only to go warm were gorgeous,
    Why, nature needs not what thou gorgeous wear’st
    Which scarcely keeps thee warm.

  4. Dale Lehman says:

    I like this proposal as well and I second the desire to see Andrew’s. Regarding the discussion to this point:
    I think the greatest damage done by wasteful and unproductive research is the growing discrepancy between the rate of technological “progress” and our ability to understand and control what we are creating. Amidst all the unnecessary/wrong/meaningless research, we have this amazing stream of new technologies which enable new products and services. Terry wonders who needs wallpaper on walls? What about wallpaper we can talk to and will provide images that we desire at any given moment? Certainly we don’t need that – but we’ll have it available (thank you, Siri). But our understanding of how these things change the nature of human existence and interaction does not keep up with our ability to create new things.

    To return to the thread – what are the “big questions?” I wonder if there would be any agreement on what these are – or even on how we would determine whether a question was “big” or not. Might this be another proposal for NSF to consider?

  5. yyw says:

    Great proposal. I agree that the structure and incentive are the root causes. The funding mechanism to me is broken. Enormous amount of resources are being spent on writing proposals (promising and selling) and evaluating proposals (supposedly only revolutionary research will get funded) and yet very little is spent on evaluating the quality of the funded research. I would propose to award research funds to individual researchers not projects. Each beginning researcher gets a small fund and can do as they please with it. After say 5 years, a rigorous evaluation of the research outputs will be conducted and those doing solid work will get funding for the next 5 years etc. Researchers doing exceptional work and might benefit from larger funding can be given more. The key is to shift from evaluating the promises to evaluating the output (quality).

    • Anonymous says:

      1) You wrote “The funding mechanism to me is broken. Enormous amount of resources are being spent on writing proposals (promising and selling) and evaluating proposals (supposedly only revolutionary research will get funded) and yet very little is spent on evaluating the quality of the funded research”

      I have wondered whether big grant agencies actually evaluate how their money is being spent. If this is not routinely done, i would find this astonishing, and i think this may have contributed to many of the problematic issues in today’s science.

      2) You wrote “I would propose to award research funds to individual researchers not projects. Each beginning researcher gets a small fund and can do as they please with it. After say 5 years, a rigorous evaluation of the research outputs will be conducted and those doing solid work will get funding for the next 5 years etc. Researchers doing exceptional work and might benefit from larger funding can be given more. The key is to shift from evaluating the promises to evaluating the output (quality).”

      Yes! I like this idea a lot, especially giving out small funds to relatively more researchers. Compare this to many proposed “solutions”, like big “collaborative” projects, that possibly waste tons and tons of resources. To me, it’s almost like getting media attention, and money, for your “solution to improve science” is simply the new “sexy study finding” of the last decades…

      In fact, my “Big Idea” would be to simply stop funding for a period of 5 years. I am hereby focusing on Psychological Science because that’s what i know most about. I reason stopping (or equally distributing) funding could lead to psychological science researchers having to come up with solutions that i reason could improve psychological science altogether (with some much needed attention for economically using resources).

      For instance, i tried to (among other things) find a solution for the waste of small (non-significant and/or non-informational) studies (that possibly get put in the file-drawer), and possible waste of grant writing. I reasoned small groups of researchers all working on the same topic/theory/phenomenon and who would replicate each other’s work pre-publication (see the format here https://andrewgelman.com/2017/12/17/stranger-than-fiction/#comment-628652) could result:

      1) wasting less time on writing grants because proposals would be written by small groups of researchers who would all propose their own studies on a single topic/theory/phenomenon. You would also have the additional possibility of one researcher getting a grant and subsequently asking other researchers to collaborate with him/her (and provide them with some funding from the received grant to do this)

      2) possibly opens the door for the proposal above to allocate relatively small funding to many individual researchers (and possibly evaluating that process). These researchers could use the funding individually, and/or work together by using a format like i described.

      3) maximizes the informational value of studies, maximizes the chances of getting published, maximizes economically based decisions concerning resources, and maximizes theory (re-) formulation and -testing. All this seems way more scientifically sound, and responsible, compared to the way things have gone in the last decades to me.

      A few simple “tweaks” in the standard research- and publication process could, in my reasoning, result in much better psychological science. In my reasoning, this all could ultimately save many resources, and improve psychological science concerning several crucial aspects. And, this would all cost no extra money!

  6. Keith O’Rourke says:

    I see this as a request for better management of science that anticipates a few changes (prescriptions) – but by whom and how to make that continuous? (Management by prescription is the less effective and wears out very fast.)

    Managing anything is hard and usually fails with passing time (as Terry put it, there is waste everywhere even places where management is “firmly” in place).

    Now, a large part of managing science involves the economy of research which despite the word economy suggesting economics is mostly a statistics problem (or should be if more broadly thought of as expediting organised experience https://andrewgelman.com/2017/11/29/expediting-organised-experience-statistics/).

  7. I would say that the thought leaders of the Evidence Based Medicine movement were the catalyst for the concept of A Sustainable Scientific Enterprise, in hearing the discussions among some academics back in Boston. And of course Stanford’s METRIC program, under the aegis of Steven Goodman and John Ioannidis [particularly have expanded its scope to apply to economics, psychology, and quite probably in the meta-analysis of research more broadly across disciplines. So I am suggesting that this has been a long term goal and objective. I recall that in the 60’s, Linus Pauling gave a speech where he mentioned that science itself needed oversight. John has given several keynotes referencing the extent of wasteful expenditures in research.

    I do think that ‘unpopular ideas’ are given short rift due to a whole variety of reasons. So I do welcome some effort to countenance them. But I have also seen them distorted to dent their impact or translated into contexts where they do not belong.

    • Keith O’Rourke says:

      > Evidence Based Medicine movement were the catalyst for the concept of A Sustainable Scientific Enterprise
      That was (is) the hope but recall https://andrewgelman.com/2016/07/21/ioannidis-evidence-based-medicine-has-been-hijacked/ which just underlines the need for continuous management with expected mishaps.

      We should expect a kidnapping attempt on Stanford’s METRIC program.
      (I think I already noticed one attempt on a past recorded Metrics symposium where Goodman disarmed the perpetrator. OK, sometimes attempts to control the vocabulary are just attempts to steal the brand rather than outright kidnappings.)

      • Keith that is a wonderful last sentence. I couldn’t have said it better myself. Also thanks for linking that 2016 discussion on this blog. I hadn’t followed Andrew in 2016. Since then, John Ioannidis has repeated the content of that letter in his talks on campuses.

        Just recently read John Ioannidis letter to Sackett and watched its main points presented in a Youtube video, which constitute an astounding confession before an academic audience. That example of ‘hijacking’ wasn’t on my mind. But it is an excellent example of what I mean.

        I’ve become more cautious in conveying ideas to just anybody b/c I have 1st-hand experience for how ideas can be so distorted and translated into contexts for which they were not intended.

        All this discussion reinforces a theme that I have conveyed consistently to international relations circles. And that is eclectics in nearly every discipline rule the idea market, even if what they hold is distasteful to the status quo. Many experts co-opt from them. In turn sometimes misinterpreting the original intent and context. I would draw on Charles Sanders Peirce, as an example of an eclectic, who could not benefit financially from his ideas. But it looks as if others did eventually.

        In 1999, I had conversations with some scientists from MIT, who said that there was a dirth of creativity in academia. I think that has something to do with the student recruitment processes, a point that Robert Sternberg has emphasized continuously since the 80’s. That may be. If so, exceptionally creative may not have the academic profiles that are sought traditionally in elite schools. Or they simply may not be academically and professionally ambitious. Every case is different.

        So I’m not surprised that NSF is enthusiastic about promoting an arena to curate ‘big ideas’. There is an apparent dirth of it in some domains so it is said.

        As an aside, I knew of Linus Pauling statements from a lady who had known Pauling and his wife. She introduced me to orange juice flavored Vitamin C supplements at the age of 12. Pauling himself recommended high doses of Vitamin C.

        Hopefully I made some sense. I’m just writing off the cuff and without proof reading. So apologies.

  8. Anonymous says:

    “More commonly, much of what’s published is correct but minor and unimportant. . .”

    Any evidence for this?

  9. Anonymous says:

    It would be naive to assume the primary goal of the NSF is to improve science rather than expand it’s own power and funding. Even the secondary goal is likely for it to become a more attractive conduit for “pork” spending and “job creation”, ie effective “jobs programs”.

    So I would think improving science could be a tertiary goal of this organization at best. Thats just how the US government works, any part of it not acting in this way will shrivel and die.

    • Andrew says:

      Anon:

      It’s my impression that people work for the NSF because they want to help support good science. I think it is your cynicism that is naive. Naivety can come in many forms, including excessive cynicism.

      • I think the difference here is you focus on the individual motivations of workers at the NSF, whereas Anon focuses on the emergent behavior of the “NSF organism”. Basically I personally think you’re both right, at the appropriate scale of observation.

        • I don’t know anything about the National Science Foundation. Nevertheless, I am sure that new ideas are welcome everywhere, some of which may be further characterized as ‘unpopular’. I would say that ‘unpopular ideas’ are always in the mix for consideration. We need to be more precise by what we mean by ‘unpopular’. Is there an assumption that they are more meritorious than ‘popular’ ideas?

          I think that David Kennedy’s latest book is well worth a read: A World of Struggle: How Power, Law, and Expertise Shape the Global Political Economy relates to the question of how ‘big ideas’ get distorted or misinterpreted.

          Actually machine learning techies that I hob nob with have said that the era of renaissance minds is over due to hyperspecialization. So the era of big ideas. That is one implication of John Ioannidis own views on the future of research in the biomedical enterprises. That is to say, John doesn’t seem to foresee any big breakthroughs in these enterprises.

          So that means that someone’s creative acumen may come into play. But that would mean diversifying the entry into universities beyond what has traditionally bee the pattern.

      • Anonymous says:

        …but it’s also possible that it is naive to believe that excessive cynicism is naive?

      • Anonymous says:

        Its no less realistic than saying the primary goal of a corporation is profit. Some people working there may have other motives but without at least some plan to make money the organization will fail to thrive.

  10. Andrew’s initial response is a very good one. I am curious how many NSF panels “Anonymous” has served on (me: 11), and how many NSF program officers he/she has spoken with. Uniformly, they care about promoting science as their primary goal. As Daniel (correctly) notes, history and organizational structure complicate this. One makes the (local) decisions one things is best, under a (global) structure that may be flawed. It is a naive cynicism that attributes this to deliberately bad intentions. “Anonymous” tries to back off of this, claiming later that all he/she meant was that the overall organization wants to survive, but this is a trivial statement — the negative, that an organization wants to die, is obviously false.

    • My understanding is that governments, non-governmental organizations, and institutions bring in outsiders to generate new ideas and organizational changes. I was thinking of Gunnar Myrdal who had considerable influence on government officials and school desegregation case law. And easily John Ioannidis could be characterized as an ‘outsider’, attempting to restore the Evidence Based Medicine movement to its original intent which was to undergird medicine with science.

      Interestingly some of the same issues raised here were raised with respect to the National Academy of Sciences, with an interesting debate between Jarod Diamond and Serge Lang over the alleged politicizing of the Academy, and science in general.

      http://www.jareddiamond.org/Jared_Diamond/Further_Reading_files/Diamond%201987_1.pdf

      https://en.wikipedia.org/wiki/Serge_Lang

      It comes back to the broader measurement questions that we discuss here on Andrew’s blog.

    • Oops my last post didn’t post.

      It’s interesting that Serge Lang and Jared Diamond had quite disparate views on the alleged politicization of the National Academy of Sciences. Serge documents his views extensively in his book Challengea

      And there is also Jared Diamond’s narrative

      http://www.jareddiamond.org/Jared_Diamond/Further_Reading_files/Diamond%201987_1.pdf

    • Anonymous says:

      I don’t see any backing off, since the shrivel and die comment was in the original post.

      Then you say you agree with Daniel Lakeland’s comment, who agreed with mine. But it seems you want to minimize the dominant force driving NSF activities by calling it a “want”. It is not a want, it is a requirement.

      Also, (like Andrew) you want to focus on the motivations of individuals within NSF, which (like Daniel noted) isn’t really relevant. Ask people why they p-hack or play other dubious games with their data, and they will almost always tell you it is “to survive”. The individual desires are overruled by the needs of the system.

      The obvious next step for people who accept this point is to consider whether it is wise to continue, or even increase, feeding an existing malfunctioning system. If an organization messes up really bad and then gets more funding as a result what is going to happen?

    • Raghuveer,

      Thanks for making such excellent recommendations. My guess is tha some subsets non-scientists [anyone] have been included for their creative proclivities. Maybe it is the function of self initiative that those subsets. Hobbyists, for example, follow the research of the scientific community. I found my way into these communities through exposure some leadership through the White House Council on Science & Technology, AAAS, and NAS. I consider myself a ‘hobbyist’ although I have garnered some substantive expertise, in a couple of areas, in these 17 years in DC. More recently through some academics funded by the NSF.

      Coming up with new ideas is a very complicated non-linear process, sometimes taking decades. Creative types, at least those pursuing purposeful endeavors, are not likely to far as well in bureaucratic settings. They need space and time. So in that sense I think the organizational cultures could be more adaptive than they are.

      This week, there was a Mindfulness conference in Washington, DC. Looks like there are some good efforts in organizational change.
      http://leadershipsummit.mindfulleader.org/#2018-mindful-leadership-summit-1

Leave a Reply