“We are moving from an era of private data and public analyses to one of public data and private analyses. Just as we have learned to be cautious about data that are missing, we may have to be cautious about missing analyses also.”

Stephen Senn writes:

For many years now I [Senn] have been making the point that obtaining a license to market a drug should carry with it the obligation to share the results with interested parties. . . .

Amongst those misunderstanding the issues, are many who work in the pharmaceutical industry. A common assumption is that any company that releases data will help competitors who will, of course, mine their publicly released information for valuable insights. The point is, however, that releasing companies will also gain such information from their competitors and this is not a zero-sum game. On average, the industry should be better off. The problem is that it is to nobody’s advantage to be the first to reveal and it is in this context that strong regulatory action is desirable.

Hey—as a political scientist, I can definitely relate to that sort of prisoner’s dilemma argument.

Senn continues:

However, some of those associated with the campaign have also misunderstood an important point. Publication in the medical press must become irrelevant. Self-publication is necessary.

Why?

The first reason is that submission to a journal is not enough to guarantee publication, since the editors may reject the paper. The second reason is that journals may have a bias against negative studies. Although a great deal of research on this topic claims that this is not true, this research is inadequate for reasons I have explained in detail elsewhere. In particular researchers have naively (implicitly) assumed that authors make their decision to submit to a given journal based on quality of the research but not on anticipated probability of its acceptance. If the latter is the case we would not necessarily see different acceptance rates of papers submitted but higher quality of the negative papers.

Good point.

Senn continues:

The third reason is that the review process, by requesting changes in statistical analysis, can harm the quality of a trial which (certainly, if it is a phase III pharmaceutical trial) will have had a detailed proposal for the statistical analysis registered prior to un-blinding the data. Whether or not, others subsequently decide that alternative analyses are superior, it is important as a matter of historical record that the results of the pre-specified analysis are available. . . .

Thus, the whole business of journal publishing simply adds confusion to the process of making trial results available. . . .

My solution is rather different. I think that we should move to a system where in addition to Q for quality, E for efficacy and S for safety aspects of a drug and, increasingly, V for value for money, we should have a P for publication requirement. By publication I do not mean publication in journals, I mean self-publication on the web or in some publicly searchable registry such as ClincalTrials.gov [emphasis added]. . . . Forget the journals. Let’s look elsewhere.

This represents an interesting convergence, from the applied side, with Larry Wasserman’s suggestion that researchers in statistical theory and methods abandon journals and just post articles on Arxiv. From both directions there is a sense that journal publication is causing more harm than good.

40 thoughts on ““We are moving from an era of private data and public analyses to one of public data and private analyses. Just as we have learned to be cautious about data that are missing, we may have to be cautious about missing analyses also.”

  1. The single best reform science can make right now is to decouple publication from career advancement, thereby reducing the number of publications by an order of magnitude and then move to an entirely disjointed, informal, online free-for-all communication system for research results.

    “Reviewers” and “aggregators” will spring up the same for academic papers as for any other kind of content. Their job will be helped immensely by the drop in publications. Anyone who wants to can “review” papers and those that good at it will attract a grateful audience.

    They’ll be the Siskel and Ebert of research papers (or perhaps more accurately, a modern day Mersenne http://en.wikipedia.org/wiki/Marin_Mersenne). People will become so dismissive of any claims when the data isn’t made available, sharing data will very quickly become the norm.

    Just let hiring/tenure decision in academia be based on cliques, personal popularity, and whether they went to the same kinds of schools as the members of the hiring committee. That’s what happens anyway and since academia is becoming less relevant every day, who cares?

    Academia is almost entirely welfare for self important boobs, so just make it official and stop requiring any “research” for their funding. Not silting up the communication channels is the single best contribution most academics can make. Let’s give them the opportunity to make it.

      • Corey,

        You should have seen it before editing. People used to talk about how the ivory tower was cut off from the world, but I think it’s gotten much worse over my lifetime. I don’t don’t think the average Professor – who very possibly has never had a job outside of academia even as a kid – realizes just out un-respected they really are. They hang around people just like themselves, congratulating each other on how smart they are and referring to each other as the “elite”. Meanwhile entire fields can go half a century will no appreciable increase in predictive power despite the expenditure of trillions of research dollars over that time.

        (The R&D expenditures in the US alone in 2007 was over $350 billion according to the NSF. see here http://www.nsf.gov/news/mmg/media/images/sei2010_4_h.jpg)

        But I’ll take you comment as an excuse to restate:

        Wasserman and Senn are absolutely right, but it means little unless academic promotion is decoupled from publication. We should just buy the bulk of academics off and keep them out of the way of those who want to do real work, untill they all get fired when MOOCs reach critical mass or whatever.

        • Entsophy: At http://www.entsophy.com/About.htm you state

          “My diverse background is built on five Masters of Science degrees: Physics, Statistics, Finance, Pure Mathematics, and Software Engineering. I hope to get a Ph.D. one day, but am currently torn between Biophysics and Economics (suggestions welcome).”

          You still want to re-enter the system you despise so much?

        • Ha! I was just about to change that. The answer’s no. I believe I have almost no chance of doing anything important outside of academia, but exactly zero chance of doing anything important inside of it. Not to mention the fact that I make far more money and have far greater control over my life then I ever could in academia. There’s just no scenario were spending three years working on someone else’s dumb thesis idea makes any sense.

          I probably will add a MS or two in economics and microbiology at some point though.

        • Yes exactly. Academia is so broken at this point, that anyone who isn’t a complete knucklehead should get a foundation level of knowledge out College and then get the hell out. They’ll be considerably happier and more productive. They’ll actually have a chance, however small, of writing something that someone hundreds of years from now would care about. Plus they’ll be able to avoid that very special kind of loser that’s attracted to academia like dung beetles are to dung.

        • If you are any good, you spend three years working on your own smart thesis idea and having a lot of fun at least some of the time. I wouldn’t expect someone with five master’s degrees to need to be told what to do.

        • Uh there’s nothing stopping me from working on “smart thesis” now. Plus I avoid the poor pay, teaching snotty freshman, and ridiculous academic politics.

          There may have been a time when academia afforded people more time and room to work on a “smart thesis” but that is long over. I have far more time and energy available for such work outside of academia than I ever would inside. Like Einstein said, if you’ve really got a good idea and know it, then get a job as a light house keeper and work on it.

          So what does that tell you about everyone who choose to be a professor rather than a light house keeper?

        • Also you’re experiences are very atypical. Most academics are stuck in whatever hell hole gave them a tenure track position. If they want to live in the NYC, too bad. If they want to live on a beach, too bad. They’re stuck at Podunk University, Podunksville.

        • To be serious for a minute, when I was 25 I chose to go for an academic research career for three main reasons: (1) completely flexible schedule which I wanted to have in anticipation of the time that I would have children; (2) when I looked at guys in their 50s and what jobs they had, the guys with research jobs seemed happy with what they were doing (even if their research was not exciting to me, they seemed to enjoy their jobs) and the guys with other sorts of jobs did not; and (3) I can work on whatever I want. I have to admit that #2 was tricky. In my first academic job, the dept chair tried to talk me out of working on what I wanted to work on. I realized that “tenure at Berkeley” wasn’t as important to me as “doing what I wanted” (and of course I knew I had other options, so it’s not like I was going to starve) so I kept going. So the incentives didn’t all line up for me, but they mostly did.

        • @Entsophy

          I’ve seen both academics & industry. There’s stupid people in both. Are you really saying industry doesn’t have its share of snottiness, politics & stupidity?

          You are painting with too wide a brush. Some people like it here, others there.

        • There’s a lot to be said about the learning from a masters versus a Ph.D. That said, getting the union card is good if you are part way through a Ph.D. track program. Just remember it is pass fail and just get the minimum done and go. Might as well wrap it up. Of course the MBA makes more sense money wise over either.

          Finally AG, you are clearly way above the mean both in terms of brains/ability. Mick Jagger may love being a rock star, but the outlook for the average wannabe musician is not so good. But I’m happy that things worked for you and that you have a good life. Appreciate the outreach and engagement.

        • Andrew,

          You caught the tail end of the golden age of academics and went to Harvard and MIT. I have a 174 IQ according to Oxford University (England), but grew up later and there’s simply no possibility of my ever duplicating your career path. At least not without spending every last waking moment on useless status games and an enormous amount of luck and an even bigger amount of misery. The vast majority of academics today faced with your circumstances at Berkeley are stuck there for the rest of their lives. Academia is irrevocably broken.

          Shalizi is bragging about getting tenure at 40 or however old he is and finally getting the same full measure of freedom that Einstein trivially got at 25 as a nobody. And Shalizi is, by all accounts, a major (and lucky) academic success today.

          If you’re serious about research today and you want to maximize the amount of time you can spend on it, and control where, when, with whom, and on what, you spend that time on, the last thing you’d choose is an academic career.

        • I agree with Entsophy that times have changed from the time Andrew started in academia.

          Most tenured professors don’t seem to be aware. I had one tenured professor tell me in his graduating class, back in 1970s, everyone got a good academic job. In my recent class I’d say 35% did.

        • Let me finish that final thought by answering the question: under current circumstances what kind of people do choose academia?

          Basically three kinds:

          (1) Supreme narcissists who need the status and social approval.
          (2) Cowards who are afraid of the real world.
          (3) Those who know their ideas aren’t good enough to make it in a real intellectual free-for-all.

          Those are the types that academia is overflowing with. The same kind of thing happens to the Officer Corps in the military whenever there’s a major drawdown. The best have real prospects in the Civilian world, so as opportunities in the military dry up, they leave. The ones who stay are the “time servers” who couldn’t get an equivalent career anywhere else.

        • No, not at all. I have two close colleagues who are working in academia with me on Stan because they think it is an interesting and important project and they like the freedom. They could easily get high-paying industry jobs (indeed, one of them left such a job to come work here) but they choose not to. Sure, if we had 10 million dollars we could just self-fund, but as it is, academia provides an environment where people can work on open-source software without pressure. Academia has lots of problems but you’re too sweeping in your criticisms.

        • I thought I’d be the odd one out, but that’s not so. Although the Marines are probably the dumbest of the services, the average Officer (according to all that psychometric testing they do in the military) wouldn’t have any trouble getting a Ph.D. in the Social or Life Sciences.

          There were a few as well who either left to get Ph.D. from to 10 programs or arrived with Ph.D.s from such programs.

          The knuckleheads in academia running around telling each other how “elite” they are, really are full of it. There are 70 MILLION people on this planet with IQs in the top 1%. Which means you’re about twice as likely to be one of them as you are to be Canadian.

          So you aint that special.

        • Oh and people who claim to be “elite” and publish hundreds of papers in “top” journals, but reside in fields that have been stagnant over long periods of time really need to re-acquaint themselves with reality.

  2. > journal publication is causing more harm than good.
    And that is not already widely and well understood?

    (The issue with things like clinical trails is that you really need access to the raw data to know if its credible or access confidentially by qualified third parties to verify. Diffrent from statistical theory?)

  3. Crystallography is a field with very good filing of data. Pre-requisite even for review of your paper. Not just a pre-req for publishing and definitely not some loosey-goosey promise to file later or worse to share with interested parties (as long as they are not critics!)

    Papers can still be useful for discussion of some things (e.g. comparison to other structures, chemical insights, comparison to physical properties, relevance to other research and economic implications). Plus, it gives academics their counting points.

    I actually think that filed government data (e.g. EPA requirements) and/or old school government/company technical reports can be much more honest and reliable than academic science papers.

    I actually think we would benefit from government payment for foundational work (phase diagrams, thermodynamic data, treering studies, etc.) that may not be Science/Nature exciting, but would be very helpful to everyone doing applied research. Since there is a market failure to produce it, would be helpful for the government to do so. However man on the moon or curing cancer is much more exciting politically. :(

  4. If we could work out the details cleverly, arXiv could be a good component of some system of post-publication review. That would permit review that does not interfere with the published analysis.

  5. Journal publication started as a way for scientists to communicate with each other.

    With repositories like Arxiv, the internet, and search engines journals have become, not just obsolete, but counterproductive (publication bias, press hype, pay walls, etc.).

    IMHO journals presently serve mostly a social and professional function in academia. One for which they were not designed, further distorting the scientific enterprise.

    • I’m not sure. Journals always provided a vetting function. Even back in the days, unknown scientists submissions were often introduced by a more established scientist. Communication is poor without curating. I don’t want to waste time reading every quack and crank that produces a paper.

      • Rahul:

        I think the peer review function is greatly exaggerated. In my experience, with few exceptions, it has proven a waste of time. Part of the problem is that peer review has two functions: (1) Assess quality and (2) make publishing recommendations. The latter can get in the way of the former when reviewers have egos, turf to defend.

        Moreover, the focus has gone so much to the publishing decision that reviewes are not even checking quality so much as assessing whether a “finding” will be “of interestto the journal’s readership”. How often has a reviewer asked for your code?

        To be honest I don’t trust the curating role of journals where quality is concerned. Besides there are much better ways of curating than through the current journal system. (I’m appealing to your imagination.)

        • Why don’t we work on these alternative ways of curating systems in parallel? The moment they are mature and useful, we’d abandon Journals. I’ve nothing against that.

          What I don’t like is this state of anarchy where journals cease to exist, barriers to publication disappear, & everyone and his uncle dumps all sorts of crap into a neo-arxiv.

          Seriously, I’d love an arxiv-like site where credentialed reviewers comment on papers & offer an informal imprimatur. I only think that the alternative curating solution must precede the journal dismantling. Not the other way around.

      • PS The internet has also greatly magnified the reach of journals, creating a superstar economy. The increasing supply of manuscripts combined with the focus on superstar journals only exacerbates the emphasis on the decision to publish. At the expense of quality.

        IMHO we should publish everything thats fit to publish, not just the few items that fit in a 20th century print publication. Quality should be the only criterion. Then journals can build a reputation on the quality of their review not the “interesting” of their published articles.

  6. The point is, however, that releasing companies will also gain such information from their competitors and this is not a zero-sum game. On average, the industry should be better off.

    That’s a silly argument I think. The auto-industry or semiconductor industry too could be better off if we forced firms to open all their data or blueprints to each other. Should we?

    “The industry may be better off” seems a pretty bad reason to push for strong regulatory action

  7. I don’t want to waste time on cranks and I also want to force people to write with proper English (consider how many foreign scientists we see submitting to US pubs). Also, just some people need pressure to organize their thoughts.

  8. Two people in two completely different disciplines who both think peer review via scientific journals in their discipline does not serve whatever (probably different) goals they think are desirable may or may not be an ‘interesting convergence’ … or it may be a wild generalization.

    Is the reasoning that counts for medical trials really going to generalize to other people in other disciplines who will probably have other ideas about what purposes scientific publication should serve?

    • Why spend serious time on that for the chance of a 3K reward? Wouldn’t time be better spent on day job?

      For that matter, um…do a lit search? Google search?

  9. Pingback: Surveys : A History | Emma's Research and Television Blog

  10. Pingback: Somewhere else, part 117 | Freakonometrics

Comments are closed.