Here’s why the scientific publishing system can never be “fixed”

Roger Peng

There’s been much discussion recently about how the scientific publishing system is “broken”. Just the latest one that I saw was a tweet from Princeton biophysicist Josh Shaevitz:

Editor at a ‘fancy’ journal to my postdoc “This is amazing work that will change the field. No one will publish it.” Sci. pubs are broken.

— Joshua Shaevitz (@shaevitz) February 13, 2014

On this blog, we’ve talked quite a bit about the publishing system, including in this interview with Michael Eisen. Jeff recently posted about changing the reviewing system (again). We have a few other posts on this topic. Yes, we like to complain like the best of them.

But there’s a simple fact: The scientific publishing system, as broken as you may find it to be, can never truly be fixed.

Here’s the tl;dr

The Marketplace of Publications

The first point is that the collection of scientific publications make up a kind of market of ideas. Although we don’t really “trade” publications in this market, we do estimate the value of each publication and label some as “important” and some as not important. I think this is important because it allows us to draw analogies with other types of markets. In particular, consider the following question: Can you think of a market in any item where each item was priced perfectly, so that every (rational) person agreed on its value? I can’t.

Consider the stock market, which might be the most analyzed market in the world. Professional investors make their entire living analyzing the companies that are listed on stock exchanges and buying and selling their shares based on what they believe is the value of those companies. And yet, there can be huge disagreements over the valuation of these companies. Consider the current Herbalife drama, where investors William Ackman and Carl Icahn (and Daniel Loeb) are taking complete opposite sides of the trade (Ackman is short and Icahn is long). They can’t both be right about the valuation; they must have different valuation strategies. Everyday, the market’s collective valuation of different companies changes, reacting to new information and perhaps to irrational behavior. In the long run, good companies survive while others do not. In the meantime, everyone will argue about the appropriate price.

Journals are in some ways like the stock exchanges of yore. There are very prestigious ones (e.g. NYSE, the “big board”) and there are less prestigious ones (e.g. NASDAQ) and everyone tries to get their publication into the prestigious journals. Journals have listing requirements–you can’t just put any publication in the journal. It has to meet certain standards set by the journal. The importance of being listed on a prestigious stock exchange has diminished somewhat over the years. The most valuable company in the world trades on the NASDAQ.  Similarly, although Science, Nature, and the New England Journal of Medicine are still quite sought after by scientists, competition is increasing from journals (such as those from the Public Library of Science) who are willing to publish papers that are technically correct and let readers determine their importance.

What’s the “Fix”?

Now let’s consider a world where we obliterate journals like Nature and Science and that there’s only the “one true journal”. Suppose this journal accepts any publication that satisfies some basic technical requirements (i.e. not content-based) and then has a sophisticated rating system that allows readers to comment on, rate, and otherwise evaluate each publication. There is no pre-publication peer review. Everything is immediately published. Problem solved? Not really, in my opinion. Here’s what I think would end up happening:

What Can be Done?

I don’t mean for this post to be depressing, but I think there’s a basic reality about publication that perhaps is not fully appreciated. That said, I believe there are things that can be done to improve science itself, as well as the publication system.

Ultimately, in a universe where there are finite resources, a system has to be developed to determine how those resources should be distributed. Any system that we can come up with will be flawed as there will by necessity have to be winners and losers. I think there are serious efforts that need to be made to make the system more fair and more transparent, but the problem will never truly be “fixed” to everyone’s satisfaction.