Make Peer Review Public

How to make science faster and more robust.

Flooding after Hurricane Sandy

Water pushed up by Hurricane Sandy splashes into the window of a building standing by the shore in Bellport, New York, Oct. 30, 2012.

Photo by Lucas Jackson/Reuters

The latest predictions from the usually dire climate scientist James Hansen made a lot of people sit up and take notice. The Washington Post ran a story about the study with the headline, “The world’s most famous climate scientist just outlined an alarming scenario for our planet’s future,” and Slate called it a “bombshell sea level warning.” Most coverage paid attention to, among other things, Hansen’s prediction that climate change is “triggering major sea level rise in a time frame of 50 to 200 years.”

But what also stood out to some—including us—was that the eye-catching study had not yet been reviewed by outside experts before being published. At most journals, editors ask three or more experts—“peers,” in the parlance of science—to evaluate a paper and recommend for or against publishing it based on soundness of the study design, quality of the data, and strength of the analysis. Hansen and his co-authors posted a draft of the paper online to a journal where discussion happens before a more official publication.

That openness should be refreshing. It’s particularly relevant for science journalists. After all, many researchers won’t talk to the media about their work until it has been through peer review and published. That’s partly because that process ensures their findings have been given at least the once-over, and partly because journals, eager for their own scoops, generally frown on the practice. Hansen, however, posted his paper in Atmospheric Chemistry and Physics Discussions, which allows peer review—and review by the public—to happen in the open.

This on-the-fly approach to peer review, with its reliance on public input on new papers, “effectively resolves the dilemma between rapid scientific exchange and thorough quality assurance,” the editor of Atmospheric Chemistry and Physics Discussions recently wrote.

Slate’s Eric Holthaus wondered whether Hansen’s move “diminishes the study.” The “the lack of traditional peer review and the fact that this study’s results go far beyond what’s been previously published will likely bring increased scrutiny,” he wrote. Another climate blogger said the push for press before peer review was “weird.” It’s very uncommon for scientists to do things this way, which prompted Associated Press reporter Seth Borenstein to say, at a press conference about the work, “This sounds like more advocacy than science if the idea is to get it out before” a major climate conference later this year.

Indeed, it was just that sort of “peer review by press release” phenomenon that led Franz Ingelfinger, editor of the New England Journal of Medicine in the 1960s, to come up with what would become known as the Ingelfinger Rule: Any publicity of a study, particularly if it involved the media, would be grounds for rejection of an unpublished paper.

In the decades since, many scientists have followed that rule, even misinterpreting it to mean they can’t talk to journalists about their work at scientific meetings, much to the irritation of us and our reporter colleagues. Some researchers feel they have no choice: Rejection of their manuscripts will spell career ruin, given how central peer-reviewed papers are to decisions about tenure, grants, and promotion. (Although it has hardly stopped everyone from talking to the media before their work is peer-reviewed. Cold fusion, anyone?)

But is waiting for peer review to occur before announcing findings always the best idea? For one thing, we’ve seen time and again that peer review does not ensure robust or even legitimate findings. Remember that study of gay canvassers and how they could dramatically change the minds of people opposed to same-sex marriage? How about the quick and easy way to create stem cells in the lab? Or the finding that men were more likely to leave their wives if the women became ill? They all received massive media coverage, and afterward they were all retracted. One former editor of a major medical journal says peer review “has little if any benefit and lots of flaws.”

For another, peer review can slow down the process of science. As Ronald Vale, a researcher at the University of California–San Francisco, notes in a recent paper that he posted—perhaps appropriately—to a preprint server much like the forum that Hansen used, peer reviewers are asking for increasingly more data before accepting a paper, to the point that many students now devote their entire doctoral dissertation to gathering data for a single publication. That lag time is bad for science, Vale argues.

One potential solution, according to Vale, is “wide-spread adoption of electronic pre-prints,” which are publicly available before peer review. As he notes, they already exist, particularly in fields such as physics, whose main preprint server, called arXiv, recently celebrated its 20th anniversary. Physicists post manuscripts there, assuring that everyone knows who “got there first,” and then take comments and responses so they can improve the paper before submitting it to journals. Faculty of 1,000 Research, which promises open peer review for life science articles, is another such approach. Papers are published within a few days of submission, but only indexed in traditional databases once peers say it is “approved.”

Scientists in some fields, particularly biology, have been slow to adopt preprint servers. If you publish your data there, you are risking being scooped by other labs that could rush a similar study into a journal faster, and that could mean grants—or patents—going to the competition. So it’s clear that the widespread embrace of Hansen’s approach will require some new thinking about incentives in science, something that is long overdue anyway. And traditional journals will have to learn to share, relaxing their fierce grip on the flow of scientific information. Perhaps they will come up with ways to make money that don’t involve being first.

Many science reporters will need to change, too. Every week, journals spoonfeed us embargoed information about upcoming papers and try to get us to write about them, which brings attention, prestige, and potentially advertising or subscription revenue—and citations—to those journals. We reporters will have to figure out a way to find great stories without relying so heavily on these tips. We’ve already seen that reporters can do a better job of peer review than journals when they have the time, inclination, and expertise. Getting out of the habit of reporting on journals’ studies of the week should also mean fewer “coffee is good for you, coffee will kill you” whipsaws that report the latest incremental advance as the last word on a subject.

Perhaps we’re being somewhat optimistic here, but we hope that reporters freed from the tyranny of the Ingelfinger Rule will stop worrying about being scooped on the latest study that everyone else has anyway, and do deeper dives into what’s really going on in science. That, of course, will also require changes to how we’re incentivized. When those papers do appear, reporters will not treat them as gospel. Instead, we hope they will give them the necessary context to explain they’re just a small piece of a huge puzzle—or not cover them at all.

We should be clear: Peer review is still important. The fact is that we never see some of the real bloopers, because peer reviewers confidentially revealed them to the authors before they were published in the scientific literature. Some of those problematic papers are fixed and then published, while some never see the light of day.

But recent history has demonstrated that traditional peer review, which happens behind closed doors, just isn’t enough anymore—if indeed it ever was. The growth of preprint servers should only improve upon peer review and make it more efficient. Embracing more of a Silicon Valley–style “fail fast, fail often” approach will help science work the way everyone knows it could. Increased scrutiny? That’s a good thing.