One of the fundamental principles of modern science, as well as other academic pursuits, is peer review. By subjecting a submitted paper to evaluation by other scientists in the authors' field, the solid science advances at the expense of the not-so-good and the interesting and relevant prevails above the unoriginal. In theory, of course. The effect is a growing body of scientific knowledge that, while still large and unwieldy, is at best authoritative and at the very least trustworthy and accurate. It's a kind of democratization of knowledge, at least in a narrow sense.
But, as in any democratic system, there are problems. Contrary to the fundamental ideals of a democracy it is inherently exclusive, although this makes sense considering the extensive background knowledge and experience needed to evaluate a scientific manuscript. Of course, individuals also have vested interests in the process as well. Although science strives for objectivity, who reviews your paper does matter, and if you step on the wrong toes by not citing the reviewer's own "definitive" papers on the subject or, even worse, refute them altogether, your findings might not get the peer review seal of approval.
Even the unimaginable can sometimes happen when results that are completely faked pass right through the peer review process and end up in a top journal. A case of this made headlines recently when a group led by scientist Hwang Woo Suk claimed in two widely-heralded papers published in Science that it had produced various lines of embryonic stem cells from cloned human embryos. In wake of the revelation that the results had been fabricated, a debate on the very nature of peer review ensued.
In that spirit, today's issue of Nature announced its own experiment in science--or democracy--by opening up the peer review process to all interested in participating and giving authors of submitting their papers to an open and public peer review process to take place online:
This publication champions the value of editorially driven or editorially selected content. But we are always keen to try new things, and we are now experimenting around the edges of that principle, to make the most of online interactivity.......Less certain is the outcome of a trial that we launch this week: a test of a particular type of open peer review. The trial is accompanied by a general online debate about peer review; see http://www.nature.com/nature/peerreview/index.html.
During the trial, which will last several months, Nature's traditional approach to peer review will continue: typically, we send selected submissions to two or three experts whose identities are kept confidential. We believe that this approach works well. Meanwhile, over the next few weeks, the web debate will explore other approaches, as well as the potential for online techniques to unpack the various functions of conventional journals, the ethics of peer review, and more.
Our online trial opens up a parallel track of peer review for submitted papers for authors willing to go down that route. The traditional process will still be applied to all submissions selected for peer review. But we will also offer to post the submitted manuscript onto an open website. Anyone can then respond to it by posting online comments, provided they are willing to sign them. Once Nature's editors have received all the comments from their solicited confidential reviewers, the open website will cease to take comments, and all the opinions will be considered by the editors as well as the authors.
The details of the trial can be found in the online FAQ. Nature published additional articles on the subject here and has made available a forum and a collection of resources on the topic.
So, is this a good idea? It's hard to say at this point, although Nature seems to be doing the appropriate thing by approaching this experiment cautiously, adding open peer review as an optional preliminary step in the longer peer review process. And, although any peer review process by definition should only include the authors' peers (i.e. experts in the same or a similar field), this seems to compliment the recent push toward open access literature, which I've discussed at length previously.
Although it never comes close enough to the ideal in practice, science should be an open enterprise, characterized by the free exchange of information between professionals dedicated to expanding our body of scientific knowledge. When one takes into consideration the fact that science is largely a taxpayer funded enterprise, a new ideal emerges, one where free open access to the scientific literature and the peer review process allow information to flow uninhibited within the scientific community and enable interested citizens to observe their financial contributions in action.
Putting the grand ideals aside, though, and in light of recent lapses, a few extra pairs of watchful eyes in the process couldn't hurt.
Update 9 Jun 2006 10:58 GMT: The first article has now been submitted on the Nature site for open peer review. The title of the manuscript is "H5N1 Avian Influenza Virus Evolution: Twice faster than Old Viruses", and it can be found here.
Update 9 Jun 2006 16:44 GMT: ...and another one. It looks like they're on a roll now.
Cross-posted at http://scientificactivist.blogspot.com/.
- Log in to post comments
Of course, we can't forget Richard Sternberg's nasty little situation for getting ID its "first peer-reviewed article", now can we?
No system is perfect.
My one conern is that the more experts you bring in, the more negative reviews will become. At some point, the review process might begin to resemble a pirahna fest. Typically, the problem, as I see it, with peer review is that the reviewers have to trust the authors to a considerable degree. That allows sleezeballs to sneak false data in.
I'm pretty interested in Nature's experiment because it seems to strike a good balance between complete exclusivity and inclusivity. I'll certainly be following it to see how it turns out, although I'm not sure how they're going to evaluate its success.
I agree with Mike, but perhaps we shouldn't overrate the importance of the comments on the blog for review in terms of their publication impact. We should emphasize that the egregious errors and silly mistakes which sometimes accompany important papers can only be reduced by the openness and criticism afforded via the blog.
I think it's a good idea overall, so long as the blog doesn't serve as equivalent in importance when deciding whether or not to publish. The more criticism, the better. It may slow down the publication process a bit, but when those papers are published, their quality and streamlining will more than compensate.