I was on the way out the door for a vacation when the journal Nature published its much-anticipated report on the results of its open peer review experiment, but I did want to offer a few comments on the report, even if I’m arriving to the discussion a bit late.
Peer review, of course, is the gold standard for academic publishing. I believe one of the reasons for Cognitive Daily’s success is our clear delineation between reports on peer-reviewed research and commentary on news items reported in the popular press (you can always click on the Just the Research tab above to see only reports on peer-reviewed articles). We try to offer several articles each week covering peer-reviewed research, and these are by far our most popular offerings.
But the peer review process itself has also been the subject of criticism. Even after review by experts, some research has later been found to be fraudulent. Some scholars have wondered whether new lines of research may be repressed by the “old guard” in charge of the peer review process. It is in this context that Nature decided to try out a new form of peer review, an open process which allowed anyone to comment on manuscripts before publication. Now, a few months later, a team led by Philip Campbell reports on the results of the process:
We sent out a total of 1,369 papers for review during the trial period. The authors of 71 (or 5%) of these agreed to their papers being displayed for open comment. Of the displayed papers, 33 received no comments, while 38 (54%) received a total of 92 technical comments. Of these comments, 49 were to 8 papers. The remaining 30 papers had comments evenly distributed. The most commented-on paper received 10 comments (an evolution paper about post-mating sexual selection). There is no obvious time bias: the papers receiving most comments were evenly spread throughout the trial, and recent papers did not show any waning of interest.
The trial received a healthy volume of online traffic: an average of 5,600 html page views per week and about the same for RSS feeds. However, this reader interest did not convert into significant numbers of comments.
In a word, the results are disappointing. Only 5 percent of submitters agreed to the process, and only 38 out of the 71 papers presented for open review received any comments at all. Most of the comments on the substance of the papers were not judged to be helpful by the paper authors. For the most part, the only useful comments covered editorial concerns — presumably, these same issues could have been addressed during the copy editing phase.
I doubt the editors of Nature were surprised by these results. I’ve been an editor myself, and finding reviewers is one of the most onerous tasks an editor can undertake. I’ve spent endless tedious hours trying to coax potential reviewers, first to agree to write a review, and later to submit the work they promised. Greta is considered a “prompt” reviewer because she never turns her reviews in more than a week or two after her deadline. Many reviewers routinely turn in their reviews months after the deadline. Reviewing is a thankless job, and without incentives for reviewers, I doubt any open review process will ever gain much traction.
What kind of incentives would work? The most obvious would be career incentives: if work as a reviewer was rewarded with tenure and promotion, it would soon become one of the top priorities of any scholar. Unfortunately this revolutionary change in the glacial world of academia is about as likely as PZ Myers undergoing a religious conversion, so we probably will need to look elsewhere. Many journals already require authors to review articles as a condition of submitting articles for publication. Perhaps this sort of incentive could be adapted to an open review process. Even so, it would be difficult to administer. How would reviewers be evaluated? By authors? But then wouldn’t there be an incentive for reviewers to rubber-stamp articles for publication? For now, it appears that the peer review process as it stands might be the lesser evil.
In the future, I think there is a possibility that some combination of blogs and wikis might become a partial replacement for the traditional peer review process, but don’t expect this sort of change overnight either. Only when contributing to these resources becomes part of the tenure rewards system are they likely to become important factors in the world of academic publishing.