A Review of Peer Review

I think absolute transperancy is essential. We had a paper rejected because (we found out later) a competitor was one of the reviewers, and we had to submit it to a lesser journal. The mantra should be "if you can't flaw the science, you must allow it to be published, even (and especially) if you do not agree with it.

Dior,

I hear you. I'm going through reviewer hell right now. But in my case it seems to be the outsider vs. the field syndrome.

I think it should be along the lines of Current Anthropology's "Star Treatment" where articles are critiqued by other scientists (who are not anonymous) and then the author responds to the criticism. Take more space, but that's what the internet is for - they could publish the article like normal and the "peer review" would be online...

I am outside the peer-review world. My world, a contractor-government-program world, has lots of technical and scientific publication, mainly in the form of reports and briefings, essentially none of which is peer reviewed. As a result, there is a tremendous amount of garbage that is presented both to the community and to the government sponsors,and there is no good way for someone not techically competent in the area to tell the difference. A great deal is self-serving, but some comes simply from lack of technical oversight by competent scientists (I was about to say pure stupidity, but that would be ungenerous.) Based on my observations, on the whole I have to say the peer review process works. It might work ponderously, unfairly and even selfishly at times, and credit might not be given where it is due, but the end result is pretty decent. Especially compared to what it would be without peer review.

A two-tier review process - panel review followed by public review:

Panel review:

1) Papers given to reviewers without the rersearchers' names appended for review, but with an option to see the names and make added comments afterwards (to allow scope for reviewers to say if they know that a lab has a good or bad reputation for accurately reporting its research, for example).
2) Reviewers names to be published with the paper (though reviewers should not be identified to researchers in advance, lest that lead to attempts at bribery or other forms of corruption).
3) All papers which pass panel review to be accepted for online publication and public review.

Online review:

Something like as being trialled by Nature, above. Since these papers will all have passed an initial peer review, the public review can only add top the rigorousness of the review process.

Only papers which survive both review processes to be accepted for full journal endorsement. Journal endorsement to be independent of print publication: this should help ensure that good research which, for example, fails to show anything exciting enough to merit print publication does not suffer (file-drawer syndrome).

Double blind peer review sounds good, but I wonder if it is possible. I don't know what it's like in other fields, but in Economics it's pretty easy to Google a couple of lines from the introduction and get the working paper version of the article. So, we have journals with double blind review, but I'd be surprised if reviewers honestly didn't know who the author was.

I completely agree with not providing the names of the authors to the reviewers. This will help the process to be more transparent.

Now regarding Nature's proposal for an open peer review, this sounds strange. If I submit a paper and it becomes available to anyone, in a way it is like publishing it without peer review, i.e. the work becomes public. However it is even worst because the paper can end up not being accepted and all the data has been made public already.

I think it is positive to get the opinion of a broader audience when deciding on publishing or not a paper, but not in a public way. Why not open the comments to a broader, but close community? Almost like increasing the number of reviewers, and creating a dynamic platform for exchanging comments between authors and reviewers, like a blog.
But again, this would work better if both authors and reviewers identity remained confidential.

Regarding Dior's comment, I don't think the system proposed by Nature's would prevent situations like the one he described. We cannot accept that a comment from a competitor is not valid just because he is your competitor. Also I can image people e-mailing their "friends" the link for submitting comments once the paper is being publicly reviewed.

If I submit a paper and it becomes available to anyone, in a way it is like publishing it without peer review, i.e. the work becomes public. However it is even worst because the paper can end up not being accepted and all the data has been made public already

I'm with evil gomez on this one - I don't see how opening up the manuscripts to the public at large will work. I certanily wouldn't choose to submit my paper that way.

I'm interested to see what happens

Having been subjected to several needlessly cruel "peer reviews", one that also included an ad hominem attack, I would like the identities of the reviewers to be made public. On one hand, I did figure out who wrote the nasty reviews, so I know to avoid them in the future (and believe me, I WILL AVOID THEM!), but on the other hand, I think reviewers will be more civilized if they know their names will be associated with their reviews. Or i'd like to think so anyway. (I've been warned that a certain number of my colleagues are just plain old assholes with no excuse for their bad behavior other than the desire to protect the old boyz network from interlopers.)

I think I agree with evil gomez. I like the idea of the "open" peer-review but I think the following could help. Following an initial round with editors to make sure the paper has merit, make the abstract available for all. If someone wants to read the entire submission, they must login and agree to review the paper (they get 1 week). Then, their review will be posted along with the paper, with their name, and the reviews and paper will be accessible to all who follow. Let the process repeat over 1 month. This would allow editors to learn a few things. 1) How much was the article accessed and how many agreed to review -- ostensibly a measure of interest for the audience. 2) Are there common comments from the reviewers -- a measure of serious problems with the experiments or discussion. 3) Are there comments that a number of reviewers did not agree with -- likely not real problems with the paper. After the period of 1 month and 1 week (for the people that agreed to review at the last moment) the editors take everything together and make a decision about what to do. Revisions could be handled by the editors or by some of the volunteer reviewers. The authors could have access to the whole thing along the way and maybe get a head start on something they didn't think of.

There is a big premium on getting the best info and getting it fast. While it would require some more work to volunteer for review, the reviewer would be getting some info early that they are obviously interested in. Lets face it, we all essentially do peer-review on papers we are really interested in, so why not write it down and make it part of the record.

One more note, this would obviously be nearly impossible for low teir journals. Just make the whole thing transparent there, either use no names, or, preferably, use all the names.

Has anyone seen how Biology Direct works? Check out http://www.biology-direct.com/ it looks like another interesting model.

The open-review process sounds very similar to the role that the ArXiv preprint often plays now in certain fields of physics.

Thanks for the posting, Alex. The comments are very interesting. I'll pass them on to my colleagues at Nature.
Please do comment on the articles in the debate - there will be 5 new ones a week through june, each of them links to the peer-review debate blog at Nature.
Thanks again, and please do come over, everyone. We at Nature want to know what scientists think and want. Now is your chance.

Peer review and critical evaluation is essential to science.

But the current system has it backwards and thus is anti-scientific, anti-democratic, a mode of censorship, cannot discriminate the ethical from unethical, fairness from arbitrariness, and is wasteful of resources.

If publication space is limited, simply exercise a lottery and reduce the waste. And limit the number of tickets (manuscripts) one can throw in, say one per quarter, annually, or per career!

If one really believes in the scientific approach, why not let the global peer review system of reproducibility, utility, self-correction and evaluation of investigator reputation go to work after publication, not before?

As opposed to the pre-judgment and censorship by a few with limited expertise, knowledge and time who arbitrarily decide what gets even to be evaluated by the whole?

The best judge of one's work is the one who has done the work, not someone who has not. Someone who has done or is doing the exact work at the exact time is by human nature going to be biased.

In the end, if you publish poop, you will be judged as a pooper, and that is the real value of peer review.

Polly A.

[Despite everything, I believe that people are really good at heart--Anne Frank]

A blind review, where the authors names are not shown, is a good start. It does two things.....1) reduce the possibility of conflicts of interest (of various sorts) and 2) evaluate the work more on its own merit and less on "group reputation" and suchlike. Makes it a little easier for young, less established investigators making it in to some journals.

Also.....nature's new "open review" policy might be a good idea.....

These reforms really can be started off with "top tier" journals.......and then trickle down to lower journals..

Regarding making the data available to public before publication. If the fear of doing it is that of losing priority i.e. the possibility of someone stealing the idea and publish it before you, perhaps that is where having something similar to arXiv may help. If the preprint is available with the date of submission, you will have a proof that you did it first if not the prestige of publishing it in Nature. I would like to hear opinions by physicists about their experience.

Journals have two roles; one is to select the papers that are worthy of publications and the other is to made them available to public by actually publishing them. But I wonder if these processes have to be inseparable. We can already get a lot of data online without actually reading the journals. I think what Nature is doing is interesting as a new way to select papers. But can't fields outside of physics also have a new way to make the papers publicly available?