You don’t have to look far to find mutterings about the peer review system, especially about the ways in which anonymous reviewers might hold up your paper or harm your career. On the other hand, there are plenty of champions of the status quo who argue that anonymous peer review is the essential mechanism by which reports of scientific findings are certified as scientific knowledge.
So how do scientists feel about anonymous peer review? A 2008 paper in Science and Engineering Ethics by David B. Resnik, Christina Guiterrez-Ford, and Shyamal Peddada, titled “Perceptions of Ethical Problems with Scientific Journal Peer Review: An Exploratory Study”, attempts to get a preliminary handle on that question. They write:
Although most scientists agree that ethical problems can occur in journal peer review, evidence has been anecdotal, consisting of personal accounts published in news stories, letters, or commentaries. In this article, we report the results of an exploratory survey of scientists’ perceptions of ethical problems with journal peer review. (306)
Specifically, Resnik et al. studied the perceptions of scientists at the National Institute of Environmental Health Sciences (NIEHS) in Research Triangle Park, North Carolina. They probed the perceptions of researchers, research staff, post-doctoral trainees and technicians using an anonymous survey distributed to at mandatory Responsible Conduct of Research (RCR) training sessions.
They distributed 556 surveys, of which 283 were returned. The researchers set aside the surveys returned by contractors, graduate students, people in non-research positions, respondents who didn’t indicate their position at NIEHS, and those who hadn’t published any papers, leaving 220 completed surveys for them to analyze.
Those 220 surveys left in the pool were from postdocs (94), principal investigators (38), staff scientists (55) and technicians (33) who together represented 22 different biomedical disciplines.
In addition to asking for demographic information (like age and level of education), the surveys asked respondents to answer the following questions:
1. Approximately how many articles have you published in peer-reviewed scientific or professional journals?
Have any of the following ever happened to you during the peer review process:
2. The review period took longer than 6 months.
3. The review period took longer than a year.
4. Comments from reviewers included personal attacks.
5. A reviewer was incompetent (i.e. he/she did not carefully read the article, was not familiar with the subject matter, or made mistakes of fact or reasoning in his/her review).
6. A reviewer was biased (i.e. didn’t give an article a fair hearing, prejudged it).
7. A reviewer breeched the conﬁdentiality of the article without your permission.
8. A reviewer used your ideas, data, or methods without your permission.
9. A reviewer delayed the review so that he/she could publish an article on the same topic.
10. A reviewer required you to include unnecessary references to his/her publication(s). (310)
For questions 2-10, the available responses were “yes,” “no,” and “don’t know”. In their analysis, the researchers focused on responses to questions 4 through 10.
Here’s the table in the paper that presents the aggregate data they collected:
Worth noting here is that some of the ethical breaches that we might consider most serious in peer review — using the veil of anonymity to steal ideas, data, or methods from the author whose work you are reviewing, using the power of the reviewer to slow down the publication of the work under review in order that the reviewer can get his or her work published first, or otherwise violating the confidentiality of peer review — rank fairly low in the percentage of respondents who say they have happened to them (less than 10%). Much larger numbers of respondents indicated that they thought a reviewer was incompetent (61.8%) or biased (50.5%). It is also interesting that none of questions 4 through 10 received only “no” or “don’t know” responses — each of them had at least 10 positive responses from among the 220 respondents in the final pool.
The researchers found some interesting patterns in the responses. Older respondents were more likely than younger ones to answer “yes” to question 4 (about whether comments from reviewers included personal attacks). Researchers with more published papers were more likely to report incompetent reviewers or biased review. As well, the respondents who were PIs and postdocs tended to identify more negative experiences of peer review than did the respondents who were technicians and staff scientists. (The researchers suggest that this may be because postdocs and PIs are more likely to be first authors on papers — and that reviewers’ critiques may feel more personal to them as a result.) Postdocs, too, gave the most reports of incompetent reviewers.
Now, there are some clear limits to the conclusions than can be drawn from the results of this study. The research focused on scientists at a single research institution, rather than respondents from different institutions. Potentially, the culture of NIEHS might not be representative of the community of biomedical researchers as a whole (even in the U.S.). Further, the surveys asked researchers whether they had ever had the particular experiences of interest with peer reviewers, but did not ask for information on the frequency of these experiences.
Perhaps most importantly, the researchers were measuring respondents’ perceptions of problems they have encountered with peer review, but there was no effort to establish whether the problems they reported has actually happened — whether the reviewers we actually biased, or incompetent, or swiping information from the manuscripts they are reviewing.
To this concern, Resnik et al. respond:
[D]ocumenting that scientists perceive that there are ethical problems with journal peer review can be an important ﬁnding in its own right, because a scientist may change his/her behavior in response to what he/she perceives to be a problem. A researcher who is concerned that his/her ideas will be stolen, for example, may not disclose all the information that is needed to repeat his/her experiments. A researcher who is concerned that a reviewer is incompetent or biased may choose to ignore the reviewer’s comments rather than address the concerns (which may in fact be valid), especially if they involve further time and effort in the laboratory. (308-309)
The effects of researchers’ perceptions about what peer reviewers are doing (or might be doing) on their own scientific conduct is, of course, a question that might merit further research.
Indeed, Resnik et al. suggest that even in the absence of this further research, the results they report may warrant action:
As mentioned earlier, commentators have made various proposals for reforming the peer review procedures used by scientiﬁc journals. Our survey provides support for these reforms, since it demonstrates that biomedical researchers perceive that there are some problems with the integrity and quality of peer review. Since incompetence and bias were by far the most frequent problems respondents claimed to have experienced during peer review, journals, research institutions, and scientiﬁc societies should consider ways of dealing with these problems, such as providing additional education and training for reviewers on the scientiﬁc and ethical standards for peer review, requiring reviewers to disclose conﬂicts of interest, and paying more careful attention to the selection of reviewers. A more radical reform, such as open review, may be needed to counteract the perceptions of the most serious violations of peer review ethics, such as breach of conﬁdentiality and use of data, methods, or ideas without permission. (309)
I welcome your discussion in the comments on the question of whether significant perception of problems with peer review on the part of the scientists whose careers depend on it (at least to some extent) is reason enough to examine the status quo and reform peer review.
Resnik, D., Gutierrez-Ford, C., & Peddada, S. (2008). Perceptions of Ethical Problems with Scientific Journal Peer Review: An Exploratory Study Science and Engineering Ethics, 14 (3), 305-310 DOI: 10.1007/s11948-008-9059-4