Manuscript review and the limits of your expertise.

At his lounge, the Lab Lemming poses an excellent hypothetical question about manuscript review:

Suppose you are reviewing a paper. Also assume, that like most papers these days, that it has multiple authors, each of whom applies his expertise to the problem at hand. And finally, assume that you are an expert in some, but not all of the fields used to solve the particular problem being reported in this paper.

What do you do if one of the key points in the paper that is not your area of expertise seems fishy. For example, if the paper is on your field area, what if some of the lab results seem fishy. Or if you are an analyst, what if the experimental setup seems odd.

Assuming that you are a successful researcher, you probably have long-time collaborators who are experts in these fields. So, what is the best way of accessing their expertise, given that some sort of confidence generally surrounds papers in review.

Let's break down some of the relevant interests at work here:

Scientists, collectively, want peer review to function as a quality control mechanism to ensure that papers that get published in journals meet scientific standards -- claims are supported with data, methodologies are spelled out explicitly enough that other researchers can figure out how the data were collected, analyzed, and so forth, some account is taken of how the reported findings fit into the existing body of knowledge in this field, etc. If there's a problem with the paper, whether it's an oversight, a result of sloppiness or an honest error, or actual flim-flam, it's better for the tribe of science that the problem is identified -- and addressed -- prior to publication.

An individual scientist acting as a peer reviewer has an interest, therefore, in being as thorough as possible in screening manuscripts and flagging potential problems. Often this will involve drawing on one's own area of expertise, but sometimes, as noted in the scenario above, one's spidey-sense will be set to tingling by part of the paper that is outside one's immediate expertise.

In the interests of being thorough and making the best informed evaluation possible, one might want to turn to a colleague with the relevant expertise for advice on the troubling bit of the paper. However, generally the peer review of manuscripts is explicitly confidential -- in agreeing to review the manuscript, you are promising the journal editor (and by extension the authors of the paper) not to disclose the contents of the manuscript to anyone else. Among other things, this is supposed to prevent the leakage of information that might help competing researchers from scooping the authors of the manuscript before it has been published.

Now, there might be ways to ask experts about the bits of the manuscript that bug you by couching your questions in hypothetical terms. However, especially in the circumstances where your expert friends are working in closely related areas, the chances are good that they may be actual or potential competitors of the manuscript authors -- and that they may be able to figure out, from your hypothetical description, just what these authors are up to. This is to say, your efforts to avoid a meaningful violation of confidentiality may fail -- and because you aren't up on the stuff you want to consult your expert friends about, you may be ill equipped to get meaningful feedback without such a violation.

Here are the alternative options that occur to me:

1. Hit the literature. You could ask your expert friends for recommendations of papers that describe the kind of experimental set up, or data analysis strategy, or whatever, that is bugging you in the manuscript you're reviewing. From those recommendations, you can use the references to collect more papers. Perusing a reasonable handful of papers that deal with the issue you're pondering may give you a better sense of whether what's presented in the manuscript is normal or weird. If it still strikes you as weird, you can flag it as such in your review (listing the articles you're using as your "baseline" for normal).

2. Note your concerns, and the limits of your expertise. You can communicate your concerns as part of your formal review of the manuscript, or you can do it in an email to the journal editor in advance of your formal review. Either way, it's not inappropriate to pass on to the journal editor names and email address of the experts you would turn to for advice. The editor can then investigate whether these experts would be appropriate additional reviewers.

When the peer review process works well, the authors of the manuscript will address the concerns raised in the reviews (including yours). Flagging a piece of the paper that concern you -- even if you don't feel you have the basis to fully evaluate whether your concerns are well-grounded -- ought to open up a discussion and present an opportunity for the authors to provide more explanation. If you're open with the journal editor about the limits of your own expertise, the journal editor can take responsibility for drawing on the advice of additional experts in vetting the manuscript.

If and when the manuscript is published, you might shoot an email to the expert friend you were tempted to consult and have a chat about the published paper. Use it as an opportunity for your friend to educate you about the details of his or her expertise as applied to the research described in the paper. If he or she sees a glaring problem in the paper, a letter to the journal editor may be in order.

More like this

There's an article in yesterday's New York Times about doubts the public is having about the goodness of scientific publications as they learn more about what the peer-review system does, and does not, involve. It's worth a read, if only to illuminate what non-scientists seem to have assumed went…
An important part of the practice of science is not just the creation of knowledge but also the transmission of that knowledge. Knowledge that's stuck in your head or lab notebooks doesn't do anyone else any good. So, scientists are supposed to spread the knowledge through means such as peer…
Over at the Nature blogs, they're soliciting comments and opinions about open peer review: The goal of any change in the peer review system must be to improve the quality of review, where quality is determined by two distinct functions: filtering manuscripts for publication in a given journal; and…
Chad Orzel takes a commenter to task for fetishizing peer review: Saying that only peer-reviewed articles (or peer-reviewable articles) count as science only reinforces the already pervasive notion that science is something beyond the reach of "normal" people. In essence, it's saying that only…

The conference I review(ed) for actually allows you to phone a friend. There's even a blank on the review form to put their name in. I guess it depends on the field and that it's a journal article

You are right; the review manuscript is confidential and should be treated as such. One possibility (which I have done) is to submit my review saying "here's my opinion on A and B, but I can't evaluate C or D; you should get a review by someone with expertise in those areas also". Another, when there is a nearby expert, is to fire off an email to the editor and say "my colleague Dr. X is an expert in this area; may I get her opinion on the manuscript?" The editor will tell you yes or no, and then all confidentiality issues are taken care of.

By ecologist (not verified) on 11 Jul 2009 #permalink

There's a third option, which is the correct one.

3. Email the editor saying you would like to consult with Dr. X about an issue in the paper. I've never seen an editor turn down such a request. Pretty much they say "Sure, just make sure that Dr. X maintains confidentiality."

1. It's really the editor's job to hit the lit for people who can competently review the paper.

2. Email the editor and let him/her know that you are qualified to review certain parts of the paper but that another reviewer should be picked that address those gaps. If you have a name of a person in mind that you think would be great for the gaps, PLEASE let the editor know. When you turn in your review, make sure you note where your skipped parts were.

You can also ask the editor for permission to bring in a co-reviewer. Name the co-reviewer and explain clearly why you want to do this. Chances are good editors will agree to this.

If you can get yourself up to speed without an unreasonable amount of effort, I say option 1.

If not, I think the 3rd option already mentioned by a couple people is the right one: email the editor and ask for permission to consult a colleague. I suspect they'd grant it. If they don't, then I'd go with option 2. It may be that one of the other reviewers that they've already chosen has the requisite expertise.

---

Not necessarily a matter of ethics, but definitely of courtesy: If you go with option 2, I think you should email the editor in advance of your formal review to give them a chance to find additional reviewers without drawing out the review process too much.