I had a weird experience dealing with journals and peer review a little while ago. Recent discussions of the CRU e-mail hack (especially Janet’s) has made me think more about it, and wonder about how the scientific community ought to think about expertise when it comes to peer review.
A little while ago, I was asked to be a reviewer for a journal article. That’s a more common experience for people at research universities than for someone like me, but it’s still something that’s part of my job. I turned down the request because I didn’t feel qualified to review the paper. That wouldn’t have been weird, except that I couldn’t figure out why the editors would have chosen me, out of all the structural geologists in the world, to ask to be a reviewer. I mean, I had written a blog post about a related paper, but…
Did a journal ask me to be a peer reviewer because I had written a blog post about a related piece of research?
The thought was horrifying enough before the blogosphere started discussing the CRU e-mails. But in these discussions of whether climate researchers were trying to unethically interfere with the process of peer review, it may be worth discussing how reviewers are found. (My experience is relevant because it was with a journal that publishes a broad range of geoscience research, including climate change.)
I think about my expertise (and lack thereof) a lot, in part because my job involves developing expertise in other people (my undergrads). I may have a Ph.D. in science, I may teach at the college level, I may go to scientific conferences, I may have published my own work in peer-reviewed journals. But that doesn’t mean I’m an expert in most things. In fact, as I was thinking about the comments on last week’s post about getting undergrads to critically read the literature, I started to categorize the way I think about scientific papers:
Level 1 – I have no clue what the abstract says, even in journals like Science or Nature (which supposedly publish papers that are aimed at a broad audience). I get my information about these subjects from science journalists or other popular writers, not directly from research articles. Anything dealing with a genomes or quantum mechanics falls into these categories for me.
Level 2 – I can understand the point of the abstract, and the basic reasoning of the paper makes sense. I probably don’t know much about the methods used, but I’ve seen other papers that have used them, or they’re based on concepts I learned in college, so they make sense to me. Most geoscience papers (including climate change work) fall into this category for me.
Level 3 – I understand the paper (including methods and context) well enough to explain it to a less-experienced audience (like a class of undergraduates). These are the types of papers that I feel comfortable blogging about. They include papers in structural geology, tectonics, metamorphic petrology, regional geology of places where I’ve worked, and some mineralogy, igneous petrology and earthquake studies. (I’ve taught all those topics in courses for geology majors.)
Level 4 – I’ve got enough expertise to see problems with the methods or with the conclusions. That means I’ve got experience with the topic myself – maybe I’ve done work on those rocks, and I know things that the authors neglect to mention. Maybe I’ve got experience teaching a course discussed in a pedagogy paper. Maybe I’ve used a technique enough to recognize when another scientist is misusing it, or misinterpreting the results. These are the papers that I would be qualified to review.
I hope that undergrads come out of my junior-level courses able to make sense of the basic reasoning of papers in the field (level 2), and I want senior thesis students to understand work related to their work well enough to explain it to other undergraduates (level 3). So one of the goals of my junior-level writing class is to push students towards a higher level of expertise in their chosen sub-field (which may not be mine, and yes, that makes the class especially challenging to teach).
I don’t expect my undergrads to become qualified to review research articles, however, and I don’t think journal editors would consider them qualified. And that brings me back to my concern. Why would an editor think I was qualified to review a paper when my C.V. (which is online, and which includes a list of all the papers I’ve ever published) doesn’t indicate any research expertise in the subject? Did they really judge my expertise based on my blogging?
I turned down the request. But there are plenty of bloggers who write about scientific topics who might be delighted to participate in peer review. Some of the people who don’t think humans affect climate, for instance. Would they give a positive review to a paper whose methods they didn’t understand, because they agreed with the conclusions?
If I were an editor, I would find it difficult to figure out where to draw the line on expertise. The geosciences are filled with people who have switched specialties throughout their careers – climate scientists with degrees in Applied Math, for instance, or planetary geologists who used to do field work on Hawaii or Antarctica. And even reviewers with appropriate expertise can be unfairly harsh when a paper disagrees with ideas that they like (or easy-going when a paper supports their view). And what’s an editor to do when faced with a paper that is genuinely ground-breaking – where experience with the topic might not exist? (Actually, I’ve got a mental list of Big Thinkers who would probably be fair and insightful reviewers of wild ideas. I bet editors do, as well.) So I’m somewhat sympathetic to the journals. But I’m also concerned that, given the politics and media noise around climate change, they may feel pressured to find sympathetic reviewers for mediocre-to-bad papers. (More pressured than if, say, an Expanding Earth paper* was submitted to the same journal.)
And if they’re using the blogosphere to find potential reviewers? Ack.
Is it ok to talk about these kinds of issues in the scientific community? Or is that an unreasonable interference in the process of peer review?
*Yes, there have been legitimate geoscientists who believed the Earth is expanding. The most famous, S. Warren Carey, is deceased, but he was still alive and publishing while I was in grad school (and received the GSA Structure/Tectonics division’s Career Contribution Award in 2000). He wasn’t publishing in highly respected journals any more, however. If there had been a lot of political attention given to people who think subduction has no evidence to support it, would Carey’s late-career work have been published in mainstream journals?