“From my close observation of writers… they fall into two groups: one, those who bleed copiously and visibly at any bad review, and two, those who bleed copiously and secretly at any bad review.” -Isaac Asimov
You’d never know it unless you were one of about six people in the entire world, but today is a landmark anniversary for me. Three years ago, I was on summer break from teaching at my local college, when I got an email from the Royal Astronomical Society in England.
The UK-based society at the forefront of astronomy since 1820, their records actually go all the way back to the founding of calculus! Their journal, Monthly Notices, was the location of my very first published scientific paper, but that wasn’t why they were contacting me.
They had a controversial paper under peer review — something that fell under my area of expertise — and they asked me if I would step in as an expert referee.
It was a little bit of a surprise to me; I was young, I had only published a handful (or two) of papers, and I’d only refereed maybe three other papers before. I hardly considered myself to be an expert. And yet, impostor syndrome aside, I knew (and know) that I legitimately am an expert in a few particular sub-fields of astrophysics and cosmology.
For those of you just starting out in a scientific career, there’s a very good chance that if you’ve ever published a paper in a peer reviewed journal, you’ll be asked to referee a paper for that journal down the road. This is a vital part of the scientific process, and it’s what allows experts to separate solid papers that deserve publication, like this one, from highly flawed papers that need significant work before they’d be suitable for it.
But no one ever tells you how to do it, not really.
Before I get into what I consider the four jobs of you, as a referee, let me tell you a few things I wished that someone had told me (and, especially, had told many of my former referees) to keep in mind:
- What you’re reading likely represents the culmination of months or even years of careful planning, execution, analysis, and hard work on the part of the researcher(s) you’re reviewing. Respect that.
- Regardless of whether you like the viewpoint expressed in their work, their claims should be addressed in an objective, unbiased manner. This is not to say that you must be unbiased or without opinion; you will have your professional opinions, no doubt, and you should. But you must be aware of any biases you have, and do your best to address their work solely on its own merits.
- Finally, and most importantly, there are going to be flaws that you find in the text, but it is not your sole job to point them out, and it is certainly not your job to belittle the authors for their errors. (No one deserves that.) Be sure to point out what the positives are, what important valid points were made, and what the strengths of the paper are as well.
That’s all the basic advice. But what about the meat-and-potatoes of peer review: the actual reviewing of the content of the paper? That’s where you — with all your expertise — come in with your four jobs. Here they are.
1.) To verify that the introductory section(s) adequately sets up, explains, and places their work into its appropriate historical and scientific context.
No science is ever done in a vacuum. Even your iconic image of Einstein as a “lone genius” is wildly inaccurate; without the centuries of scientific foundation that work is built upon and grown out of, none of the modern science undertaken today would be possible. There are many perspectives one can take, but the reality is that on most issues, there is a scientific consensus, there are dissenting opinions, there is wiggle-room, and there are important nuances that need to be addressed.
Most importantly, perhaps, there are points of contention surrounding any issue, and presumably this paper is attempting to take a step towards either strengthening or weakening one (or more) of these points. It’s important to give the authors a lot of leeway here; this is not the place where new material is introduced. But it is the place to make sure that everyone who deserves credit is given credit for their prior work, and to ensure that the authors are very clear about what the rest of their paper is going to entail.
2.) What are the methods used to obtain and analyze the data presented in this paper? Are they sound? Are there questions about their validity?
The (rough) sections of a scientific paper are the abstract, intro, materials/methods/experiments/observations (depending on the field), results, and conclusions/analysis/discussion, followed by the references. If there are large flaws in the raw data that is gathered, then the results and conclusions are meaningless! If I’m trying to draw conclusions about a position down to angstrom accuracy, but I’m only using a device that can measure distances as finely as 10 nanometers, there’s no way that should pass peer review!
The job of the referee here is to make sure that — to the best of your ability — that every source of error and uncertainty is quantified and understood as well as possible. (That’s why this paper, for example, is absolutely worthless.) To make sure that the appropriate scientific controls are in place. It’s also your job to determine whether there’s any suspicious (or possibly fraudulent) methodology at work.
Do your best; intentional deception is the toughest of these to catch, but you can do a great service to the scientific community if you keep a scientist (or a team of scientists) from fooling themselves.
3.) Results: are they clear? Has anything been omitted (or included) that shouldn’t have been? Is all the data that needs to be there actually there? And does it contain and show all the information that it was designed to?
This is often the easiest section to referee. If you know how something’s supposed to work, and they took their data exactly as they said they would (or did their calculations, etc.), are these reasonable results? Are their calculations, curve-fits, etc., repeatable? Are they using well-known, well-understood, easy-enough-to-follow analysis methods?
Basically, based on their methods section, you should be confident that you’d be able to reproduce their results if you did this work, again, yourself. Unless there’s a calculational error, a glaring omission, or some other gross mistake, there are rarely catastrophic problems in this section.
And most importantly…
4.) Can the conclusions reached be justified based on both the combined, pre-existing breadth of knowledge of the field and the work done in this paper?
There was a reason that everything in the introduction was necessary, and that the authors needed to be so careful about the work they did up until this point. It’s all led up to this: the conclusions that can now be drawn in light of this paper’s results, synthesized with everything else we know.
This, honestly, is where I — for my own opinion — think that peer reviewers need to be more diligent. Not because we need more uniform conclusions, but because we need fewer papers that draw grandiose conclusion based on a cherry-picking of historical and scientific facts. Authors should be free to express their professional opinions and assign their own weights to various findings, even if that supports a contrarian position. But authors should also be held accountable to at least mention the scientific findings that support the other side, particularly if the other side:
- is the mainstream/established consensus,
- contains well-known facts and arguments that the authors choose to overlook,
- or has recently published something challenging this point-of-view.
Science is a process and a discussion, and peer review is both a right and a responsibility. It’s what’s necessary to prevent “science” like the following.
For what it’s worth, I wound up rejecting the paper I was called in to be the expert referee on, and I did so in particular because of the strong (and unreasonable) conclusions that it drew based on the lackluster evidence it cited. But I also pointed out which aspects of the paper were strongest, and what a version of the paper that would be acceptable (and scientifically interesting and valuable) would look like. When there is quality work that’s been done, you should never belittle or berate an author simply on account of the portions of a paper that need changing.
But the peer review process is also our first line-of-defense against outrageous and unreasonable claims; it’s what allows us to both discover and challenge scientific truths, as well as to present new possibilities.