Classic Edition: Hey, Ref!

Having made reference to the referee system in my post about a paper being accepted, this seems like a good point to dust off an old post about the peer review system in physics.

Like many of the other Classic Edition posts I've put up here, this one dates from July of 2002. Apparently, I wrote a lot of stuff about physics in July of 2002.

Anyway, the original text is below the fold, if you'd like a look inside the sausage factory that is the scientific publishing process...

I've just been asked to referee a journal article, for the fourth time in the last seven months. Since "peer review" is often cited as the cornerstone of modern scientific research, and since most people probably don't have a clear idea of what's involved, and since blogging about it is a convenient way to procrastinate my way out of actually reading the paper in question, I'll talk about the peer review process a little bit.

The basic idea, found in any philosophy of science textbook, review of Stephen Wolfram's new book, or rant against "Intelligent Design" theory is that science advances toward the truth through a process of tests and verification. New results by a particular research group are submitted to the larger scientific community, described in enough detail for other researchers to be able to check the validity of the work and attempt to duplicate the results. If new experimental results are repeatable, or new theoretical predictions are confirmed by experiment, those results will gain acceptance, and form the basis for the next incremental step forward. Peer review, the initial vetting of submitted papers by qualified scientists which takes place before publication of a journal article, is one of the cornerstones of this process.

The reality of the process is, of course, a little messier. The way the process works in practice, from the author's point of view, is that you pour sweat and blood and tears into crafting a concise but complete description of your work, which you ship off to a journal, who sends it out to some anonymous reviewers (usually two of them), who spend the next six weeks ignoring the paper and the badgering notes sent to them by the journal editors, then callously rip your work to shreds and insist that you make major changes that push the paper over the journal's length limits, leading to a big fight with the editors.

From the referee's point of view, of course, this looks completely different. As a referee, the way the system works is that the journals wait until you're out of town for some rest and relaxation, then email you a note saying "We'd like you to review this paper. If you don't respond in the next fifteen minutes, we'll assume you agree, and send you the article." By the time you get back, you're implicitly committed to reading some inscrutable gibberish from a field you've never worked in, which was originally written in Chinese before being translated into Portugese, then Pashto, then Finnish, and finally into English via the Babel Fish. You're teaching four classes, writing a paper for another journal, and you've been called for jury duty on a capital murder case and sequestered, so you put off reading the paper for a little while, only to get a barrage of nasty letters asking when they can expect your report. Then, when you do submit a report ("Please, please, please, ask a native speaker of English to proofread this for you."), the authors have the temerity to take issue with your carefully constructed comments.

OK, so maybe that's slightly exaggerated.

The way it works is that a paper presenting new results is sent to a journal, where the editors check it quickly for general suitability (i.e, they make sure that it deals with physics rather than political sciecne), then send it out to two referees, chosen from a pool of people whose names, addresses, and areas of expertise are kept on file at the journal (these, in turn, are drawn from the people who've published papers in that journal in the past). The referees are asked to read the paper, and judge the quality of the science: is it original work, is it factually correct (or at least plausibly so), does it provide all the necessary details, does it generally increase the store of human knowledge? Referees also make recommendations based on the form of the paper-- is the writing clear and are the figures comprehensible?-- and depending on the journal may be asked to make judgements on more nebulous criteria like "importance in its field" or "general interest." Based on the comments of the referees (who remain anonymous to the authors), the paper can be accepted immediately, rejected outright, or sent back to the authors, with the comments attached, for whatever revisions are needed to make it acceptable.

As indicated by the exaggerated descriptions above, this is a gigantic hassle for everyone. Papers in the more important journals (Science, Nature and Physical Review Letters) are subject to very strict length limits, so it's not always easy for the authors to make the changes, and scientists are a prickly bunch when it comes to outsiders criticizing their work, so referee comments can lead to big fights. On the referee's side, reading a paper closely enough to make the necessary comments is incredibly time-consuming-- it's not quite as bad as the "source cite" process student law journals use (mostly due to the lack of free student labor to check the fiddly little details of the formatting), but getting enough of a picture of a slightly foreign field to be able to place the paper in context and jdge its importance takes a bit of work (though it often turns out to be rewarding). Additionally, the length constraints tend to force a terse and jargon-laden writing style that can be difficult to decipher (but easy to parody), even for a trained professional. In general, authors submitting papers tend to regard the process as one of those pain in the ass things that you just have to get through by whatever means necessary (sort of like meeting the seemingly arbitrary formatting requirements most journals set), while people tapped to referee papers try to get out of it whenever possible, by passing the papers off to colleagues, post-docs, or grad students.

Hassle aside, though, it is a critical part of what makes things work. It's what separates the Intelligent Design crowd and the Time Cube guy from actual scientists, and the process does work. In the handful of papers I've published, one referee caught an embarassing typo in an equation that had somehow slipped past the five authors, while another asked a very good question which later came up during my PhD defense (not an event which really changed the history of science, I suppose, but I felt a lot better for knowing the answer beforehand...). One of the papers I've refereed had a sign error in the very first equation, which changed everything that followed (they didn't make it into print). This knowledge of the importance of the task is pretty much the only thing that keeps the system working-- referees aren't paid for their time, and it's very much a thankless job, but I try not to turn down requests to referee papers unless I'm really not qualified to comment on the work in question, just because somebody has to do it, and it's an important part of scientific citizenship (basically, it's the science equivalent of jury duty).

The biggest problem with the system is that the criteria for publication can become ridiculously variable. One referee's idea of "important and of general interest to the physics community" may be another's "uninteresting crap." I hardly ever read Physical Review Letters without thinking "who the hell thought this was important enough to publish here?" and many an author has suffered rejection from a major journal only to look in the next issue and find essentially identical work with a slightly different spin. There are some safeguards built into the system to prevent political crap (authors can suggest referees, and request that some people not be given the paper to review, and journals try to avoid sending papers to close colleagues), but accusations that so-and-so spiked a competitor's work as a referee are a staple of ugly physics gossip. I've had the good fortune to work in a field where there's relatively little riding on any individual publication, but I've heard horror stories from other fields about authors deliberately leaving out or falsifying critical information to protect a competitive advantage or patent bid (Derek Lowe would be the guy to ask about that).

Still, for all its ungainly aspects, it's a far cry from the extreme Kuhnist/ post-whateverist picture of science as "socially constructed." There are ugly controversies from time to time, and a little bit of "grade inflation" as the standards for importance and general interest seem to be slipping, but by and large the system works. Blatantly incorrect work gets weeded out, basically correct but badly written work gets whipped into slightly better shape, and Science, as they say, Marches On. The proof is all around us-- peer review is essential to the progress of science, and the progress of science has led to the dizzying array of technological conveniences that make everyday life bearable.

To paraphrase a famous quote about democracy, the current system is the worst system except for every other method that you can think of. It's inelegant and inconvenient, and the subject of frequent hand-wringing pieces about how everything's falling apart, but somehow or another it hangs together well enough to make everything else work.