You often hear scientists and philosophers of science talk about the peer-review process as if it's some epistemological magic trick, as if it automatically sifts through the mass of submitted articles and finds The Truth. Of course, if you've ever been through the peer-review process you know that there's nothing magical about it. The process can be just as arbitrary, unfair, illogical, and irrational as, well, everything else that humans do. My own encounter with peer-review left me acutely aware that scientists have big egos, and that big egos don't like contradictory data. That said, the process did lead to a better published article, with more conclusive data. So it worked. But my lingering impression was that it worked for the wrong reasons.
Despite the essential role played by peer-review in the scientific process, it's astonishing how little we actually think about it. If I were a philosopher of science, I'd stop trying to solve the demarcation problem and try to figure out practical ways to improve the system governing the acceptance of papers. How can we make our peers better critics? Is anonymity a necessary evil? Or does it encourage unfair comments?
That said, Terry McDermott's wonderful series on Gary Lynch is one of those rare works of scientific journalism that actually explores the fraught nature of the peer-review process. It's a rare peek inside the sausage factory of science:
When a paper is submitted to a scientific journal, the journal editors send it for review to panels of scientists. Peer review is the backbone of contemporary scientific legitimacy and lauded by everyone involved. It is also an opportunity for mischief and misunderstanding.Lynch's history of antagonizing his peers sometimes made peer review more a gantlet than a critique. Richard Thompson of USC, a renowned neuropsychologist, said he had more than once nominated Lynch to membership in the prestigious National Academy of Sciences, but was told by other members Lynch would not be elected so long as they lived.
"There's a reason for his paranoia. There are a lot of people out there who don't like him. Gary doesn't suffer fools gladly," Thompson said, then paused for a moment. He chuckled and said: "And there are a lot of fools in the world."
The reviews on Kramar's paper seemed not to even acknowledge its main point -- that the lab had for the first time demonstrated the physical reorganization of cells that occurred in the final stage of long-term potentiation, or LTP, which Lynch believed was the biochemical process underlying memory.
One reviewer, in recommending against publication, complained that the scientists had only looked at a specific set of synapses, which was inexplicable as criticism. They looked there because that's where they were doing the experiment, that was where the condition they were examining existed. It was as if a traffic engineer, having proposed adding carpool lanes to the San Diego Freeway, was asked why he hadn't examined four-way stop signs in Barstow.
For me, the most mysterious thing about the peer-review process is that it works as well as it does. Richard Rorty said it best:
On this view, there is no reason to praise scientists for being more 'objective' or 'logical' or 'methodical' or 'devoted to truth' than other people. But there is plenty of reason to praise the institutions that they have developed and within which they work, and to use these as models for the rest of culture. For these institutions give concreteness and detail to the idea of unforced agreement.
Comments (10)
Excellent post, Jonah! Just like the US Constitution, the structural design of the institutions is the key to the success of the enterprise. Like politicians, scientists are all too human, and need to be governed by structured rules to prevent mis-steps. Thus, there is undoubtedly room for more checks & balances in the scientific peer review process, both in publications and grants. Hopefully I'll have more time later to return to this post with examples.
Posted by: Neuro-conservative | August 23, 2007 11:28 AM