You often hear scientists and philosophers of science talk about the peer-review process as if it's some epistemological magic trick, as if it automatically sifts through the mass of submitted articles and finds The Truth. Of course, if you've ever been through the peer-review process you know that there's nothing magical about it. The process can be just as arbitrary, unfair, illogical, and irrational as, well, everything else that humans do. My own encounter with peer-review left me acutely aware that scientists have big egos, and that big egos don't like contradictory data. That said, the process did lead to a better published article, with more conclusive data. So it worked. But my lingering impression was that it worked for the wrong reasons.
Despite the essential role played by peer-review in the scientific process, it's astonishing how little we actually think about it. If I were a philosopher of science, I'd stop trying to solve the demarcation problem and try to figure out practical ways to improve the system governing the acceptance of papers. How can we make our peers better critics? Is anonymity a necessary evil? Or does it encourage unfair comments?
That said, Terry McDermott's wonderful series on Gary Lynch is one of those rare works of scientific journalism that actually explores the fraught nature of the peer-review process. It's a rare peek inside the sausage factory of science:
When a paper is submitted to a scientific journal, the journal editors send it for review to panels of scientists. Peer review is the backbone of contemporary scientific legitimacy and lauded by everyone involved. It is also an opportunity for mischief and misunderstanding.
Lynch's history of antagonizing his peers sometimes made peer review more a gantlet than a critique. Richard Thompson of USC, a renowned neuropsychologist, said he had more than once nominated Lynch to membership in the prestigious National Academy of Sciences, but was told by other members Lynch would not be elected so long as they lived.
"There's a reason for his paranoia. There are a lot of people out there who don't like him. Gary doesn't suffer fools gladly," Thompson said, then paused for a moment. He chuckled and said: "And there are a lot of fools in the world."
The reviews on Kramar's paper seemed not to even acknowledge its main point -- that the lab had for the first time demonstrated the physical reorganization of cells that occurred in the final stage of long-term potentiation, or LTP, which Lynch believed was the biochemical process underlying memory.
One reviewer, in recommending against publication, complained that the scientists had only looked at a specific set of synapses, which was inexplicable as criticism. They looked there because that's where they were doing the experiment, that was where the condition they were examining existed. It was as if a traffic engineer, having proposed adding carpool lanes to the San Diego Freeway, was asked why he hadn't examined four-way stop signs in Barstow.
For me, the most mysterious thing about the peer-review process is that it works as well as it does. Richard Rorty said it best:
On this view, there is no reason to praise scientists for being more 'objective' or 'logical' or 'methodical' or 'devoted to truth' than other people. But there is plenty of reason to praise the institutions that they have developed and within which they work, and to use these as models for the rest of culture. For these institutions give concreteness and detail to the idea of unforced agreement.
- Log in to post comments
Excellent post, Jonah! Just like the US Constitution, the structural design of the institutions is the key to the success of the enterprise. Like politicians, scientists are all too human, and need to be governed by structured rules to prevent mis-steps. Thus, there is undoubtedly room for more checks & balances in the scientific peer review process, both in publications and grants. Hopefully I'll have more time later to return to this post with examples.
"That said, the process did lead to a better published article, with more conclusive data. So it worked. But my lingering impression was that it worked for the wrong reasons."
Are you sure it worked for the wrong reason?
The strongest systems are the ones which USE inevitable problems. Evolution is a fantastic algorithm because it uses sloppiness, error and imperfection as a source of raw material and creativity. Peer review is a strong system because it harnesses human faults -- ego, personality conflict, and an attendant desire to poke holes in each others' work -- to force arguments to be of a high standard. As long as basic groundrules are observed, like "physical reality counts", and "if the data are good and the logic is good then the argument has to stand", then hostility is better for rigor than friendly criticism.
So what were the "wrong reasons" in your particular experience? Can you share, or is it inappropriate for internet airing?
Agreeing with Luna: reviewers may comment on the "wrong" things, but indeed, knowing this will happen (and will be somewhat arbitrary and unfair) encourages us to make our arguments airtight.
That said, it's still a great question whether the process could be improved.
Luna makes an excellent point. I think that, at least in my case, peer-review worked for "the wrong reasons" in the sense that the paper didn't improve through astute criticism from the process itself. I remember being distinctly unimpressed by the criticisms we received. The paper got better, but that's just because we had more time and got some good advice from people outside the peer-review process. Hostility is great for accuracy, but its important to make sure that the hostility is focused on the facts, not the personalities.
Ni Hao!The current peer review system is nothing short of an expensive socio-political system of censorship of the most innovative and sorely needed science.Based on the basest graft, corruption, hypocrisy, egomania and ignorance characteristic of the human character, it is the single largest promoter of mediocrity, redundancy and support and publication of essentially meaningless trivia to date.It needs to be totally overhauled.The MOTYR Group
"Democracy is a terrible form of government, but it's better than all the alternatives" -- Churchill
A similar analogy applies to peer review perhaps?
I have seen a somewhat expensive but efficient extension for peer review in some computer science conferences (some of which have very rigorous reviewing): after having all reviews made, the comitee would get together and review the decisions, discussing whether the criticisms which motivated them were fair or not. I've heard that occasionally more than half of the papers which would be accepted without this meeting are actually rejected in favor of others.
I wonder whether some variation of this review of reviews where reviewers loose anonimicity only other reviewers could be done in scientific journals as wel...
Peer review fails for two reasons (and it is usually very hard to discover when it does go wrong):
First, publication in a scientific journal is thought to some extent to represent the achievement of some kind of scientific acceptance - yet publication should really simply be entering an idea "into the fray". These are two very different entities, and since "scientific acceptability" does not occur until an idea enjoys some kind of consensus, new ideas find it very hard to meet the criterion for publication of being "scientific through [near-universal] acceptance".
The other reason peer review fails is because the scientific process is a form of expression of the kind of algorithm that optimises behaviour - in science's case, through optimising beliefs. With science, the idea should be exposed to as many scientists as possible, to see how well it explains, in many scientists' minds, a particular observation, or indeed many observations. This "offering-up in front of many" is not honoured by the peer review system that offers to typically three or four, who then prevent further exposure, sometimes on grounds of inconvenience to their own careers.
Peer review is an unholy alliance of editors who seek to save some time and much responsibility, with "experts", who sometimes are and sometimes aren't, who offer a little effort in exchange for advance notice of new currents of thought, and power, sometimes to prevent them.
I speak as one who has never had a paper in his main field rejected. However I can confirm that peer review does not always get things right. There is at least one scientific field that has been completely ruined, and filled with rubbish, by peer review. It has occurred because the field is made up of parts which are actually specialisms in other fields, but the "experts" in the main field have little understanding of those other specialisms. Of course, those specialists are never consulted, and as they consider they have more important things to do in their own field they don't make a fuss.
Peer review does cause problems, and should be replaced by something much better - and don't think there is nothing better! Remember, a new social institution only comes to pass if at least some of those affected don't lose out on the change.
I'm not going to publish my name, since I still work as an astronomer (even though that might change pretty soon).
While the idea behind the peer review system is noble and good, in reality it doesn't work. It suffers from a massive amount of problems, which lead to crap papers being published.
It's easy to see why that is: You just have to submit a paper to a journal and look at the review you get. Typically, the reviewer decides to remain anonymous, thus allowing him/her to send in comments that often are little more than insulting drivel. From my experience most reviewers don't bother to actually carefully read the paper at all (as is typically apparent from their comments). Instead, they tend to focus on whether or not they agree with the paper - something that is NOT subject of the review - and on whether you cite their papers.
As for the latter, it's usually quite easy to identify an anonymous reviewer by the papers you are required to cite. I once was asked to cite a completely obscure paper from 1975, which no one I talked to had ever heard of. The reviewer insisted that that paper should be cited instead of the one that had been cited literally thousands of times in the exact same context. Go figure.
As for the former, the role of the reviewer is to point out flaws in the paper and to make sure those flaws get fixed. If the reviewer disagrees with the conclusions, that's not a flaw. It's not up to the reviewer to decide which explanation is correct. I know of one case where a reviewer held up a paper for a whole year, solely because the paper dared to show that one of the reviewer's theories did not work. In the end, the authors asked the journal to get a new reviewer - which was happy to fix actual problems and have the paper published.
This kind of abuse of the review system is so common that the whole process is nothing more than a travesty. The same goes for the grant application process, where the same behaviour rules. There, it's even worse, since grants typically go to friends of the reviewers on the panel.
Ditto for job applications. Jobs often get filled with cronies. At a conference in Europe last year (I work in the US) various people asked me in utter disbelief why one position had been filled with a certain person, who - in their view - was clearly incompetent. What can one say about this?
After dealing with this bullshit for years now, I've really had it. People tell me this kind of bullshit is common in "the real world" (aka in companies), but at least if you work with some company they don't pretend they're after some lofty goals.
End of rant.
To: A scientist
In the "real world" -- corporate world -- you get all the b.s. you've described, plus you get zero loyalty, zero compassion, zero 2nd chances, cronyism, incompetence, huge political b.s., back stabbing, unfathomable logic (better known as illogic), incredibly expensive mistakes that are swept under the carpet and repeated many times, layoffs without notice, poor benefits, zero ethics at the board level while trouncing hard on anyone else for an ethics violation, and on and on. It ain't any better, just different.