Posted by Over at the ARN blog, Denyse O’Leary has a four-part article up attacking the peer-review system. Rob Crowther, of the Discovery Institute’s
Media Complaints Division, has chimed in with his own post on the topic. There’s a great deal of humor in watching anti-evolutionists try to dismiss peer review as not worth the effort anyway. It bears an amazing resemblance to this really cute old fable about a fox, but I’ll be kind and pretend that there is actually something more to the O’Leary and Crowther rants than good old sour grapes.
Their major complaint about peer review is, of course, that their stuff, for some bizarre and unaccountable reason, has a really hard time surviving the process. In Crowther’s words:
To sum up, science journals that are wedded to Darwinian evolution refuse to publish authors who explicitly advocate intelligent design. Then Darwinists attack intelligent design as unscientific because it isn’t published in peer-reviewed journals.
O’Leary puts it a bit differently, but the basic concept is the same:
There is a modest but growing number of ID-friendly peer-reviewed publications. But – given the woeful state of peer review – papers that support or undermine ID hypotheses would probably be neither better nor worse recommended if they were never peer reviewed, just published, amid cheers and catcalls..
Of course, they try to justify their criticism of peer review on grounds other than their inability to reach the grapes. Peer review, they claim, doesn’t identify fraud. It’s not that good at catching incorrect findings. It squelches new ideas. It places “intellectual pygmies” in judgement of intellectual giants. It favors consensus. It sucks the life out of people, and is entirely responsible for global hunger and bad hair days. OK, I made the last two up, but you should still get a taste for the basic strategy that’s being employed here – it’s an oldie, but a goodie. Throw as much crap as you can at the wall, and hope that some of it sticks.
In this case, some of it does stick. It should. Peer review is not a perfect system. It is absolutely flawed. It is, in fact, not good at catching fraud. It does not catch many flawed studies. It does make it more difficult to publish new ideas, and it is absolutely capable of sucking the will to live from people. (Just because I made that one up doesn’t mean it isn’t right.) To paraphrase Churchill, peer review is the worst system out there, except for all the others that have been tried.
At this point, I must in fairness note that O’Leary, in Part 2 of her “critique” of peer review, specifically takes exception with the Churchillian analogy. However, she does so in a breathtakingly (yet unsurprisingly) asinine manner:
But the convenient analogy to democracy fails. In the first place, the secrecy in which peer review operates make it a poor analogue to democracy. Second, democracy aims primarily to give every citizen a vote. The fact that some citizens vote for cranks or criminals does not mean that democracy has failed. But peer review’s primary aim has been quality control, and it has been failing for decades. It squelches too many good ideas while failing to prevent too many frauds.
To begin with, the reference to the Churchill quote does not attempt to compare the way the peer review system operates with the way that democracy operates. The analogy suggests that, like democracy, peer review is a flawed system but one that is less flawed than the alternatives. That much should be obvious to anyone with better reading comprehension skills than the average functional illiterate.
Any attempt to compare the way the peer review system operates with the way democracy operates would be stupid. Peer review is not democratic. It makes no pretense at being democratic. Science is a meritocracy. Peer review is an attempt to ensure that published scientific papers have at least some merit.
As an aside, it’s worth mentioning that O’Leary’s analysis of the target of the analogy is almost as skilled as her characterization of the goals of democracy. The goal of democracy has never been to give every citizen a vote. If it was, it probably wouldn’t have taken quite so long to extend that right to women and minorities, and the right wouldn’t be stripped from felons. The guiding principle of democracy, as it is generally practiced today, is the belief that a government of the people will be more able to meet the needs of the people than a government run by a monarch (or other dictatorial leader). This usually works out quite well, but the fact that total bloody idiots still get elected is a flaw. It’s an acceptable flaw, however, since (a) there is no other system available that is capable of ensuring that no bloody idiots get elected; (b) later elections can (usually) correct much of the damage caused by the idiot; and (c) democratically elected governments are in fact better at representing the people than appointed or anointed governments.
Similarly, the goal of peer-review is to provide a basic check on the quality of papers prior to publication. It doesn’t always work. But, on average, it works better than the other quality control options.
It is possible, however, that O’Leary’s criticue of peer review has more merit than her analyses of written English and Political Science were, so let’s look at a couple of the specific critiques.
One of her major complaints is that peer review does not detect fraud. Personally, I don’t believe that it was intended to do so. Preventing fraud would require the external examination of all of the raw data in every published study, and that is quite simply not feasible – and, in many cases, not possible. O’Leary believes that, “embarrassing frauds have created a demand for a system that can detect fraud.” To support this claim, she baldly asserts that:
A host of individual acts of sloppiness (or malice!) can get lost in the smoke generated by a really big fraud like the stem cell scam. Defenders of the system can then safely claim that the Big One is unrepresentative. That is usually not true. It would be more accurate to say that the ensuing uproar is unrepresentative. With scandals, as with rats, if you see one, there are probably a dozen, and the rat that caught your headlights was just unlucky. And a big one always gets more attention than a bunch of little ones.
Even if we accept that assertion – and there is absolutely no evidence that it is the case – it is difficult to see that the numbers demonstrate a significant problem. A search for articles on PubMed that were published during 2005 returns 617,207 results. Limiting that search to show just articles written in 2005 that have been retracted reveals 34, most of which were retracted by their own authors after they discovered problems themselves. However, even if we assume that each and every case where the article was retracted represented a case of fraud – which is clearly not the case, and that there are three dozen (triple O’Leary’s assertion) that weren’t caught for each of the ones that was, the grand total is still less than two tenths of one percent of the total number of papers published. 99.8% isn’t perfection, but it ain’t too shabby. Balance that against the massive amount of effort that would be required to check all of the data in every study prior to publication, and it’s hard to see how the benefits would outweigh the costs.
Some of the other concerns raised by O’Leary do have a bit more merit. It is sometimes difficult to get new ideas published, in large part because reviewers are more likely to scrutinize every detail of a paper that does not match up with what they think they know about the subject. That’s just human nature, and it’s hard to get around. O’Leary and Crowther point out that ideas that eventually resulted in Nobel Prizes were originally rejected for publication, but they miss one important detail – the ideas were, in fact, eventually published. The authors might well have needed to do a lot more work to get them published, and provide a lot more evidence, but they did eventually succeed.
It would be nice if there was a system that could get around this, and one might eventually be developed, but we definitely aren’t there yet. Some of the proposed changes to peer review – publishing the reviewers’ names, publishing reviews along with papers – have merit, but do not address that basic problem (as rejected papers won’t be published). Suggestions that do publish every submitted paper, along with critiques, suffer from a different limitation. Right now, more papers are published than can possibly be read. That 617,207 figure for 2005 came from a single database that primarily indexes journals with biomedical applications. It doesn’t capture the entire biological literature, buch less the entire scientific literature. Peer-review, while imperfect, does serve to keep the worst papers out of the mix. If everything was published, every individual scientist would effectively have to conduct his or her own review on every paper of possible interest. With so many scientists doing and publishing so much work, that just isn’t remotely practical.
November 17, 2006
November 17, 2006
November 17, 2006
November 17, 2006
November 17, 2006
November 17, 2006
November 17, 2006
November 17, 2006
November 18, 2006
November 18, 2006
November 18, 2006
November 18, 2006
November 18, 2006
November 18, 2006
November 19, 2006
November 19, 2006
November 20, 2006
November 24, 2006
