The Island of Doubt

Goldacre, whose regular columns in the Guardian should be required reading for anyone interested in how science operates within the larger societal context, had a chat with the editor of the Journal of Medical Hypotheses, who

once gave me two excellent reasons for publishing loopy papers (my phraseology). The first was that academics had to be free to just get on and publish things that outsiders might find weird or misinterpret, without worrying about what the wider public thinks.... [and] this is a view I heartily endorse.

The thing to know about the Journal of Medical Hypotheses is that it isn't properly peer-reviewed. You really should read the whole thing, but here are couple more excerpts:

Academics should be free to write tenuous papers, and the infamous 1998 MMR Lancet paper is a perfect example: it covered 12 children who had autism and some bowel problems, and had been given the MMR vaccine. It didn't tell us much about MMR causing autism, but no one should censor themselves from publishing such work on the off-chance it might trigger a 10-year epic scare story from mischievous journalists.

...

Two weeks ago we saw that only one in four cancer trials were published. There are widespread demands that all negative findings be published, so that at least they are accessible. But this will often mean that inadequately analysed data from less competent studies gets published in journals that will take poor quality papers. The signal-to-noise ratio in the scientific literature is getting ever lower, and the simple fact that something has been "published" is becoming as meaningless as it always should have been: ideas are there to be read and critically appraised. Science is not about arguing from authority, and the era of "it's published so it must be true" is mercifully drawing to a close.

Goldacre's frustration with peer review, I think, stems from the fact that there are far too many low-quality journals that will publish low-quality papers after subjecting them to low-quality peer review. And there are too many dilettantes out there, especially in the blogosphere, who will herald a paper that has been peer-reviewed, regardless of the quality of that review, as proof of one thing or another.

Goldacre's argument is worth considering. He's not arguing that there ever was a link between MMR and autism; he's just point out that peer review made any difference to keeping dangerous ideas out of the public consciousness. It's bad enough that the media will run with a scientific story that hasn't been put through any kind of peer quality control. But even when a study has been reviewed by some anonymous peers, it can still be dreck.

It's becoming increasingly clear that peer review needs some kind of peer review. I have no idea how to go about fixing the problem. Goldacre doesn't offer an alternative, and I'm not sure he wants one. So what are we to do?

By the way, although my own experience suggests the ejaculation-sinus connection has merit, the paper in question finds it only "tenuous."