The system of peer review, the bulwark of academic publishing, has served scholars for centuries. The principle behind the system is simple: If experts in a field find a research report noteworthy, then that report deserves to be published.
But who is an “expert”? And who decides who the experts are? Couldn’t a group of individuals committed to promoting their own research — which may or may not be well-founded — get together to form their own “journal,” which they could legitimately claim publishes “peer-reviewed research”?
They can, and they do.
BPR3 Danny Chrastina asks:
What if it’s a pseudoscience paper which has been peer-reviewed by other pseudoscientists? I’m thinking of the reviews I’ve written of papers published in Homeopathy or J. Alt. Complement. Med. which point out their glaring misuse of quantum mechanical ideas.
Would a blog post citing such “research” qualify to use the BPR3 icon? There are a couple possibilities to consider:
1. The article is blatantly false and misrepresents its conclusions as “scientific.” Informed scholars agree that the work is simply wrong. My take: Citing this article alone would not qualify a blog post to use the BPR3 icon. If a legitimate, peer-reviewed article is cited to show why the article in question is wrong, then the icon could be used.
2. The article actually offers good information, yet it appears in a journal whose claims to be “peer-reviewed” are suspect. My take: If the work appearing in this journal is not generally accepted as reliable by true experts, then one exceptional article should still not qualify to use the icon. If the article didn’t undergo a rigorous peer review process, then citing the article offers no more authority than citing a newspaper article or a blog post.
A final question is perhaps the most difficult: How do we identify journals offering acceptable levels of peer review? Who’s to say whether a given journal is good enough? After all, even the most rigorous scholarly journals sometimes make errors — indeed, one of the most important parts of the scientific process is identifying and correcting problems in earlier work. Too rigorous a standard of peer review can stifle research just as much as too lax a standard.
I think the best way to maintain standards for BPR3 is probably to not start off with too many standards, but to be open to feedback from readers and bloggers about what factors we should take into account. In the case of the example Danny Chrastina offers, we can clearly see why the articles he describes don’t qualify as peer-reviewed. While so-called “experts” may have reviewed the articles, since they misrepresent well-established scientific principles, it’s immediately clear that the review process was not rigorous. When a journal exhibits a pattern of accepting this type of article and makes no effort to correct the problem, then we may be able to make a definitive statement and say that no article from that journal should qualify for our icon. But such a statement should only be made after we receive extensive feedback from readers and bloggers.
What we’re trying to do with BPR3 is establish a clear way to show where thoughtful discussions about serious research are taking place throughout the blogosphere. This isn’t to say that there isn’t great discussion taking place elsewhere. There are lots of great posts debunking pseudoscience; however, if they don’t cite serious research, in my view, they shouldn’t qualify for our icon. Our guidelines for using the icon, while offering flexibility, make that clear.
But I’d be interested to hear what others have to say — BPR3 is a community-run site, and it’s important to have reader input into our policies. Is there a place for posts that don’t cite serious research — perhaps to discuss a general scientific concept or debunk a problematic news article? Should these posts be marked with a different sort of icon? What standards would we use to ensure that they were high-quality posts?