Chad Orzel takes a commenter to task for fetishizing peer review:
Saying that only peer-reviewed articles (or peer-reviewable articles) count as science only reinforces the already pervasive notion that science is something beyond the reach of “normal” people. In essence, it’s saying that only scientists can do science, and that science is the exclusive province of geeks and nerds.
That attitude is, I think, actively harmful to our society. It’s part of why we have a hard time getting students to study math and science, and finding people to teach math and science. We shouldn’t be restricting science to refereed journals, we should be trying to spread it as widely as possible.
Peer review and refereed journals are a good check on science, but they do not define the essence of science. Science is, at its core, a matter of attitude and procedure. The essence of science is looking at the world and saying “Huh. I wonder why that happens?” And then taking a systematic approach to figuring it out.
I see what Chad is saying — and to the extent that science can be said to have an “essence” think he’s hit on a nice way to describe it. But I’m going to speak up for peer review here.
It’s worth noting that “peer review” can encompass different things.
Peer review describes the formal process through which manuscripts that have been submitted to journal editors are then sent to reviewers with relevant expertise for their evaluation. These reviewers then reply to the journal editors with their evaluation of the manuscript — whether it should be accepted, resubmitted after revision, or rejected — and their comments on particular aspects of the manuscript (this conclusion would be more solid if it were supported by this kind of analysis of the data, that data looks more equivocal than the authors seem to think it is, this part of the materials and methods is confusingly written, the introduction could be much more concise, etc., etc.). The editor passes on the feedback to the author, the author responds to that feedback (either by making changes in the manuscript or by presenting the editor with a persuasive argument that what a reviewer is asking for is off base or unreasonable), and eventually the parties end up with a version of the paper deemed good enough for publication (or the author gives up, or tries to get a more favorable hearing from another journal).
This flavor of peer review is very much focused on making sure that papers published in scientific journals meet a certain standard of quality or acceptability to the other scientists who will be reading those papers. There’s a lot of room for disagreement about what sort of quality is produced here, about how conservative reviewers can be when faced with new ideas or approaches, about how often reviewer judgments can be overturned by the judgment of editors (and whether that is on balance a good thing or a bad thing). As we’ve discussed before, the quality control here does not typically include reviewers actually trying to replicate the experiments described in the manuscripts they are reviewing.
Still, there’s something about peer review that a great many scientists think is important, at least when they want to be able to consult the literature in their discipline. If you want to see how your results fit with the results that others are reporting in similar lines of research, or if you’re looking for promising instrumental or theoretical approaches to a tenacious scientific puzzle, it’s good to have some reason to trust what’s reported in the literature. Otherwise, you have to do all the verification yourself.
And this is where a sort of peer review becomes important to the essence of science Chad describes.
The scientist, looking at the world and trying to figure out some bit of it, is engaged in theorizing and observing, in developing hunches and then testing those hunches. The scientist wants to end up with a clearer understanding of how that bit of the world is behaving, and of what could explain that behavior.
And ultimately, the scientist relies on others to get that clearer understanding.
To really trust our observations, they need to be observations that others could make as well. To really buy our own explanations for what we observe, we need to be ready to put those explanations out for the inspection of others who might find some flaw in them, some untested assumption that doesn’t hold up to close scrutiny.
Science may be characterized by an attitude toward the world, an attitude that gets us asking particular kinds of questions, but the systematic approach to answering these questions requires the participation of other people working with the same basic assumptions about how we can engage with the world to understand it better. Those other people are peers, and their participation is a kind of review.
In short, I agree with Chad that there is a lot of scientific thinking and activity beyond that whose products are reported in scientific journals after a formal peer review process. But I also think that it is vital to recognize that even the less formal, more everyday scientific thinking is inherently a team sport where verifiability by others is a criterion of success.