Over at Unqualified Offerings, "Thoreau" offers some musings about peer review. I saw this and said, "Aha! The perfect chance to dust off an old post, and free up some time..." Sadly, I already recycled the post in question, so I feel obliged to be less lazy and contribute some new content.
I generally agree with most of what he says, but I would raise one quibble about his list of criteria:
What scientists are looking for when we evaluate a paper is whether the paper clearly addresses 3 points:
1) What is the question or issue being studied in this work?
2) What are the methods being used, and are they described in a sufficiently detailed manner so that somebody else can replicate the work? (Remember that replication is the real gold standard of scientific knowledge. Until we have independent replication of a result, it's suspect. Hell, even after independent replication we're still skeptical.)
3) Does the data presented support the conclusions that the author is drawing?
I think this leaves out one important question:
1.5) Is the result interesting?
"Interesting" is a loaded word, and it covers a lot of ground. It includes both the normal sense of "is anybody likely to care about this result?" but there's also an important element of "is this a new result?"
This is probably a more significant criterion in my mind than for some other people, as most of my refereeing has been for Physical Review journals, which are in the upper echelon of physics journals. Physical Review Letters in particular asks that papers be important to the field and of general interest to physicists. It's not uncommon to have a referee say that a paper is perfectly good, but not interesting enough for PRL. (In my opinion, it should happen more often than it evidently does, based on the large amount of highly specialized and uninteresting crap that's in PRL, but that's a different rant...)
This is an important point for any scientific paper, though-- the work presented needs to be new (you're not going to get a research publication out of proving that objects near the Earth's surface fall with a constant acceleration due to gravity), and it needs to produce a result that's interesting in some way (you're not going to get a paper out of "If you put an atom in this type of potential field, absolutely nothing will happen to it"). The threshold for "interesting result" isn't all that high-- there's some pretty "meh" stuff published every week-- but null results are pretty much out.
Other than that, I agree with "Thoreau." His conclusion is also worth quoting, and might generate some interest around here, though I can't spend much time on it:
The simple fact of peer review is that it's actually quite a modest hurdle. All you have to do is find one editor and a couple of reviewers who find the work plausible and well-executed. Once you've been on both sides of the process, you realize that it's just a preliminary quality check, a first pass before it's put out there for a wider audience. Some laymen seem to attribute too much significance to it, and other laymen seem to recoil against that misperception by concluding that peer review is too weak of a system. The truth is that it's not supposed to be a stringent filter. It's just supposed to be a first pass.
- Log in to post comments
"but null results are pretty much out"
this is both an over-generalization and a symptom of a problem
null results can be intensely interesting and important: to take a historical example, the many "no \delta T" found to some confidence level for fluctuations in the microwave background, were interesting before the sensitivity of the experiments got high enough to actually measure a non-zero delta T.
As a future example a short paper saying "we find no signature of Higgs" from one of the LHC collaborations is also interesting.
More generally, it is useful to have a lot of null results, like "atoms of A, B, C, ... don't show any effect in potential X,Y,Z" if only to eliminate too much duplication. Such results can also be strong tests of theories, if some theory predicts B and D should show the effect in Y potentials.
YMMV
One of the most important null results came when the laboratory in the Nazi nuclear physics project was systematically bombarding with neutrons every heavy isotope they had. On the day they tried U-235, a lab assistant accidently forgot to remove the shield (a square centimeter of gold foil) that blocked the detector, and no fission products were observed. They gave up on U-235 and moved on to other isotopes.
I hate to drop names around Chad, but I've discussed this at length with Freeman Dyson. He absolutely agreed that this was historically true. Where we differed was on the allohistorical interpretations of what would have happened if the Nazi bomb had been accomplished, with uranium from the mines of captured Silesia. I thought it could have at least delayed the outcome of WW II. Freeman Dyson said: "Okay, they nuke Paris, or London. What difference would that have made?"
I've been stuck for over a year being 90% done with writing my novel manuscript (Fast Times at Stuyvesant High) on what would have happened with a small Nazi nuclear weapon as payload of the Raketa Amerika -- a 2-stage scaled-up V-2 that could strike New York City. This A-9 or A-10 was the actual goal at Peenmunde.
Thoreau's concluding remark is the key point: "The truth is that it's not supposed to be a stringent filter. It's just supposed to be a first pass." Most if not all scientific journals have a comment and reply section, in which there are routinely disputes about some point which the referees may have overlooked. I've been involved in at least one such dustup myself. That's just the cases where somebody thinks, and the editor agrees, that there is a legitimate point that is worth arguing; I've encountered many papers where my reaction was along the lines of "How did that obvious FUBAR get past the referees?"
And also keep in mind that peer review operates on a basis of trust and is therefore not a good tool for detecting fraud. That's how both Jan-Hendrik Schön and Woo-Suk Hwang were able to publish such spectacular fake results for as long as they did: only when their peers had specific reason to believe that the results were fake did scientists check the work thoroughly enough to confirm the fraud.
I do think that there is a difference between interesting and important. That there are a hierarchy of journals that promote this is, I think, a reflection of that.
To give an example from my field, a paper on astronomical photometric standards will generate 1000+ citations and be used by an enormous part of the community, but you do not want to actually read it. It does not increase our direct understanding of astrophysics, but will enable a huge amount of interesting science to be done. Of course, no one would publish a letter about calibration.
Hey, guys, thanks for linking to it. Glad to hear that my colleagues concur with my statements on peer review. A lot of our readers are non-scientists, so I frequently post stuff to give an inside perspective on academic science. It's always good to hear that my observations ring true with fellow professionals.