Peer review

Yup, you've pretty much exactly described peer review as I've experienced it. There really is a remarkable lack of understanding of peer review (of science/academia in general I would say) within the "skeptic" community. From some discussions I've had, if they had their way they'd spend all the research money on auditors/overseers and none on the actual research.

[That's a bit of a relief. I had a terrible feeling that I'd be out of date, or my own experiences would have been atypical -W]

I published through a peer review and arXive. You are correct; many unfamiliar with the process get it wrong. However I suggest one other dimension to the process. The dimension is novelty or, rather, disagreement with prevailing standard models. This is the social dimension.
H. Arp book “Seeing Red” explores the reaction of the science community to a nonstandard interpretation of the data.
One criterion you stated is the internal review that presumes working in a group. Members of groups get published more than individual researchers. The group dynamic helps advance some things, but retards out-of-the-box ideas.
Another part of peer review is the multiple submissions required. This is a real pain and takes away from research time.
If you are paid for your research – it’s publish or perish. The formula is become highly published, the strike out with out of the box material. For my part, I am retired. I don’t need the groupthink. So I get to be wild. So, I’ve migrated to a forum that has minimal review (I was invited, my prior work qualified me). I have thought quite a bit about the tradeoff between ease and wide distribution. I think I am the crazy scientist with way out ideas working alone. I think I like it that way, but sometimes I wonder.

It takes considerable time to prepare a paper for peer review.

Its prospect can act as a deterrent against lazy carelessness and the use of intentional obscurity*. If you don't believe me consider the difference in quality between the two kinds of work from some authors who are well known for both kinds of communication.
-----------------
* Although reviewers vary enormously. Some want to understand every step ; others just look for originality and interest.

Although there is much to agree with in this post, you would choke if I left it at that.

Although isolated instances of peer review date back to the 17th Century, as a systematic practice it is much more of a recent phenomenon. Most of the important science we look to was not in fact peer reviewed--hence its role can not be considered essential.

As a gate-keeping mechanism it has obviously failed--bad, even fraudulent science gets past peer review regularly. You speak of it as a filtering mechanism, blocking out the noise, but there are so many journals publishing so many papers that that is no longer true, if ever it was.

Of course skeptics will bitch about peer review--but the subtext is that they're really bitching about Mann and Co. trying to lean on a couple of journal editors--a tactic that worked, you may recall, or the publicized cat fight about the dueling Antarctic Ice papers a year or so ago. You are correct that a scientist can venue shop until s/he finds a congenial editor and that skeptic climate scientists such as Lindzen, Christy and even non-skeptic but non-consensus scientists such as von Storch are not shut out of the literature. But they haven't made recent news, which is what drives blog conversations.

Now that Elsevier is cracking down on scientists who are guilty of the heinous crime of putting their published papers on their own blog, the fledgling movement to open access may gain some momentum. What would work best would be a combination wiki with spawning blogs, or some way to provide electronic marginalia for comments. We'll see.

Side question: Why do you make so much sense in your comments over at Planet 3 and champion such marginal arguments elsewhere?

[If by "elsewhere" you mean the denial-o-sphere, the problem is the low base all the conversations start from, and the poor reasoning ability of those commenting, so its pretty hard to have an intelligent conversation. JoNova's is a fair example. If you had a specific instance in mind, it would be easier to be give a specific answer -W]

As a gate-keeping mechanism it has obviously failed–bad, even fraudulent science gets past peer review regularly.

Well, it's obviously not perfect, but do you really think it doesn't at least limit the amount of "bad, even fraudulent science" published?

My negative experience with peer-review is mostly lazy reviewers. "How did that one get by?" seems more common. The secon one is the particular "figure of merit" types that either can't think beyond their own, old paper or want a citation. The third is the ladder climbing department head that basically dominates a particular field through subordinate publishing and review to build their CV. They become the walled garden of topics that no one will ever mention again (or at least publish to the journals they do). Patents in some instances count as peer review for the "publish or perish" group and University patent trolls is one of my pet peeves now.

bad, even fraudulent science gets past peer review regularly

Peer review is not designed to catch fraud: the presumption is that the authors of the paper performed the described experiment(s) and obtained the described result(s). Most reviewers don't have the time or resources to duplicate the experiment.

I take a Churchillian view of peer review: it's the worst possible system, except for everything else that's been tried. The current peer review system does have problems, but not of the kind you see in denialist circles. The biggest defect I see is the perverse incentive: there is no explicit reward for doing the job well; instead, those who do it well get to do more of it (generally without compensation), while those who do it badly tend not to get asked again. This is particularly an issue in countries like the US where success is measured in terms of winning grants: those proposals are peer reviewed, too, and success rates are low and declining (depending on field, there will be between 4 and 20 proposals submitted for each award made). One of the effects of this phenomenon is the "lazy reviewer" issue Tim @6 mentions.

I've gotten to experience peer review on 3 sides now - published articles, reviewing articles, and being a journal guest editor. I've used the expression "peer review as a spam filter" before - weeds out a bunch of dross (though some gets through), and lets through the good stuff (though some gets blocked erroneously). I think that this does some discredit to the role of peer review as an improvement process: I find there's actually a decent percentage of "the occasional well-considered thoughtful comment that genuinely improves things" and have seen one of my own papers improve dramatically through the peer review process, have seen one Nature editorial that I reviewed change (and improve) dramatically (all 3 peer reviewers had similar concerns, which helped), and as guest editor have tried to convince the authors to take into account the most important comments and seen less dramatic changes but definitely strengthening.

I will also say that I have not seen serious evidence of the "enclosed system" in the climate change field: I did, however, have a friend in a sub-field of chemistry where apparently there were two camps about a certain phenomena, and if you got peer reviewers from the wrong camp, your paper would be doomed. Which was unfortunate, and made life difficult for my friend's research group... but they did still publish, so not impossible.

I've tended to find peer review a positive, albeit tedious, process in the main. My papers have always benefitted from a bit of tweaking from less-involved parties. This is particularly the case when embarking on projects that touch on areas outside my groups core specialities.

Reviewing papers is always rewarding, both in helping other researchers and also as a means of keeping abreast of developments in my field that may have been missed otherwise. Although it can be a pain if a paper is particulalry bad or non-innovative.

My main bugbear is having to review papers from researchers who's first language isn't English - I've no problem with receiving them direct from the research groups but those that have apparently gone through "scientific editing" to improve the language used (at some cost to the research group) that have had no demonstrable impact on the quality of the English. There's definitely room for improvement in the Scientific Editing business if anyone is casting around for work...

The most ferocious peer review process I've ever seen was at Bell Labs, internally, because they did not wish poor papers to get out. if a paper got through that, it tended to get published.

Suppose Member of tech Staff writes paper for external publication. It went up their management chain:
a) Supervisor
b) Dept Head
c) Director
d) Executive Director (who would manage 1000-1500 people)

The ED would send it to 2 other ED's, who would send it down their chains to review, reviews come back up to them, then back to the original ED, and back down the chain through your line of management.

One of my MTS's (I was a Supervisor) savaged a paper that came to us, I signed off on the review (I'd read the paper and agreed), sent it up. My ED copied me on the note he attached: "Dear ED XX: once again, I'm afraid some of my people think a paper by some of yours is junk and I agree with them."

Anyone with any sense knew they'd better publish internal memoes* first, pass them around, talk to experts if needed, given the visibility of this process at all levels, taken seriously. Career-limiting moves could be obvious. Imagine doing a paper with junk statistics ... it would likely end up with John Tukey or somebody in his organization and meet a dire fate, visible to all.

* There was an extensive system for this, already well emplaced in early 1970s. One would write a Technical Memorandum, get it reviewed/approved, probably up to DH or DIR level, attach subject codes and other people's names you thought should be alerted. Each person would maintain a profile of codes of interest.
The Library would combine all this, so that every week, you'd get a bunch of cover page/abstracts of memos, If you wanted one, you circled your name, folded the page over, stapled it, and threw it in the internal mail. Then a few days later, the full memo would arrive.

Have you seen what we are trying to do to change peer review?

thewinnower.com

http://membercentral.aaas.org/blogs/driving-force/can-winnower-flip-sci…

Somewhat tangential, but there is an example of a particular orthodoxy suppressing publication of opposing points of view. Just not in science. Economics. Krugman has talked about it.

[{{cn}} -W]

The market idea of Jo Nova could be a good idea, but I see one major problem: how do you determine how good a article is. You could use the number of citations. And then set up a market trading in futures for the number of citations 10 years after publishing.

However, if I look at my own publications, I would say that the number of citations is only weakly correlated to the quality, contribution to the field (in my subjective estimate).

[Yup. And the last thing we need is more people writing "citation ready" papers rather than unshowy but good stuff. I think the idea using "the market" is appealing in principle, but rather hard to see how it could work in practice -W]

You could set the papers to multiple reviewers and pay them for how well they get the median right. But that would be highly dangerous, it will be hard to stop people collaborating and this would easily lead to gate-keeping.

[Plus no-one wants to pay reviewers anything at all, and I don't see that changing. Unless... one could concieve of the payment-structure of science changing. Less base salary, more people expected to top their income up from reviewing? -W]

Maybe some social media type of system could work, Like Slashdot. But it would be hard to make sure that important articles in niche fields get sufficient attention relative to average articles in large fields.

[This sort-of feels like the kind of thing traditional publication is missing: at the moment, journals are moving on line, but its just a bulk shuffling of dead trees into dead electrons: its not using the power of the new meeja. I almost like the idea of no journals, no pre-review at all. People just stuff papers online, and then some reputation-based system pulls out interesting ones. Perhaps journals could just become places that review existing stuff, and publish reviews, not papers? -W]

I have the feeling, I get better reviews at European journals as in US journal. Maybe because "publish or perish" is stronger in the USA, maybe because in Europe people know me.

Two Krugman posts relevant (I think) to Greg's comment:
http://krugman.blogs.nytimes.com/2013/12/17/the-facebooking-of-economics
http://krugman.blogs.nytimes.com/2013/12/13/rudi-dornbusch-and-the-salv…

Josh Nicholson, What is so special about the Winnower?

In the Geosciences we already have a range of Open Review Open Access journals.
http://publications.copernicus.org/open_access_journals/journals_by_sub…

The only novelty I see it that the Winnower would not have anonymous reviewers, only names ones for increased transparency. The Copernicus journals invite several anonymous reviewers and named reviewer can also comment. Experience tells me that it is very rare of named reviewers to show up.

Some of the problems of peer review are discussed on twitter with the hashtag #SixWordPeerReview.

https://twitter.com/search?q=%23SixWordPeerReview&src=hash

Pear review is kinda fruity.

Less base salary, more people expected to top their income up from reviewing?

This would help with the perverse incentives I mentioned above.

I almost like the idea of no journals, no pre-review at all. People just stuff papers online, and then some reputation-based system pulls out interesting ones.

There is an implementation of this idea in physics: the arXiv. In some subfields of physics (though not the one I'm in), the arXiv is the main vehicle for publishing papers, and subsequent peer review by a journal (e.g., Physical Review Letters or the Astrophysical Journal) is considered a formality. The system isn't perfect: people try to time submissions so that their article is near the top of the list in the weekly e-mail. But it seems like a good starting point toward what you have in mine.

Paper came back from review today. Lead author's Email, good reviews except #3 and we know who he is. (no joke)

Everyone has read Michael Nielsen's "Reinventing Discovery", right?

If I have seen further it is by standing on the shoulders of giants. - Isaac Newton

Information wants to be free. -- Stewart Brand

"...no amount of scientific research, carefully recorded in books and papers, and then put into our libraries with labels of secrecy, will be adequate to protect us for any length of time in a world where the effective level of information is perpetually advancing." -- Norbert Weiner

Capitalism wants information to be expensive; a commodity to be bought and sold. The unhampered exchange of scientific information was seriously curtailed during WWII. In the aftermath governments and corporations were loathe to lose the control they had gained over scientists in the name of national security. Few were willing to resist; Norbert Weiner was one who did.

Scientists have always built upon the work of their predecessors; what we're really talking about is making that communication of ideas more efficient.

Digital libraries, open access journals, scientific blogs, web-seminars, etc all make the communication of ideas more efficient. Why think when you can just look it up?

A brief example: I saw a job ad for a programmer that detailed a coding exercise. The employer wanted code that would return all the occurrences of a certain phrase in Wikipedia. I could sit down and write the code, but it would take *me* at least a couple hours. Instead I did a couple of Google searches for freeware that would do the same thing. Within 5 minutes I had installed two software packages and not only had all of the Wiki pages with the phrase, but had them sorted by page rank.

The efficiency of information grows exponentially when it is openly shared. I doubt even an experienced programmer could beat that time :)

All we need do is address the concerns of Norbert Weiner -- that information should not be controlled by economics, but by the what's in society's best interest.

Pier review should be the name of the tide gauge version of the surfacestations project.

This is especially for John Mashey, some personal history about Bell Labs that I've had to scrub a bit:

"... an idealised, early version of Bell Labs review process ... in the '50's, which later broke down in physics, to soome extent at the hands of careerist types ... and certainly had completely broken down by the Schon scandal. It may have lingered more iin some departments. ... ED having shouting matches ... [with someone who] put out nonsense, even into the 90's. But ... a claque supporting never could be completely controlled."

Don't forget the world some people live in:

http://www.ssrc.org/publications/view/the-rise-of-the-dedicated-natural…

"... take neoliberal theorists like Hayek at their word when they state that the Market is the superior information processor par excellence. The theoretical impetus behind the rise of the natural science think tanks is the belief that science progresses when everyone can buy the type of science they like, dispensing with whatever the academic disciplines say is mainstream or discredited science."

Mirowski, Philip, "The Rise of the Dedicated Natural Science Think Tank" (New York: Social Science Research Council, July 2008). http://www.ssrc.org/publications/view/the-rise-of-the-dedicated-natural…

Sad, but may be true, especially after my time.,
I was only there 1973-1983, and generally worked for some of the savviest lines of management at the Labs, for supervisors to Executive Directors, in some sense at Bell Labs' peak.

The world starting changing ~1983, as the 1984 Bell System breakup was nearing. Inside Bell Labs, people with entrepreneurial streaks could once get huge leverage to make things happen if they knew how to do it ... but then things got harder and harder as the effects of the oncoming breakup started to appear. Some of us foresaw frustrating times ahead, and headed to Silicon Valley, thus forced to move to places like Palo Alto. :-)

As for Schon, I strongly recommend Eugenie Samuel Reich's well-written Plastic Fantastic, fascinating ...
and very sad, even for those of us who were gone 2 decades by the time of the fraud revelations.

If I have seen further than others, it is by treading on the toes of giants - James "PRP Killer" Annan

[Or, in this case, of pygmies. But I don't think James is claiming to see any further because of this. Maybe the PRP people would like the alternative version: "if I haven't seen further, its because giants were standing on my shoulders" -W]

James and the Giant PRP Impeachment.

[:-). It is interesting that a few of the more awake denialists - those who've read far enough to notice his involvement - have decided he is Da Man. Whereas I presume he is merely the one who wrote about it on his blog; many others must have complained -W]

It is indeed likely many complained, but James has made it public he complained. And as we all know, the septics love to play the Mann^^man.

Another fundamental role of peer review that is pertinent to the amusing "Pattern Recognition in Physics" journal escapade:

Peer review further induces the manuscript submitter to pay a huge amount of attention into making his/her paper as solid as it can possibly be - especially if s/he chooses to submit to a decent journal. I submit on average 2-3 papers a year, but could probably submit 5 or 6 if I knew that there wasn't a quality-control hurdle to jump. Of course I wouldn't do so since I have the internalised critical facility that most serious scientists have/develop. But if one knew that any old stuff could make it into publication, what might be the point of trying to produce something good? - this in fact is a minor problem in contemporary science publishing with its massive tail of largely rubbish in a huge flush of new largely junk journals. At least this stuff is easily ignorable.

That's something of which the "Pattern Recognition in Science" shower are likely oblivious. The point of science publishing is not simply to publish, but to publish something good and meaningful. The expectation of a robust peer-review in a decent journal focuses the mind towards quality. The peer review itself reinforces this imperative towards quality and weeds out those that have deliberately or unwittingly over-reached themselves or have tried to present their mutton as if it's lamb...

You don't have to be a biased climate scientist to know which way the wind blows. Take catastrophic rising sea levels: you don't have to ask 97% of climate scientist about this, just ask someone with no axe to grind; say a seaside ice cream van driver. I mean these people are extremely territorial but yet there hasn't been one news story throughout the breadth and depth of Britain since the end of WWII where a summer time Mr Whippy route has been forced to change due to rising sea levels!!!!!!

That's all you need to know, that Rossi's Ice Cream van still pulls up at those sacred summer coastal spots. I'd call that an inconvenient Mivvy for the alarmists, wouldn't you Mr Cone olley?

Warmist year ? 99 with a flake

Now that's what I call Pier review

Ahhh the climate wars...

So peer review is what it is - sometimes helpful, sometimes not, and if your advisor knows the journal editor your paper will be better received that if not. And that has been my experience. It is ritualilstic, innefficient, flawed - but not the worst way for journals to manage their content. It is more-or-less irrelevant to science.

What matters is reproducibility. Considerable science is done in secret behind closed doors by evil capitalist who then use those discoveries to make money. No peer review required. Nor patents, BTW. Perfect Science, though.

The question at hand is this: Do certain elements of the climate sciences constitute science or not - specifically the forward looking projections about a scary future.

The answer, of course, is NO. Or at least not yet. At best it is an excuse to do computer science.

That doesn't mean it has no value, or the work should stop, or that it isn't entertaining, or that it shouldn't be published, or even peer reveiwed. And whether it should drive expensive policy is a slightly different argument.

It just ain't science. And peer review doesn't make it so. And that's really the point.

[I don't see any argument from you re the "ain't science" point; just an assertion. Do you have an argument? -W]

I would argue that peer review is actually most important for outsiders. First of all, it forces the peer reviewers to study the manuscript and in this way brings fringe ideas to their attention that may otherwise have been ignored. Also the reader is much more likely to take notice and to take seriously a fringe idea in a peer reviewed journal as in a blog post.

Hawkins just wrote an new manuscript on black holes and posted it to Arxiv. Scientists will read that and even the press noticed. Hawkins does not need peer review to give his paper some initial credibility, to suggest people that it is worthwhile to invest their time in trying to understand it. People on the fringe do need that help, peer review helps them.

P.S. Maybe PR, for "peer review", is not such a good abbreviation in an article about climate ostriches. Each time I see it, I first read "Public Relations".

I thought Monckton's "peer review" was funnier: http://www.wnd.com/2014/01/the-thermageddon-cult-strikes-again/ or http://notrickszone.com/2014/01/24/monckton-blasts-prp-journal-shutdown… -W]

Thanks for the mention of Halton Arp; I've long admired his catalog of galaxies; the cosmological stuff, I'd never heard of.

WMC, nice links. I had already seen the link to WND. Monckton actually has a weekly column there. A great resource for his extremist positions and foul language.

The discussion at NoTricksZone is revealing. I was not aware there was so much conflict in ostrich-land. There is normally almost no critique at WUWT, but apparently that is not for lack critique, but because of a policy not to make it public. NoTricksZone calls on his readers to forgive and forget the conflict between WUWT and PatternRecognition, to make sure that critique does not lead to public debate about the merits of ostrich positions.

I just want to thank all you hive-bozos... [Um. Well, after that unpromising start it didn't get better, or have any obvious relevance to the topic. But its over at the Burrow if you want to see for yourself -W]

Oh dear. Still in moderation.

Really?

[Apologies - released -W]

As I recall, this piece opened with
"Seldom in the field of human conflict has so much been written by so many people on a subject about which they know nothing."
fo which unidentifiable commenters kindly offer more evidence.

kdk33:

It just ain’t science. And peer review doesn’t make it so. And that’s really the point.

[I don't see any argument from you re the "ain't science" point; just an assertion. Do you have an argument? -W]

I'd like to see your argument too, kdk33, starting with what you think "science" is.

Can we first agree that peer review is just the way journals manage their content.

[No. I think that's naive and simplistic -W]

The work could be technically right, wrong or mixed; could be an incremental improvement, a breakthrough, or just another example; could be boring, exciting, or scary... And all of these are independant of peer review.

[No. Generally, there's a relationship between the quality of a published paper and the results of the review. In that good papers get published in good quality journals, and poor ones are rejected or improved to a better standard. Noting that rarely bad papers are published, or good papers rejected, doesn't change that -W]

Agreed?

Agreed?

[Do you think asking the same question twice is more likely to make people agree with you? -W]

That then is the crux of the problem.

[What problem? -W]

You've incorrectly linked peer review with technical accuracy (I'll give up the word science as I foresee intractable symantics)

The statement "peer review is how journals manage their content" is factually, perfectly, accurate.

[Its nice to know that you know that you're correct. But notice the statement you've made now is not the same as the statement you made earlier. Spot the difference -W]

I suppose by "naive" and "simplistic" you are implying incomplete. I further suppose that incompleteness is linked to "quality" and "good". These being perfectly subjective - simply the judgements of the reviewers who determine the journal content, and judgements vary as you know..

Yes, I am correct. Do you disagree? If so, can you forumulate a response?

I'm not interested in your spot-the-difference game.

Editor of a major journal in my field was on my dissertation committee. He gave me my first paper to review, because it was like what I had done in my dissertation. It was immediately obvious that the whole thing was based on an arithmetic mistake. I knew the coauthors personally, and felt good about keeping them from looking foolish. I have a similar, but less important, mistake in one of my papers. I missed it, my coauthors missed it, and the reviewers missed it. It was discovered by a colleague in France. I have reviewed papers from colleagues where English was not the first language. I helped them out, and their use of English improved over time. I like to think I helped them along.

My most cherished honour is a simple note at the end of a paper that I reviewed:
"We wish to thank the anonymous reviewer for their comments that have greatly improved this paper."

We went through several rounds of reviews that turned a piece of junk into a modest but useful contribution to the literature. They had (and still have) no idea who I am, so there was no effect of reputation. Just the acknowledgment that they had received god advice & had gained insight as a result.

I count that as a win.

god == good, of course

I wish I could give god advice...

In the old days, before my time, the editor did all that; no referees.

[Snip] I click on the link to "the Burrow" and, contrary to the promise made in your no. 37 comment, my "zinger" is not there [Snip. You're wrong; it is there. Perhaps you need to refresh your browser? -W]

I stand corrected, WMC--thank you!

"Seldom in the field of human conflict has so much been written by so many people on a subject about which they know nothing."
Indeed, I think this rivals the 100%-confident assertions by people who are obviously clueless about defamation law, as widespread around blogs about this case, which grinds along.

A link-rich page worth perusing, cautionary:

http://retractionwatch.com/2014/01/25/weekend-reads-trying-unsuccessful…

I have developed the idea mentioned above a bit further and made a blog post out of it: "Peer review helps fringe ideas gain credibility.

To stay on topic, I did not start about the market of ideas. But you could see the peer reviewed literature as the regulated top market, Dow Jones and WUWT & Co. as an unregulated grey market with penny stocks.

[Apologies for the delay (and the previous lost comment). You ended up in the spam bin. I don't know how that happened. Its a bit worrying in fact. Perhaps "market" or "Dow Jones"? -W]

Again, along the same line as #50 but over at Retraction Watch we see many comments by thsoe who know nothing, or less, although at least one (correctly) started:
"I am not well enough informed to argue specific cases,..."