Survey of climate scientists

James Annan tells the story of how no-one seems to want to publish a survey of climate scientists done by Fergus Brown. The survey found that the IPCC report represents the middle ground of climate scientists with most of them agreeing with it. However about 15% felt that it was too optimistic and another 15% felt that it was too pessimistic.

I think the survey was much better than Bray's useless survey of climate scientists. I would have liked to have seen more than one question, so that the areas of agreement and disagreement with the IPCC could be measured. The response rate was very low, so the results should be taken with a grain of salt, but if The Economists' Voice will publish a survey on what economists agree on, it seems that this survey should be publishable as well.

More like this

But read the longer discussion on Fergus' blog .

I've occasionally refereed papers, written some, run program committees, and guest-edited technical magazines.

I'm unimpressed by the various publications' handling of it. I *really* hate it when somebody sits on a paper for long time, and then basically says "Not our kind of stuff." I certainly prefer to get good feedback, and when I'm on the referee/editorial side, I try to give it, although in some cases, it can be hard.

But I also wouldn't have accepted it, giving it either a Reject or "Accept with major changes", for reasons that I posted in the blog as feedback for Fergus.

I did think it was essentially fixable, even with the survey as it was, although I would have preferred better-parameterized questions.

By John Mashey (not verified) on 01 Mar 2008 #permalink

Yikes. That survey instrument is a disaster.

Fergus:

When students ask for advice on primary data collection, my usual advice is to tell them not to do it -- however, if they persist, I tell them to hone their hypotheses down to one or two and then ask the fewest questions needed to test those hypotheses, pre-test rigorously, and find a way to ensure representativeness of the response.

You've got an interesting question but not a testable hypothesis. And, while you cut your questions down to one, you made it so complex that you can't be sure what part of each option the respondent was reacting to. A pre-test might have revealed that. In your paper, you seemed to suggest that the single question you asked was chosen on the basis of easy presentation of the result. That's probably not the right way to go.

Even if you're not interested in a testable hypothesis and are simply interested in recording opinion, you'd still need some comparison. Most opinion surveys ask the same question with exactly the same phrasing over time so what they're measuring is a change in response, not the response itself. Since you're doing a one-off survey, you'd need to create your own comparison by asking the respondent to answer some other question or questions to put their response to the IPCC question into context. For example, you might think about asking if they agree or disagree (on the same scale you used) with the statement that the average global temperature has increased over the last decade. If you find a similar distribution for what should be a factual question, it would help in interpreting your results. There are other questions that help do similar things and you could ask a series of them.

I can see you've put a lot of work into what you've done thus far. Doing a good survey isn't easy and there are many subtleties both in design and implementation. Consistent with the advice I give students, I'd discourage you from continuing down this path -- however, if you persist, you really need to consult with experienced survey designers.

Fergus, I would add that there are too many ambiguities in the questions. You refer to 'scares', for instance - but as a loaded word the interpretation is bound to be dependent on viewpoint. Should the respondent understand 'scare' to mean Independent and Lovelock-style human-race-may-well-not-outast-the-century stuff, or claims such as 'given the already-depleted state of ocean fish stocks, acidification of the oceans and the effects this could have on marine biodiversity runs a risk of causing a catastrophic collapse of this vital food resource'?

Fergus, while I like your blog and think something along these lines is always better than nothing, Robert in particular is very good at this (as his helpful but unheeded immediate correction of David Kane showed [that IS the same Robert, right?]. Moreover, the data is still out there and won't go away.

I think doing that survey, and it being raised on blogs, is good enough. It should be tried again with the strictures Robert suggested and in general closer to what Naomi Oreskes did (and every survey can be improved), either by you or someone else. We can't complain yet, I think. This is just round one. Just my ¥2.

By Marion Delgado (not verified) on 03 Mar 2008 #permalink

I'd discourage you from continuing down this path -- however, if you persist, you really need to consult with experienced survey designers.

As a grad student in a multidisciplinary ecology program, we raised the specter of doing a survey for data collection for a project I worked on. We knew survey question writing was hard, but the path we were going down kinda required it.

The professors dutifully marched in some PhD students who recently ran the gauntlet of Human Subjects to get survey questions approved and blessed, & thus were fresh on the travails. We listened to this for an hour or so and retraced our steps and chose a different path.

Anyway, surely Fergus learned much from this exercise and I state: keep up the good work, sir.

Best,

D

Tangent -
can anyone recommend an online reference for how to design a survey?

By Anna Haynes (not verified) on 03 Mar 2008 #permalink

I wish to point out that this means that if I make a bowl of porridge, and get three little bears off the street, (because unnamed little bears know all about this stuff,) while admitting that my poll "cannot be tested for statistical significance," AND ALSO admitting that it is "not presenting the results as representing anything other than the views of those who responded as we have no way to assess the relationship [sic] of the responders with the total relevant population," and then those little bears say it's too hot, too cold, and just right, --and THEN the scientific press won't print this, well then: I am therefore morally and intellectually justified in accusing them (at climatesci.org) of "suppressing evidence that there is diversity of views" on porridge, if I merely only assert that my study, by EOS's lights, could "stimulate further discussion," and by Nature's, has "relevance and quality."

Do you really think these journals, even though not the peer-reviewed brands, should open the floor to an endless cycle of vague stuff on this issue? Do you have any idea how many unscrupulous oil industry hacks would line up to be published? Would howl in high dudgeon at not being published? This is exactly the stuff coming from the people hoping to dispute the Iraqi mortality studies. Everybody should be held to the same standard.

Dano wrote:

The professors dutifully marched in some PhD students who recently ran the gauntlet of Human Subjects to get survey questions approved and blessed, & thus were fresh on the travails. We listened to this for an hour or so and retraced our steps and chose a different path.

Your professors were very wise.

Marion Delgado wondered:

that IS the same Robert, right?

I'm not sure which is scarier: that there are two smart-asses here named Robert, or that there's only one.

My two cents: Accepting all the eminently reasonable points made above about the pitfall of conducting surveys, I see no reason why this survey should be suppressed. Does this survey contain dangerous information that should be hidden from the public? Flaws should be pointed out, not be seen as grounds for censorship.

I see no reason why this survey should be suppressed

A journal's decision to not publish a paper it deems unworthy is not "supression". Sheesh.

Does this survey contain dangerous information that should be hidden from the public?

Misinformation. Pielke, Sr. is already saying that this proves that there is no consensus regarding AGW in the relevant scientific community. And, yes, that's potentially dangerous.

> A journal's decision to not publish a paper it deems unworthy is not "supression".

Of course it is. It may be what you consider legitimate suppression, but it is still suppression.

> Misinformation. Pielke, Sr. is already saying that this proves that there is no consensus regarding AGW in the relevant scientific community. And, yes, that's potentially dangerous.

Suppressing articles because they may be used for misinformation seems like a wrong thing to do.

Of course it is. It may be what you consider legitimate suppression, but it is still suppression.

Wow. So when a journal gets 10 times as many submissions as it can publish, it is suppressing 90% of its submissions?

That's an odd view.

Suppress:

# To put an end to forcibly; subdue.
# To curtail or prohibit the activities of.
# To keep from being revealed, published, or circulated.

Now, how exactly does rejection of a submission amount to SUPPRESSION?

Suppressing articles because they may be used for misinformation seems like a wrong thing to do.

And what makes you think this was the case? The poll was poorly designed and the results not statistically significant. Why should it be published in a venue that places importance on such criteria?

I'm sure E&E would print it.

> So when a journal gets 10 times as many submissions as it can publish, it is suppressing 90% of its submissions?

Somebody makes the choice which 10% are published, and which 90% are not published (i.e., suppressed - in the sense of "to keep from being revealed, published, or circulated."). That choice is not made at random - it is made to serve the ideas, biases and interests of whoever makes the choice.

>> Suppressing articles because they may be used for misinformation seems like a wrong thing to do.

> And what makes you think this was the case?

We will never know, since editors always have more convenient grounds to reject papers, but for a second I had the clear impression that you thought it was a good idea.

Somebody makes the choice which 10% are published, and which 90% are not published (i.e., suppressed - in the sense of "to keep from being revealed, published, or circulated.")

You can reveal your paper to whomever you want, publish it elsewhere, or circulate it via snail-mail or by standing on a street corner saying "jesus saves" and foisting it on unsuspecting members of the public.

It's not been suppressed.

If their poll results had been suppressed, we wouldn't be discussing it.

Good grief.

> You can reveal your paper to whomever you want, publish it elsewhere, or circulate it via snail-mail or by standing on a street corner saying "jesus saves" and foisting it on unsuspecting members of the public.

Would you use the same standard for what counts as non-suppression if at issue was some report about corruption in the Bush administration which is rejected as "unworthy" by every wide circulation news outlet?

Suppression may not be complete - it aims (with varying degrees of success) at reducing the exposure to certain pieces of information or certain ideas.

Ah, the spectre of Galileo.

Using Lee A. Arnold's example, if I decide to do a survey of bears' like and dislike of porridge, and push for it to be published in Nature Climate Feedback, and Nature Climate Feedback refuses to publish it, then it's "censorship"?

it's a free country, and i am free to insist you publish my magnificent opus or i scream that i am suppressed.

just like if you don't use DDT as underarm deoderant, you are de facto banning it.

and if somebody is prevented from harassing clients at planned parenthood or a gay bar or something, then they are being persecuted because of their beliefs.

> Ah, the spectre of Galileo.

?

> if I decide to do a survey of bears' like and dislike of porridge, and push for it to be published in Nature Climate Feedback, and Nature Climate Feedback refuses to publish it, then it's "censorship"?

By definition, yes. It may be a case where you approve of the censorship, but it is censorship nonetheless. Calling what other people do censorship but what you do "rejecting" or "refusing to publish" is employing double standards.

Once we put the matter of terminology behind us, we can discuss reasonably if we support censorship under some conditions, and if so, under what conditions. It seems to me that the case of the survey of climatologists is not a plausible candidate for censorship.

z,

As you say, anyone can say anything, and, I might add, many people do.

What we are trying to get at, however, is what actually makes sense. Coming up with examples of nonsensical things does not make the sensical things any less sensical. If you have an argument to make let's have it.

BTW, I am still waiting for your attempt to reconcile your statement that "everybody agrees" that Israel should withdraw to the 67 borders with Israel's colonization policy of the West Bank.

By definition it is not "censorship" because it is not an official or governmental act. Even in the modern vernacular it is not "censorship," because it doesn't hold over the field of relevant publications or venues.

It is not "suppression" because, in accordance with the additional meaning that all speakers of English give to this word almost without exception, there is no intent to keep the results secret.

It was rejected. You can go somewhere else with it. They have a perfect right to reject it, or anything else for that matter, until subject to a complaint from a sufficient number of their board of members.

Now, why are they correct to reject it? Because a survey without a proper methodology proves nothing, and since we already know the IPCC is a consensus, it also suggests nothing that is new.

In addition, they are very well-advised to reject it. Because, however well-intended this paper might be, accepting it would open the floodgates to every boneheaded clown at the Heartland Institute's New York Nonsense and Snivelfest, hoping to claim the credence of a Nature or AGU citation. For what possible reason could their stuff now be rejected? Because it isn't up to standards? What standards?

> By definition it is not "censorship" because it is not an official or governmental act. Even in the modern vernacular it is not "censorship," because it doesn't hold over the field of relevant publications or venues.

As I already pointed out, this kind of standard is both unreasonable and does not correspond to normal usage.

If a newspaper decides to kill a news piece because it is unpleasant to one of its advertisers then most people would call it censorship - even though no government and no official is involved, and even though, theoretically the author could go to a different newspaper (which, most likely, also carries a lot of ads from the same advertiser).

> there is no intent to keep the results secret.

I would say that there is intent to reduce the audience of the results, in just the same way that in the hypothetical example above the newspaper is trying to reduce the audience of the unpleasant story.

> They have a perfect right to reject it, or anything else for that matter

Exactly - they have the right to censor, just like the newspaper has the right to kill the story.

> Now, why are they correct to reject it? Because a survey without a proper methodology proves nothing, and since we already know the IPCC is a consensus, it also suggests nothing that is new.

Why won't "we" let the audience decide? Why should "our" opinion be the last word on the subject?

> In addition, they are very well-advised to reject it. Because [...] accepting it would open the floodgates...

It sure sounds like you are advocating suppressing the results at least partly because acknowledging them would be inconvenient.

Sortition:

Coming up with examples of nonsensical things does not make the sensical things any less sensical.

Well, now we have an idea of Sortition's bizarro thinking.

Who decides what's "sensical" and what's not? You?

Who decides what's "reasonable" and what's not? You?

Why should readers not be allowed to see a survey of bears' like and dislike of porridge in Nature Climate Feedback, just because you deem it "nonsensical"?

Of course, if you start your own "scientific" journal, you jolly well have every right to publish any amount of "uncensored" (read: half-baked) research that you wish. And we also have every right to mock every ridiculous piece of "work" that goes into your journal.

And Nature Climate Feedback has every right to decide what goes into their forum. Their money, their rules.

This is awesome. My novel no longer sucks; it's being suppressed by the big New York publishers.

Lexicography for conspiracy theorists.

Very definitely it would be "inconvenient," by a couple of the OED's definitions. Because of the authors' following statement: "we have no way to assess the relationship of the responders with the total relevant population."

bi,

> Who decides what's "sensical" and what's not? You?
>
> Who decides what's "reasonable" and what's not? You?

I'm glad that you asked. My proposed solution for the matter of academic censorship (a.k.a. peer-review) is here (last paragraph).

Boris,

> This is awesome. My novel no longer sucks; it's being suppressed by the big New York publishers.

If publishers "reject" you manuscript then it is - by definition - being suppressed. Sorry - it may still suck. It may even be the it is being suppressed because it sucks, although I would certainly not take this for granted.

Lee,

> "we have no way to assess the relationship of the responders with the total relevant population."

I agree that the low response rate makes these findings less informative. I don't even know that the entire matter of consensus is that informative. But these are just my opinions - I don't see why my opinions, even if they are informed opinions, are grounds for suppressing someone's work.

(BTW, I found your cartoons interesting.)

I have to admit I'm almost humored when someone thinks they get to redefine commonly used english words like "censorship" and "suppression"...

dhogaza:

I have to admit I'm almost humored when someone thinks they get to redefine commonly used english words like "censorship" and "suppression"...

w00t!

Sortition's "proposal":

My proposed system is, thus, to give each author a quota of publications - say one article every 3 years.

In that case, all I need to do jam lots of crap together and call it "one article". Which is, in fact, what Castles and Henderson did in their latest "contribution" to Energy and Environment.

But seeing that Sortition values abstract ideals over facts, I guess that's not much of a problem to him.

dhogaza,

> someone thinks they get to redefine commonly used english words like "censorship" and "suppression"...

It is not a matter of redefining those words - as a look in the dictionary shows. It is a matter of using the words consistently rather than describing the same activity with different words depending on the context: others "censor" or "suppress", we "reject" and "choose not to publish".

bi,

> all I need to do jam lots of crap together and call it "one article".

Why would that be a problem? If you want to waste your publication quota on nonsense that is your loss: your readers will presumably be able to tell that it is nonsense and will move on. I believe most researchers will do their best to produce what they consider high quality work.

Sortition:

It is a matter of using the words consistently rather than describing the same activity with different words depending on the context:

Rubbish. "Censorship" gets rid of all traces of a work in the entire public sphere. A journal "rejecting" doesn't do anything remotely close to that.

If you want to waste your publication quota on nonsense that is your loss:

"One article every 3 years" isn't a "quota" in any real sense. I just need to gather up every piece of half-baked junk I've done in the last 3 years and submit it as a single "article".

your readers will presumably be able to tell that it is nonsense and will move on.

Oh, so even if I still get to publish all the nonsense that I want to publish, even if the "quota" isn't a quota at all, it's still perfectly OK... because at the end of the day, it's the readers' fault anyway.

With this sort of "logic", who needs facts?

> "Censorship" gets rid of all traces of a work in the entire public sphere.

As I already pointed out twice, most people would use the term censorship in a much broader sense. See the examples I gave above in my responses to dhogaza (#18) and to Lee (#24).

> I just need to gather up every piece of half-baked junk I've done in the last 3 years and submit it as a single "article".

I don't see this as much of a problem since most people would not read beyond the first, say 10 pages anyway, so why would you bother? But if you think this is a problem, the quota could have page limit for an article - say 10 or 20 pages. Does this resolve your objection?

> because at the end of the day, it's the readers' fault anyway.

I would say that at the end of the day it should be the readers' choice. The function of the channel (journal, newspaper, TV) should be to reduce the traffic to a level that is comfortable to the readers and do so in such a way as to promote the most illuminating content.

most people

One idiot on the internet is not "most people".

And, yes, in a past life I *was* paid to write non-fiction for national magazines and large dailies and I *do* know what I'm talking about.

And the dictionary definition I provided above, for the word "suppression", makes clear it is you who are misusing the two words in question.

> One idiot on the internet is not "most people".

Ok, how about the ACLU, then: Censorship of Student Art Project Chills Creative Expression. The students in question could find a different venue for their display - yet the ACLU still considers this censorship.

> And, yes, in a past life I was paid to write non-fiction for national magazines and large dailies and I do know what I'm talking about.

I don't see your point.

> And the dictionary definition I provided above, for the word "suppression", makes clear it is you who are misusing the two words in question.

How so? Clearly, what I am talking about falls under the third definition: "To keep from being revealed, published, or circulated."

Ok, how about the ACLU, then: Censorship of Student Art Project Chills Creative Expression. The students in question could find a different venue for their display - yet the ACLU still considers this censorship.

sigh... now you're mixing up your misunderstanding of suppression and censorship, which are not the same thing.

1. A person authorized to examine books, films, or other material and to remove or suppress what is considered morally, politically, or otherwise objectionable.

So you have a university, which has the authority to exam the artwork to be displayed and, if it deems it objectionable, reject it.

As you can see by the definition, rejection because it sucks is not censorship. Censorship is a very specific activity resulting from the work being deemed objectionable.

And don't get too excited about the word "suppression" used in the definition. It's being used in a very narrow way, specific to the venue or means of distribution and the tangible object being censored.

Perhaps you should take your argument over to some blog written by an english prof.

"BTW, I am still waiting for your attempt to reconcile your statement that "everybody agrees" that Israel should withdraw to the 67 borders with Israel's colonization policy of the West Bank."

well, ya caught me. everyone except the people who are colonizing the west bank. and the people who are taking over their apartments when they move there.

PS, since it appears i'll get stuck for this later too, i didn't say "withdraw to the 67 borders"; not a lot of people say that precisely; those are just random places where the opposing forces happened to be when the armistice was signed, not the type of defensible borders which are widely recognized to be required by a state. however, what people realize is that there will be removal of israeli citizens from the west bank, a la gaza, and removal of israeli government support for anyone who fail to be removed, followed by establishment of two states, one consisting of arabs and jews, one nicely judenrein, along an approximation of the 1967 borders, altered according to my first sentence above.

shouldn't this be on the open thread?

ironically, "sensical" is, literally, nonsensical.

The students in question could find a different venue for their display - yet the ACLU still considers this censorship.

The ACLU isn't a dictionary. (And remember, these are the guys who wanted Walter Polovchak to go back to the USSR because his parents have the "right" to decide where he should leave. The ACLU aren't exactly the kind of people I'll go to for definitions.)

But if you think this is a problem, the quota could have page limit for an article - say 10 or 20 pages.

Which rules out the material which do merit a more extensive and more detailed treatment. (As I write this, I'm reading a journal paper which is 34 pages long.)

The ACLU isn't "most people" either.

re the idea of a publication quota.

Isn't that what the internet is for nowadays? This thread is a perfect example. The poll didn't pass whatever review the authors submitted it to, so they published it online. They can do this as often as they want, no need to wait three years and cram everything into one huge paper.

Peer-review adds value. That is why it is desirable to have a paper published in a peer-reviewed paper. Having a paper published is not a right.

z,

> everyone except the people who are colonizing the west bank.

Actually, the colonists (with the exception of a small hard core of messianic ideologues) are merely pawns. The colonization is not a grassroot operation - it is the Israeli government that organizes the colonization, and it is the Israeli government which does not plan on pulling out, ever. If the Israeli government ever decides to change its expansionist policy, a solution to the conflict would be achievable.

> i didn't say "withdraw to the 67 borders"

Not that it matters much, but, yes, you did.

> shouldn't this be on the open thread?

It was, but you offered no response. Thanks for addressing the matter here.

gater,

> Isn't that what the internet is for nowadays?

No. Because there is no limit to the number of publications that a person can put on the web, there is no motivation to reduce quantity in favor of quality. If someone does put a lot of effort into a paper, it is hard to find it within the haystack of uninteresting stuff. As I already mentioned above, the role of a publication venue is to reduce the traffic to a manageable volume, while promoting quality.

Brown et al. were lucky that Tim decided to link to their work - otherwise most likely none of us would have heard of their paper. On the other hand, under the quota system, it is quite possible (if not probable) that Brown et al. would have decided not to waste their precious publication quota on such an unconvincing paper.

> Peer-review adds value.

I argue that an author is in a better position than "peers" (who are really in most cases not peers at all, but an academic aristocracy) to evaluate the quality of his or her work. But reviewers have a positive role to play in the quota system - instead of being gatekeepers, their role is to offer constructive criticism.

> Having a paper published is not a right.

Why not? I see this as a natural part of the right to free speech.

bi,

> The ACLU isn't a dictionary.

> The ACLU isn't "most people" either.

The point is that the ACLU is unlikely to use the word in a sense that most people would find unreasonable.

>> But if you think this is a problem, the quota could have page limit for an article - say 10 or 20 pages.

> Which rules out the material which do merit a more extensive and more detailed treatment. (As I write this, I'm reading a journal paper which is 34 pages long.)

In such a case, the author should publish a 10-20 page short version and refer to his/her website for the extended version. If, after reading the journal version, the readers have a taste for more they will make the effort and go to the website.

> Censorship is a very specific activity resulting from the work being deemed objectionable.

"Objectionable", meaning "undesirable", meaning "not wanted", easily covers "not wanted because it sucks".

But more importantly, anyone engaging in censorship for any reason can easily and credibly claim that the speech in question "sucks". If we accept this as an exception to the censorship definition, any censor can claim not to be engaging in censorship ("that news piece embarrassing one of our major advertisers was rejected because it sucks").

Also, note that you have shifted your grounds. At #17 your argument was that censorship was not taking place as long as I can talk about my ideas to my friends and people passing on the street. At #36, censorship is not taking place as long as the claimed grounds for not publishing is that my article "sucks". It seems that you have fixed the conclusion upfront ("peer-review is not censorship"), and you are engaging in a rationalization process in which you produce arguments to support this pre-conceived conclusion.