Remember Dennis Bray's useless survey of climate scientists? The URL and password were posted to the climatesceptics mail list, so the results were biased and included responses from people who were not climate scientists. Bray refused to concede that this meant that the survey was hopelessly flawed. Now he has conducted another survey. Bray has avoided the problems of his previous survey by surveying a list of climate scientists he compiled from journal publications and scientific institutions, and only allowing one response per invitation. However, Gavin Schmidt finds that some of the questions are vague and poorly worded.
Bray (and his co-author Von Storch) has a response to Schmidt over at Prometheus. The bit that concerns me is this:
In 2003, in an effort to reach a broader audience it was decided to conduct an on-line survey. The link and password to the survey was distributed on a number of on line lists (CimList for example) to at least limit the responses to members of those lists and not open to the general public. Negative, preemptive comments were almost immediately online (see Useless Online Survey of Climate Scientists). But interestingly enough, there were no overly major differences between the 1996 and 2003 results. In all there were some 500 respondents but response rates were, of course, not possible to calculate. This sampling method is known as saturation sampling. Saturation sampling attempts to survey all identifiable targets and overcomes the lack of reliable sampling frames. The low cost of internet research makes this possible.*
*Turner WJ (1989). Small business data collection by area censusing: a field test of 'Saturation Surveying' methodology. Journal of Market Research Society Vol.3, No.2 April 1989.
I would have thought that the fact that he changed his methodology was an implicit admission that his 2003 was flawed, but Bray doesn't seem to want to concede this. First, my post was not "preemptive", but was a response to the Telegraph reporting his results like this:
Prof Dennis Bray, of the GKSS National Research Centre in Geesthacht, Germany, submitted results from an international study showing that fewer than one in 10 climate scientists believed that climate change is principally caused by human activity.
As with Dr Peiser's study, Science refused to publish his rebuttal. Prof Bray told The Telegraph: "They said it didn't fit with what they were intending to publish."
Second, "saturation sampling" is a contradiction in terms. If you survey an entire population then you are not sampling.
Third, he did not do saturation surveying since he did not survey all climate scientists. I suppose he could argue that the population he surveyed was "readers of Climlist", but he reported his results as a survey of climate scientists.
I am dismayed by Bray's continuing defences of his badly flawed 2003 survey. All he is accomplishing with this is casting doubt on his 2008 survey.
I havea go here: http://scienceblogs.com/stoat/2008/10/revenge_of_the_pollsters.php
Boy, I sure am glad RP Jr provides a forum for even-handed and balanced analysis of current climate issues, aren't you? Friends, that is balance we can live with, you betcha golly!
I wonder what would happen if someone got some funding to do a real survey...would denialists' heads explode?
Best,
D
Actually, here is what seems to me like the most reliable survey that I've seen, although still not without its problems: http://stats.org/stories/2008/global_warming_survey_apr23_08.html It is not just of climate scientists though (despite the wording of their report) since it is a random sample of those people in the current edition of American Men and Women of Science who list themselves as members of the AGU and the AMS. Unfortunately, they do not seem to provide all the details of the survey such as a list of all the questions that were asked.
Hi Joel: Interesting article - thanks. If anyone is interested in reading it, a better link can be found here.
Best,
John
Bray has a tantrum:
My foray into blogworld has been what ... amusing I guess. But I must bid farewell to the Hatfields and the McCoys. Alas, I must return to more responsible duties - I am milk monitor of the week!
Before I go I would like to offer a little prayer, penned by Robertson Davies long before Digital Daze. It goes 'God give me oblivion from the small small voices of small small people'
http://www.realclimate.org/index.php/archives/2008/09/a-new-survey-of-s…
Why is that when garbage on climate science like Bray's response ends up on the internet, it almost always finds its home on Roger Pielke Jr.'s blog?
Thom,
I think that you just answered your own question....
And here is an easier read for William's link above.
And Bray attempts some Rabett carving but stabs himself opening the milk carton.
Seriously, this is one of the worst surveys ever, other than the usual student evaluations of teaching.
I think all that needs to be said has been said (very politely!) on the RealClimate thread, but I do have to concur with Eli's conclusion at #9 above.
I've constructed multiple-choice opinion surveys myself, and I claim no expertise in this at all, but really - some of those questions are just sheer clumsiness. Especially so if it's not the authors' first time asking them. Taking the point that Eli asked of Bray several times at RealClimate, wouldn't (shouldn't) a surveyor's alarm bells ring if 50% or more of respondents chose 'other' as their answer for a question?
Bray said over there:
Yes, it has been amusing, but far more so for others than himself.
Then he said:
Ah, an attempt at humour to disguise his avoidance of the real points that remain to be adequately addressed.
Alas, it only serves to make Bray look like an ass.
Does hee-haw..lways look thus?
Thankyou for the offer Dennis. I accept it and I will use it, starting with your 'survey'.