Remember Dennis Bray’s useless survey of climate scientists? The URL and password were posted to the climatesceptics mail list, so the results were biased and included responses from people who were not climate scientists. Bray refused to concede that this meant that the survey was hopelessly flawed. Now he has conducted another survey. Bray has avoided the problems of his previous survey by surveying a list of climate scientists he compiled from journal publications and scientific institutions, and only allowing one response per invitation. However, Gavin Schmidt finds that some of the questions are vague and poorly worded.
Bray (and his co-author Von Storch) has a response to Schmidt over at Prometheus. The bit that concerns me is this:
In 2003, in an effort to reach a broader audience it was decided to conduct an on-line survey. The link and password to the survey was distributed on a number of on line lists (CimList for example) to at least limit the responses to members of those lists and not open to the general public. Negative, preemptive comments were almost immediately online (see Useless Online Survey of Climate Scientists). But interestingly enough, there were no overly major differences between the 1996 and 2003 results. In all there were some 500 respondents but response rates were, of course, not possible to calculate. This sampling method is known as saturation sampling. Saturation sampling attempts to survey all identifiable targets and overcomes the lack of reliable sampling frames. The low cost of internet research makes this possible.*
*Turner WJ (1989). Small business data collection by area censusing: a field test of ‘Saturation Surveying’ methodology. Journal of Market Research Society Vol.3, No.2 April 1989.
I would have thought that the fact that he changed his methodology was an implicit admission that his 2003 was flawed, but Bray doesn’t seem to want to concede this. First, my post was not “preemptive”, but was a response to the Telegraph reporting his results like this:
Prof Dennis Bray, of the GKSS National Research Centre in Geesthacht, Germany, submitted results from an international study showing that fewer than one in 10 climate scientists believed that climate change is principally caused by human activity.
As with Dr Peiser’s study, Science refused to publish his rebuttal. Prof Bray told The Telegraph: “They said it didn’t fit with what they were intending to publish.”
Second, “saturation sampling” is a contradiction in terms. If you survey an entire population then you are not sampling.
Third, he did not do saturation surveying since he did not survey all climate scientists. I suppose he could argue that the population he surveyed was “readers of Climlist”, but he reported his results as a survey of climate scientists.
I am dismayed by Bray’s continuing defences of his badly flawed 2003 survey. All he is accomplishing with this is casting doubt on his 2008 survey.