Wegman Report on Hockey Stick

Joe Barton‘s Committee has released a report they commissioned on the hockey stick by Wegman, Scott and Said (WSS). The focus of the report is much narrower than the NRC report and the results are basically a subset of the NRC report. In particular, both reports find that “off-centre” method used in Mann Bradley and Hughes’ 1998 paper (MBH98) tended to produce hockey stick shapes in the first principal component (PC1). Unfortunately, WSS stop there and do not address the question of what difference this makes to the reconstruction (which is not the same as PC1). The NRC panel did address this question and found that it made little difference.

It would be cynical of me to suggest that the terms of reference for WSS were crafted so that WSS would only check this aspect of MBH98 and not whether it made a difference to the reconstruction. Not surprisingly, the usual suspects are using WSS to claim that the hockey stick is shattered into a bazillion pieces. For instance, the Wall Street Journal editorial page:

[WSS's] conclusion is that Mr. Mann’s papers are plagued by basic statistical errors that call his conclusions into doubt. Further, Professor Wegman’s report upholds the finding of Messrs. McIntyre and McKitrick that Mr. Mann’s methodology is biased toward producing “hockey stick” shaped graphs.

Needless to say the NRC panel’s findings are never mentioned in the WSJ editorial. (Nor in the WSS report for that matter.)

Wegman, Scott and Said are statisticians, not climatologists and this has lead to some errors in their interpretation of the literature. For example, the temperature graph in the first IPCC report is schematic and not quantitative, but they interpret it as if it was quantitative.

WSS do have a section that is not a subset of the NRC report — Chapter 5, which claims to present a Social Network Analysis of authorships in temperature reconstructions. They have some pretty graphs but there is no quantitative analysis. It’s possible that the temperature reconstruction community is so small that it causes difficulties with peer review but WSS don’t have any numbers to support this. I think such an analysis would require you to come up with a quantitative measure and compare with other authorship networks.

All in all there seems little reason to refer to this report rather that the NRC one.

But the most serious flaw in WSS is this claim:

This committee does not believe that web logs are an appropriate forum for the scientific debate on this issue.

I mean, really.

Update: Comments from William Connolley and Michael Mann.

Comments

#1 MarkR
July 19, 2006

Hi Tim

From your RealClimate link:

“5) What happens if you just use all the data and skip the whole PCA step?

This is a key point. If the PCs being used were inadequate in characterizing the underlying data, then the answer you get using all of the data will be significantly different. If, on the other hand, enough PCs were used, the answer should be essentially unchanged.”

From the Wegman Report (page 33):

“Figure 4.4: One of the most compelling illustrations that McIntyre and McKitrick have produced is created by feeding red noise [AR(1) with parameter = 0.2] into the MBH algorithm. The AR(1) process is a stationary process meaning that it should not exhibit any long-term trend. The MBH98 algorithm found ‘hockey stick’ trend in each of the independent replications.

Discussion: Because the red noise time series have a correlation of 0.2, some of these time series will turn upwards [or downwards] during the ‘calibration’ period and the
MBH98 methodology will selectively emphasize these upturning [or downturning] time series.”

The point is, because of the Mann computation method, any data series with a correlation of 0.2 or above will produce a Hockey Stick, including Red Noise.

So to that extent, RealClimate are correct. It really doesn’t make any difference at all to the result, as long as at least one of the data series, any data series, has a correlation of above 0.2.

A number of series of random data will produce a hockey stick.

The only problem is that the result will fail the statistical skill test:

“This example illustrates (if the code is correct) a situation, similar to MBH98, where the R2 statistic correctly indicates no statistical skill in the predictions, but the RE statistic erroneously indicates statistical skill.

Conclusions hinge on the choice of statistic and where you set the benchmark. MM05 obtain a critical value for RE of greater than 0.5 using random red-noise data in a replication of the procedure used in MBH98. Non-existent statistical skill of the models is one of the main arguments in MM05 against the reconstruction method in MBH98.”

http://www.climateaudit.org/?p=655

That is why Mann chose to ignore, or hide the R2 test result.

All his results failed this test.

Ever wondered why Mann didn’t just use a simple moving average?
#2 KFL
July 19, 2006

Hi MarkR – a never ending story

If you will spent some time in investigating

a. the memo written by Joe Walker, American Petroleum Institute, 3.april 1998

b. the techniques developed by Edwards L. Bernays – then father of spin and public relation

c. the way M&M and Joe Barton are attacking the Hockey Stick team

d. the way many third party organisations are dealing with Global Warming

you will realize, that you are a victim of a public relation campaign.

This is to me a big surprise to se, that several intelligent and well educated individual are more or less defenceless for such public relation campaigns.

The first six lessons in public relation are:

1)The message you want to convey to the public much be simple, but not necessarily truth

2)The message must be simple to be read, simpel to be understand and soft in language

3)The massage much be understandable for the most stupid 2/3 part of the population

4)The message must focus on the inner devil or engel…

5)The message must not be directly false or at first hand not to be proven to be false

6)The message must be repeated again and again even if it is not correct

To protect the message against being unveiled by knowledgeable individuals or individuals with a high etical standards, a public relation campaign – parallel to the main campaign – will also discredit these individuals.

A public relation campaigns could be adressed to the public, to the politicians , to TV, to Radio or letters to the editors, etc.

The Hockey Stick case fits into this description:

American Petroleims Instituts message:

Global warming is harmles
Cllimate science is uncertain
Kyoto is bad for US and won’t work
To protect their message from beeing unveiled they have a parallel campaign saying:

Michael Mann et al are producing flawed science

The grassroot organisations are manipulation the public

This public relation campain will continue over many years until American Petroleum Institute will realize, that it will be too costly for the shareholders not to fight global warming. Joe Bartons inquiry or similar campaigns will follow 1, 2, 5, 10, 15,.. years from now. A never ending story.

This techniques have been or are being used by the Catholic Church, The facists, The communists, The Nazzi regime, etc. After the 2.world war these techniques have been adopted by the American Corporations, The Neo-conservative, the Bush Administration, The third party organisation like American Petroleum Institute(Exxon Mobile) for god or for bad.

I am not advocating for, that these public relation techniques can’t be used at all. There will be many cases where these methods to some extent could make good sense. However, there are many examples from Corporate America, where the use of these method are similar to a criminal act. The Hockey Stick case is just one example.

I think, this broader view must be included in the Hockey Stick Discussion.
#3 mtb
July 19, 2006

KFL: So far as I can see, your analysis applies more strongly to the hockey team than it does to the sceptics!
#4 Dominion
July 19, 2006

I love that justification! As long as the data improves the statistical validation – INCLUDE IT! – nanny

Oy vey! Then what do you suppose? If the data improves the statistical validation we should IGNORE IT?

So far as I can see, your analysis applies more strongly to the hockey team than it does to the sceptics! – mbt

So why not help us see as far mtb. Do you have anything half as impressive as what Lee has offered? Or just an bald assertion devoid of any evidence? If so, why take it seriously?
#5 James
July 19, 2006

“Dear James, “not using PCA” is not the same as “taking the arithmetic mean”.

Tim, what, then, does “not using PCA” mean? The RC link you provide simple asserts that you get the same result. It says nothing about what this alternative method is. And if it gives the same result, why use PCA at all?

(As a footnote, the Mann method is not really PCA at all, but we can set that aside).
#6 Jimbo
July 19, 2006

nanny_govt_sucks says “trees don’t grow so well in too-hot, or too-cold conditions, thereby making them a poor temperature proxy.’

This statement makes me laugh out loud. Bristlecone pine trees “don’t grow so well” period — even under “ordinary” circumstances.

Anyone who has ever seen Bristlecone pines firsthand atop a windswept mountain ridge in the American southwest will know that they often grow under what can only be termed inhospitable conditions: extremes of heat and cold, poor soil, low moisture, etc.

Some of these trees are thousands of years old and are stunted to the max, often looking more like a bush than a tree, sometimes with only a few live branches.

There may be a legitimate reason not to use bristlcones as a temperature proxy but based on what I know of Bristlecones, I’d have to say that the one given above — “trees don’t grow so well in too-hot, or too-cold conditions” — is most probably not one of them.

Climatically related temperature changes over the past thousand years are undoubtedly much smaller than seasonal (sometimes even daily) temperature changes to which the Bristlecone pines are normally accustomed.

If anything, the bristlecone pines are better adapted than most trees (indeed, perhaps all other living things) to temperature fluctuations. If they were not, they would simply not be able to live as long as they do.
#7 nanny_govt_sucks
July 19, 2006

My comment was about trees in general, not specifically about bristlecone pines.
#8 hank
July 19, 2006

Oh, Tim, in the original post you wrote:

“It would be cynical of me to suggest that the terms of reference for WSS were crafted so that WSS would only check this aspect of MBH98 and not whether it made a difference to the reconstruction.”

No worries. Dr. Wegman has just said this is exactly the limit that was placed on his work and on his presence today in the committee — that’s all he was asked to do and that’s all he is there to do.
#9 Jimbo
July 19, 2006

nanny_govt_sucks says “My comment was about trees in general, not specifically about bristlecone pines.”

That may be, but much of the criticism of using trees as temperature proxies is aimed specifically at the Bristlecones and the first post above where you made the point that “too hot or too cold conditions inhibit tree growth” was a response to/reinforcement of a post by MarkR, who states categorically that “bristlecones are not a temperature proxy”.

What matters is not whether trees “generally” are not a good temperature proxy but whether the specific trees being used are good temperature proxies.

I’m not a tree expert, so I’ll have to let those who are make the final judgement in that regard.
#10 MarkR
July 19, 2006

Hi Jimbo

It just means that Bristlecones are doubly useless as temperature proxies.
#11 James
July 19, 2006

Jimbo,

Well, we could look at the peer-reviewed literture by the “dendro guys”.

Graybill & Idso (1993) – who collected much of the bristlecone data used by MBH – concluded that the 20th century growth pulse could not be explained by temperature, and speculated about C02 fertilisation as the cause. Hughes (yes, the Hughes in MBH) stated that the anomolous 20th century growth was “a mystery” (Hughes & Funkhouser, 2003)
#12 Dano
July 19, 2006

MarkR:

would you like to take “your” factless argument to a dendro listserv to try to defend “your” argument?

That is: do you have the courage of “your” convictions to defend them to folk who do this for a living?

Let me know.

James:

Hughes …stated that the anomolous 20th century growth was “a mystery” (Hughes & Funkhouser, 2003)

Hence the instrumental record.

If you have citations from empirical evidence that the calibration is off because of x,y,z, please cite them.

———-

Can’t wait for the astroturfers energy to wane!

Best,

D
#13 Jimbo
July 19, 2006

MarkR says: “It just means that Bristlecones are doubly useless as temperature proxies.”

You might be right, but I’ll reserve judgement on that until I have read the papers referenced above by James and some more recent papers on the subject as well.
#14 caerbannog
September 7, 2006

Folks who are confused about the “hockey-stick” wars and who want to see how deniers “pull fast ones” should google up a PDF copy of the Wegman Report and have a look at figure 4.1. A major argument used against Mann et al is that Mann’s data-centering convention “mines” noisy data for “hockey stick” leading principal components. To make that case, M&M generated a big set of random noise time-series and computed principal components from it using Mann’s data-centering convention. And yes, in many cases,leading principal components computed from this sort of random noise do have that “hockey stick” shape. But there’s a *big* catch here, and someone with sharp eyes should have no trouble spotting it.

To see what I mean, check out the Wegman Report Figure 4.1. Figure 4.1 shows Mann’s “hockey-stick” plotted right next to a “noise-only” hockey-stick. They look pretty similar, don’t they? Looks pretty bad for Mann, doesn’t it? But take a closer look at fig 4.1 — in particular, look at the Y-axis scales of the two “hockey-stick” plots. You’ll see something **very** fishy.