Quote mining code

David Kane asks me to look at two of the strongest arguments made by the “other side” following the break in and theft of data from CRU. OK, once he sees how weak the strongest arguments are, we can all agree that the affair is a beat up.

Today I’ll look at Eric Raymond’s alleged “siege cannon with the barrel still hot”:

From the CRU code file osborn-tree6/briffa_sep98_d.pro , used to prepare a graph purported to be of Northern Hemisphere temperatures and reconstructions.

;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,- 0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,'Oooops!'
;
yearlyadj=interpol(valadj,yrloc,timey)

This, people, is blatant data-cooking, with no pretense otherwise. It flattens a period of warm temperatures in the 1930s — see those negative coefficients? Then, later on, it applies a positive multiplier so you get a nice dramatic hockey stick at the end of the century.

But what is the code directly following the fragment Raymond quotes? Look:

;
;filter_cru,5.,/nan,tsin=yyy+yearlyadj,tslow=tslow
;oplot,timey,tslow,thick=5,color=20
;

IDL uses a semi-colon to indicate a comment, so the only code to use yearlyadj has been commented out. Raymond must have known this since he is an Emacs user and Emacs colour codes the comments. This doesn’t seem to be a smoking gun so much as a gun that hasn’t been fired.

Furthermore, another piece of commented out code shows how the adjusted values were used: they were clearly labelled as “MXD corrected” and plotted along with the uncorrected values. They were not shown as temperature values despite what Raymond says.

;legend,['Northern Hemisphere April-September instrumental temperature',$
;  'Northern Hemisphere MXD',$
;  'Northern Hemisphere MXD corrected for decline'],$

In the comments to Raymond’s post, others pointed this out to him

As other have repeatedly pointed out, that code was written to be used for some kind of presentation that was false. The fact that the deceptive parts are commented out now does not change that at all.

In fact, it was it was labelled as “corrected for decline”, so it was not false or deceptive.

Raymond continues:

It might get them off the hook if we knew — for certain — that it had never been shown to anyone who didn’t know beforehand how the data was cooked and why. But since these peiple have conveniently lost or destroyed primary datasets and evaded FOIA requests, they don’t deserve the benefit of that doubt. We already know there’s a pattern of evasion and probable cause for criminal conspiracy charges from their own words.

In fact, they did not destroy primary datasets, and they did not have permission to redistribute the data requested using the FOIA.

Raymond has made no attempt to find out if the graph was actually used anywhere. The file name was osborn-tree6/briffa_sep98_d.pro, so we should look for a paper with authors, Briffa and Osborn published in 1998 and sure enough there’s Briffa, Schweingruber, Jones, Osborn, Harris, Shiyatov, Vaganov and Grudd “Trees tell of past climates: but are they speaking less clearly today?” Phil. Trans. R. Soc. Lond. B 1998:

In §4, we referred to a notable correspondence between
‘hemispheric’ MXD series (averaged over all sites) and an
equivalent ‘hemispheric’ instrumental temperature series.
Despite their having 50% common variance measured
over the last century, it is apparent that in recent decades
the MXD series shows a decline, whereas we know that
summer temperatures over the same area increased.
Closer examination reveals that while year-to-year
(i.e. mutually ten-year high-pass filtered) correlations are
consistently high between tree-growth and temperature
(ca. 0.7 for 1881-1981), the correlations based on decadally
smoothed data fall from 0.89, when calculated over the
period 1881-1960, to 0.64 when the comparison period is
extended to 1881-1981. This is illustrated in figure 6,
which shows that decadal trends in both large-scale-
average TRW and MXD increasingly diverge from the
course of decadal temperature variation after about 1950
or 1960.

And figure 6 is basically the graph plotted by the code above and it does not include the “corrected MXD” data:

i-f4d73c2366292d1fb592d8ac4534e05b-Briffa98fig6.png

Oh, and Raymond reckons the greens are controlled by commie puppet masters:

Most of the environmental movement is composed of innocent Gaianists, but not all of it. There’s a hard core that’s sort of a zombie remnant of Soviet psyops. Their goals are political: trash capitalism, resurrect socialism from the dustbin of history. They’re actually more like what I have elsewhere called a prospiracy, having lost their proper conspiratorial armature when KGB Department V folded up in 1992. There aren’t a lot of them, but they’re very, very good at co-opting others and they drive the Gaianists like sheep.

There’s more paranoid raving, but you get the picture — Raymond is the sort of person who will add 2 and 2 and come up with a commie plot.

Comments

  1. #1 dhogaza
    December 1, 2009

    Apparently similar code appears in another snippet the results of which weren’t commented out, but still, it doesn’t matter.

    People trying to understand data tweak its processing all of the time. It’s perfectly normal behavior when you’re asking “what if?” scenarios. Someone proposes a corrective factor, presumably based on some plausible snippet of knowledge, applies it to the data to see how that changes things, etc.

    I’ve been involved for years with monitoring raptor migration at various sights in the semi-arid western United States. Some people were using our hourly reports to try to figure out how local geography affected the daily pattern of migration at two sites about 200 or 300 miles apart, east-west (migrating hawks roost at night, how far north from a site they roost impacts how early in the day they start showing up at a given site).

    They couldn’t get the times to jive with the distances, so finally I asked, “did you correct the times for the site positions within the timezone?”. Local sun time was something like 30 or 45 minutes different.

    Now, I’m not sure what the little program they wrote looked like for certain, but if Eric Raymond were to see a line saying “fudge site A times by 30 minutes” I’m sure he’d scream “FRAUD! FRAUD! FRAUD!”.

    I don’t doubt it for a moment.

    These people aren’t interested in investigating *why* a researcher might be trying out various ideas on the data, in their minds, if they have a program showing adjustments then IT IS ALL A COMMIE PLOT!

    It’s really insane, people.

  2. #2 Greg Laden
    December 1, 2009

    Nicely done. Thanks.

  3. #3 Boris
    December 1, 2009

    IT looks like they are onto our dream of restoring the Soviet Union, comrades.

  4. #4 Brian D
    December 1, 2009

    I made the same observation here a few days ago, although I was a bit off since I thought he was referencing version D when he was talking about version E.

  5. #5 ali baba
    December 1, 2009

    heh, socialism rescued capitalism from itself last century and environmentalism will rescue it again this century, libbies need to learn gratitude and to stop shitting their bed.

  6. #6 Betula
    December 1, 2009

    I thought it might be nice to lift the shades and open the windows for some of the commenters here.
    This will allow you a glimpse of the world outside your box, while giving you a bit of cool fresh air at the same time.

    Follow the money:

    http://online.wsj.com/article/SB10001424052748703939404574566124250205490.html

    Now, back to your regular prophesying.

  7. #7 Bud
    December 1, 2009

    Nicely done Tim.

    “This doesn’t seem to be a smoking so much as a gun that hasn’t been fired.”

    I think you’re missing a weaponry-related noun in that sentence.

  8. #8 Harald Korneliussen
    December 1, 2009

    I have argued elsewhere that if you’re going to falsify results, then

    ; Apply a VERY ARTIFICAL correction for decline!!

    isn’t the sort of thing you’d stick in your code. In fact, a comment like that is the kind you make when you want to _avoid_ that snippet accidentally ending up in production code.

    Raymond, of all people, should have known this. He must have disabled his code-writing psychology skills as well as his syntax highlighting, in order to find evidence for his commie plot theory.

    (OT, do you remember that this gun-toting Pan worshiper thought he would save free software from the bad PR management of RMS?)

  9. #9 Jody
    December 1, 2009

    Is it just me, or did Betula not follow a word of Mike’s post?

    As I forget the proper name for the paper describing the ignorant measure of one’s own expertise, I’ve started referring to it as “The Freddy Krueger Paper.”

    Seems apt, here.

  10. #10 Rob Jase
    December 1, 2009

    “There’s a hard core that’s sort of a zombie remnant of Soviet psyops. Their goals are political: trash capitalism, resurrect socialism from the dustbin of history”

    Because the folks who brought us Chernobyl were clearly ecologically friendly.

  11. #11 Eric Lund
    December 1, 2009

    @Jody: You’re probably thinking of this paper: J. Kruger and D. Dunning (1999), Unskilled and unaware of it, Journal of Personality and Social Psychology 77, 1121. Which definitely applies to Mr. Raymond in this case.

    As anyone who has worked with data collected in the field knows, sometimes you have to figure out why that sensor is giving you strange results, and often you come up with an empirical fix before you figure out why that fix works. As Tim said in the post, there is nothing wrong with doing this as long as you explain what you are doing and why.

  12. #12 carrot eater
    December 1, 2009

    The above would still hold even if those lines weren’t commented out in whichever file they found; the true test is seeing what was actually presented to others, and how (as you indeed conclude).

    In the process of doing work, I’ll also sometimes apply ‘artificial’ manipulations, just to see what happens. I’ll also have multiple versions of code lying around, as I make various updates and try different things. Do I need to stop doing that, in case somebody gets a hold of some code and misunderstands what he’s looking at?

  13. #13 William Wallace
    December 1, 2009

    The code is commented out. So? Explain why it was ever written, and when it was commented out. Have these clowns ever heard of CVS, VSS, or sundry other tools to track code changes?

    You really wouldn’t need CVS, though, if you never intend to release the code.

    There’s more paranoid raving, but you get the picture — Raymond is the sort of person who will add 2 and 2 and come up with a commie plot.

    Now that takes balls, almost as much as the Gay and Lesbian Victory Action fund (a group with a homosexual agenda) attacking Sally Kern of OK for talking paranoid about a homosexual agenda.

    LOL. Sometimes I wonder if the left actually believes their own lies.

  14. #14 Neil
    December 1, 2009

    > Because the folks who brought us Chernobyl were clearly ecologically friendly.

    Quite; And he’s obviously never heard how the environmental movement defied the soviets in Estonia and Latvia. Or, at least, he’s ignored those events.

  15. #15 mb
    December 1, 2009

    If you never expected your code to leave the confines of your Soviet-built underground conspiracy-bunker, then the comment could be innocuous; serving to admonish against further alterations to prevent being too obvious.

    Plus it’s not up to Eric to show that anything was ever published using that specific plot. Instead it’s up to everyone else to prove that not once, anywhere, did anyone use such a plot to convince someone–perhaps an influential homeless man–that Al Gore should receive the Nobel Fields Medal and thus bring about end of the long and prosperous era of global laissez-faire capitalism, using the same underhanded methods so-called “public health experts” tried with the “throwing feces and garbage into the streets causes disease” scam. After all when looking at evidence from people who use a “trick” to “hide the decline” the onus isn’t on those of us that pretend to be a programmers, generals, freedom fighters, and sexperts, but rather on everyone that doesn’t suffer from Narcissistic Personality Disorder to prove that there was no wrong-doing.

  16. #16 Dave X
    December 1, 2009

    CVS sucked in 1998, which is why people use commented bits in script-chunks in a text file notebook to create repeatable graphics for publication.

  17. #17 mb
    December 1, 2009

    Ah, I also see that people are missing the point of the Soviet infiltration of the watermelon patch. It’s not because they are fond of the environment, but rather it’s to introduce inefficiencies such as sustainable resource exploitation to the Free World(tm) so that they have the competitive advantage they need to retroactively sustain the Soviet Union.

  18. #18 Joseph
    December 1, 2009

    Instead of “VERY ARTIFICIAL … !!” they should’ve added a comment like “THIS CORRECTION IS PART OF OUR COMMIE PLOT TO RULE THE WORLD!!”

    It would’ve made Raymond’s job that much easier.

  19. #19 Joseph
    December 1, 2009

    The code is commented out. So? Explain why it was ever written, and when it was commented out.

    Clearly it was written to see what the tree-ring temperature plot would look like with that artificial correction given by an array of 20 offsets. The tree-ring series diverges (declines) from a real temperature series, starting in the 60s or 70s. They admit to that in the paper Tim quoted.

    Have these clowns ever heard of CVS, VSS, or sundry other tools to track code changes?

    Unless they are software developers, there’s not much of a reason they should. It would be cool if they made everything open source and accessible via CVS. (I’d rather they use CVS – makes it easier to get it through Eclipse.)

  20. #20 jre
    December 1, 2009

    Like most of us with some interest in the world of free software, I’ve been more or less aware of esr’s doings for some years now. I can tolerate his personality just fine as long as I believe in his cause — e.g. the Just War against SCO. But his pose of the righteous truth-pursuer wore thin long ago, and now he is just deranged.

    To see how far this once-respected mind has slipped into self-delusion, we only need to reflect on the fact that one of the four or five best-known hackers in the world:

    1) couldn’t tell the difference between IDL and Fortran,

    2) didn’t know when a line of code was commented out, and

    3) thought that “Apply a VERY ARTIFICAL correction” in a comment was a self-damning admission.

    I’m sad to note — because I once admired esr even when I disagreed with him — that he has lost it.

  21. #21 ali baba
    December 1, 2009

    Follow the money:

    What a stupid article, most so-called green funds are regular business investments by regular greedy bastards who’d soak the planet in kerosene and light a match if there was profit in it. They are insignificant compared to non-green funds. The rest is funding for research. The only apt comparison is between funding climate science and funding it’s denialism, and only an imbecile would compare these two on an absolute basis: the cost of paying someone to lie in editorials is basically his salary, which is more than a scientist makes but nothing compared to the cost of scientific research, what with the cost of vexatious FOIA requests and everything.

  22. #22 sod
    December 1, 2009

    Phil Jones steps aside, until investigation is finished.

    among the first comments on this subject on [Wattsup?](http://wattsupwiththat.com/2009/12/01/phil-jones-steps-down/)

    Goal!

    nice people, really.

  23. #23 D. C. Sessions
    December 1, 2009

    WRT CVS:

    I learned a long time ago that keeping my one-off hacks under revision control saved me a boatload of grief and overall made me more productive — even as a solo SPICE coder.

    I also confess that I’m much better at offering this advice than at following it.

  24. #24 William Wallace
    December 1, 2009

    sod, great news, thanks. I am very glad that the creepy Phil Jones did the right thing.

  25. #25 dhogaza
    December 1, 2009

    Pffft, witch hunt rewarded. Expect attacks on other climate scientists to escalate.

  26. #26 MarkB
    December 1, 2009

    Re: #23,#25

    One down (temporarily). Thousands more for denialists to slander.

    http://www.eecg.utoronto.ca/~prall/climate/climate_authors_table.html

    Dr. Liss appears to be the temporary acting director while the investigation is conducted. He co-authored an interesting recent paper with Lovelock.

    Climate change: the effect of DMS emissions
    Liss, Peter S | Lovelock, James E
    Environmental Chemistry [Environ. Chem.]. Vol. 4, no. 6, pp. 377-378. 2007.

  27. #27 Alex
    December 1, 2009

    Because the folks who brought us Chernobyl were clearly ecologically friendly.

    Don’t forget the Aral Sea!

  28. #28 Douglas Watts
    December 1, 2009

    Is “trash capitalism” the guy who comes by the house and asks for your returnable bottles?

    I need to re-read the conspiracy memos Soros sends me.

  29. #29 Joseph
    December 1, 2009

    Phil Jones steps aside, until investigation is finished.

    I’m actually not surprised. I had commented elsewhere that Phil Jones’ apparently tends to say things that can be easily misinterpreted, and just blurts out the wrong stuff.

    The “trick to hide the decline” comes from him. The correct characterization would be “artificial correction for the divergence.” He’s also the one who suggested some emails should be deleted (even though that was apparently not carried out.) Additionally, he said he’d rather hide behind data protection laws than comply with FOI requests.

    Again; not necessarily evidence of wrongdoing, but some of his behavior has not been very professional.

  30. #30 caerbannog
    December 1, 2009

    Just thought I’d quote-mine some comments in the Linux source-tree code to show folks how much Linux sux:

    Fucking broken ABI
    IOC3 is fucked fucked beyond believe …
    brain-damage, it’s managed to fuck things up one step further..
    What the fuck is going on here? We leave junk in the beginning
    Shit happens..
    all the algo is pure shit and should be replaced
    THIS IS A PIECE OF SHIT MADE BY ME
    long delays in kernel code are pretty sucky anyway
    setup the pointer arrays, this sucks
    loose some. This sucks :-(
    It sucks. I totally disown this
    extern calls and hard coded values here.. very sucky!
    assume we found an overflow. This sucks.
    performance sucks for guests using highmem.
    This sucks, but it is the best we can do..
    this sucks [tm] :-(
    XXX I know this sucks
    This SUCKS.
    method really sucks. You can only read or write one location at a time
    SuckyIO interrupt routing for PICs on function 1
    This sucks. There is a better wa
    TODO: use a hash or array, this sucks.
    This sucks, and it is a hack
    Sucks! We need to fork list.
    Things still suck.
    Note that the arbiter/ISA bridge appears to be buggy
    Disable arch_idle() by default since it is buggy
    really buggy
    something weird, or if the code is buggy
    support for disabling the buggy read-ahead
    Some old kernel bugs returned
    The original driver looks buggy/incomplete
    That turned out to be too buggy to support
    give_up_console() is obviously buggy as it
    This feature appears to be buggy.
    usb-uhci seems buggy for async unlinking
    crap – we crashed before setup_arch()
    This piece of crap needs to disappear
    That’s crap, since doing that while some partitions are opened
    One more crappy race: I don’t think we have any guarantee here
    Piles of crap below pretend to be a parser for module and kernel
    this bit masking stuff is crap.
    horrible the crap we have to deal with is when we are awake
    What is all this crap for?
    Locking and life cycle management is crappy still.
    This is a crappy interface.
    This means that the ip6tables jump stack is now crap.
    XXX fix this crap up
    totally crap,
    FIXME: get rid of this crap
    useless crap (ugh ugh ugh).
    This is such a hack
    So, here’s this grotty hack… :-(
    So, here’s this additional grotty hack… :-(
    UGLY HACK: workaround regulator framework bugs.
    Wheee, hackady hack
    Following is a work-around (a.k.a. hack)
    ugly hack, I can’t find a way to actually detect the disk
    Crude hack to get console output
    To keep this hack from interfering
    very hacky
    This #define is a horrible hack
    Hack alert !
    it’s a bit of a hack.
    It’s still quite hacky,
    The hack below stinks…
    Now do the horrible hacks
    One gross hack
    So we use a hack,
    This looks like a dirty hack to me
    This is an ugly hack,
    Hack warning
    It’s a gross hack,
    XXX sleazy hack:
    cheap hack to support suspend/resume
    FIXME: this hack is definitely wrong
    whacked out.
    The following is just a hack
    a hack :-(
    My guess is that this is a hack to minimize the impact of a bug
    vt.c for deeply disgusting hack reasons
    Another Hack :-(
    Brutally hacked
    HACK WARNING!!
    HACK ALERT:
    FIXME: temporary hack
    FIXME: this is a hack -
    nstead we’re going to do a total hack job for now
    dirty hack time.
    THE PADDING THIS STARTS WITH IS A HORRIBLE HACK THAT SHOULD NOT LIVE
    Right now we use a sleazy hack
    which is an ugly hack.

    And this just goes on and on and on….

    All it takes is a few minutes of grepping/quote-mining Linux source-code comments to prove that Linux is inferior to Windows! ;)

  31. #31 dhogaza
    December 1, 2009

    I agree with you regarding the FOI requests, but in this:

    The “trick to hide the decline” comes from him. The correct characterization would be “artificial correction for the divergence.”

    “decline” is synonymous with “divergence problem”, i.e. it refers to the recent decline in MXD growth in certain tree ring data.

    Unfortunately the denialsphere has jumped on this by interpreting it as being “hiding a real decline in real temperature the last few decades”, but they know they’re lying.

  32. #32 dhogaza
    December 1, 2009

    caerbannog: For a good time, post that over at Eric Raymond’s blog … seriously!

  33. #33 sharky
    December 1, 2009

    I’m frankly lost and hoping someone can help me out a little. It’s about this article, in which the raw data from climate change at a university has been lost for about thirty years:

    http://www.timesonline.co.uk/tol/news/environment/article6936328.ece

    How many of the projections of global warming are/were made with data that turns out to have been processed, not raw? How many organizations still have all the raw data from 150 years back or longer that was dumped by that university?

    I haven’t seen that article mentioned anywhere, but since it’s been presented as another DEATH OF ANTHROPOGENIC CLIMATE CHANGE!!!111 argument, I thought this might be a relevant place to ask. (I know I might come off as a concern troll; educational flaming welcomed.)

  34. #34 Betula
    December 1, 2009

    “Pffft, witch hunt rewarded. Expect attacks on other climate scientists to escalate.”

    dhogaza,

    Hope and Change…

    http://www.usnews.com/blogs/paper-trail/2009/11/30/penn-state-will-investigate-climategate.html

  35. #35 MarkB
    December 1, 2009

    Funny how denialists assume an investigation is proof of guilt (climate scientists are hoax perpetuators until proven otherwise). They will praise such investigations until they reveal nothing wrong was done, in which case they will claim the universities are part of the conspiracy.

  36. #36 dhogaza
    December 1, 2009

    How many of the projections of global warming are/were made with data that turns out to have been processed, not raw?

    Hopefully all of them. Ironically denialists are burning both ends of the candle here:

    1. They’ve long said that surface stations exaggerate warming due to the urban heat island effect and other problems. Correct for this, they say, and warming will disappear.

    2. Products like GISTEMP include algorithms intended to counter the UHI and other siting issues. Denialists say “but that’s using adjusted data, you need to use the raw temp data, then the warming will disappear because the adjustments are used to show warming that doesn’t exist!”

    They can’t have it both ways.

    How many organizations still have all the raw data from 150 years back or longer that was dumped by that university?

    That’s not CRUs problem. If you burn me a copy of a CD knowing that I make no promise to keep it forever, and you destroy your CD, it would be a bit odd if you’d blame me for its disappearance.

    But apparently this is what is being done regarding CRU.

  37. #37 Lars Karlsson
    December 1, 2009

    Sharky,

    Check here:
    http://www.eenews.net/public/Greenwire/2009/10/14/3

    ‘Refuting CEI’s claims of data-destruction, Jones said, “We haven’t destroyed anything. The data is still there — you can still get these stations from the [NOAA] National Climatic Data Center.”‘

  38. #38 Lars Karlsson
    December 1, 2009

    The irony:

    “The research unit has deleted less than 5 percent of its original station data from its database because the stations had several discontinuities or were affected by urbanization trends, Jones said.”

  39. #39 dhogaza
    December 1, 2009

    Funny how denialists assume an investigation is proof of guilt (climate scientists are hoax perpetuators until proven otherwise). They will praise such investigations until they reveal nothing wrong was done, in which case they will claim the universities are part of the conspiracy.

    They also haven’t thought through the fact that UEA will have the full record of correspondence rather than the selected set distributed by those that stole the file or files.

    And that Penn will have the same regarding Mann’s correspondence with CRU.

  40. #40 Douglas Watts
    December 1, 2009

    I can only imagine if these folks had stolen Einstein’s scraps of paper when he worked at the Swiss patent office.

    See, he crossed out that equation. It’s on the paper, next to the coffee stain !!!

    We demand the troof about the gravity !!!

    BTW: nice dissection, Tim. Thanks.

  41. #41 Bruce Sharp
    December 1, 2009

    You know, I was just looking Eric Raymond’s source code for [ssh-installkeys](http://www.catb.org/~esr/ssh-installkeys/). I’m horrified, horrified, I say! There are things like this:

    “I can’t figure out how to check if a process is alive on Solaris.”

    and…

    \# This means (I think) that the Solaris process is “defunct”
    \# and a second wait must be called without the WNOHANG.
    \# This is dangerous because if I am wrong then this could block.

    What kind of programmer does things that are DANGEROUS BY THEIR OWN ADMISSION? What kind of programmer ADMITS THAT HE DOESN’T EVEN UNDERSTAND THE OPERATING SYSTEMS THAT MAY BE RUNNING HIS CODE?? I believe this calls for an investigation. How can we trust ANYTHING that Raymond has touched? We can’t.

    This is what happens when software is written by people who aren’t trying to make a profit. You get substandard, DANGEROUS code. Thank heaven we have brave people like the Heartland Institute, standing firm against these open-source communists.

  42. #42 dhogaza
    December 1, 2009

    Yes, Lars, those commie bastards have been leaving out stations that would tend to exaggerate the warming trend.

    Denialists are outraged! Because, as usual, they don’t understand …

  43. #43 el gordo
    December 1, 2009

    It’s hard to fathom why the ABC and The Age don’t run a story on ClimateGate if it’s so innocuous. Rupert’s Wall St Journal isn’t afraid to spell out the ramifications, but for aunty its all a bit much.

  44. #44 dhogaza
    December 1, 2009

    Thank heaven we have brave people like the Heartland Institute, standing firm against these open-source communists.

    In ESR’s case, I think open-source fascist is more accurate …

  45. #45 Oroboros
    December 1, 2009

    I just sent this article to a friend who writes IDL for “correcting” temperature and humidity data. I suggested some kind of version control to him a year or two ago. One of these days he’ll have a need and we’ll discuss it again. I definitely agree that in 1998 CVS just wasn’t feasible for him or most people. My company didn’t get into it until at least 5 years later. We’re still looking at distributed CVS as a replacement for svn, though not aggressively because code isn’t our main product but just a tool to get the other jobs done (networking mostly).

    A lot has changed in 11 years since the suspect IDL code was written. I appreciate the iterative nature of the field but many don’t have the same history as I do. We had to re-use a lot of code before we figured out a solution which was Object Oriented programming. We had to run into concurrency issues in collaborative projects to figure out the CVS solution. My friend works on his code alone most of the time, so the real benefits to him are somewhat questionable compared to the overhead of setting it up.

  46. #46 William Wallace
    December 1, 2009

    What kind of programmer does things that are DANGEROUS BY THEIR OWN ADMISSION? What kind of programmer ADMITS THAT HE DOESN’T EVEN UNDERSTAND THE OPERATING SYSTEMS THAT MAY BE RUNNING HIS CODE?? I believe this calls for an investigation. How can we trust ANYTHING that Raymond has touched? We can’t.

    Your analogy might be stronger if ssh were an organization that all but lobbies for command and control economies.

  47. #47 Alex
    December 1, 2009

    Produce the data and the algorithms or shut-up.

    The dog ate my data doesn’t cut it. This train is wrecking, I suggest you get off at the next stop or you’ll go tumbling down the gorge with it. There will be a new blacklist, and those who defend too much and too often will be remembered.

    Trust is gone forever, at least during your lifetime.

  48. #48 D. C. Sessions
    December 1, 2009

    They can’t have it both ways.

    I do so love innocence. It makes the neatest sounds as it’s crushed.

  49. #49 Dave Andrews
    December 1, 2009

    dhogaza et al,

    Why clutch at straws here?

    The fact is Jones and CRU have no idea how they processed the data in the early years and what adjustments they might have made. Witness Tom Wigley’s anxious email to Jones about whether he might be asked to provide his code. Jones said the fact that Wigley had retired and was no longer a CRU employee was something he could ‘hide behind’.

    Jones also said at one point that, to paraphrase, ‘GISS was crap’. None of his correspondents appeared to take him up on that and GISS has been shown elsewhere to mirror the problems now surfacing from CRU.

    So we now know that both surface temperature records are badly flawed but you still expect us to make extremely important and costly decisions based upon them?!!!!

  50. #50 Jimmy Nightingale
    December 1, 2009
  51. #51 Harald Korneliussen
    December 1, 2009

    caerbannog: That’s – that’s IMPOSSIBLE! Because I KNOW for a fact that the Linux kernel uses revision control, and William Wallace says people who use revision control NEVER leave such comments in their code!

  52. #52 Harald Korneliussen
    December 1, 2009

    Bruce Sharp:
    “What kind of programmer does things that are DANGEROUS BY THEIR OWN ADMISSION? What kind of programmer ADMITS THAT HE DOESN’T EVEN UNDERSTAND THE OPERATING SYSTEMS THAT MAY BE RUNNING HIS CODE?? I believe this calls for an investigation. How can we trust ANYTHING that Raymond has touched? We can’t.”

    The argument is faulty, but this time the conclusion isn’t ;)

  53. #53 winnebago
    December 1, 2009
  54. #54 el gordo
    December 1, 2009

    What’s with the caps? No need to shout.

    Phil Jones steps down to await the outcome of an inquiry.

    http://wattsupwiththat.com/2009/12/01/phil-jones-steps-down/

  55. #55 Bruce Sharp
    December 1, 2009

    Re William in #46:

    Didn’t you know that the goal of open source software is to control our lives, and crush our freedoms? Don’t take my word for it. [Surely you saw this, on the Heartland Institute's site:](http://www.heartland.org/full/26205/Net_Neutrality_Is_Socialism_Not_Freedom.html)

    Eben Moglen’s 2003 treatise The dotCommunist Manifesto is more honest about the thinking behind net neutrality — it’s sprinkled throughout with the language of communism’s great and bloody revolutionaries. The people must “struggle” to “wrest from the bourgeoisie, by degrees, the shared patrimony of humankind” that has been “stolen from us under the guise of ‘intellectual property.’ ”

    The Heartland Institute concluded: “Net neutrality divests control over the Internet from the private sector to the government. And in typical Marxist fashion, innocuous words — the language of neutralism and liberty — cloak an agenda that would crush freedom.”

    Thoughtful people might wonder what it means when various right-wing windbags begin to accuse each other of being socialists.

    Less thoughtful people, on the other hand, might not even notice that the factions have degenerated into the realm of self-parody.

  56. #56 Paul UK
    December 1, 2009

    Geez, this is crazy la la land.
    It is pointless analysing code when you haven’t a clue when it was used, if it was used or what version it is etc.

    I have loads of bits of code that are on my computer that trials and tests etc.
    If someone came along and collected it all, then published it, some probably wouldn’t work, others would but were never used in a project etc.

    Fuck this is ludicrous.
    It’s not even worth engaging the denial nut jobs over this.

  57. #57 Jon H
    December 1, 2009

    “Have these clowns ever heard of CVS, VSS, or sundry other tools to track code changes?”

    Speaking as a tech in a 2009 neuroscience lab: you can lead the postdocs to Subversion, but you often can’t make them use it.

  58. #58 Julius
    December 1, 2009

    Well, that’s strangely comforting – with most climate change denialists I know of, I consider them to be clueless twats, paranoid nutters or worse *because* of their views on climate change. However, I’ve considered Eric Raymond a paranoid nutter since well before I ever knew about climate change. I feel vaguely vindicated.
    I’ve certainly noticed his tendency to see communist plots behind everything previously, plus his classic off-the-rails libertarian view (ever so realistic) that it’s every (American) citizen’s duty to own a gun to overthrow the government if that becomes necessary. I’d like to see you try, Eric…

  59. #59 Ezzthetic
    December 1, 2009

    The AGW meme awakens chiliastic emotional responses in a lot of Gaianists.

    This clearly is nonsense, and you have to ask how much Raymond knows about the subject.

    It’s warmiastic responses we’re gripped by, fool.

  60. #60 Bruce Sharp
    December 1, 2009

    Paul, let me just state for the record that I’m with you. (William Wallace didn’t seem to realize that I was making fun of Raymond’s quote mining, and not Raymond’s code.)

    Stupid arguments are bad enough when the people who make them don’t understand that they’re stupid. But Raymond… ffs, he’s a programmer. How can he not know that a single snippet of code is meaningless if you don’t know what the rest of the program does with it?

  61. #61 David Marjanović
    December 1, 2009

    Behold! I shall feed the troll!

    Your analogy might be stronger if ssh were an organization that all but lobbies for command and control economies.

    How can you possibly accuse anyone of desiring anything as stupid as a command economy!?! We’ve all seen how that sort of thing fails, and it fails hard.

    Why don’t you invest in a company that produces photovoltaic cells or something? It’s perhaps sad, but true, that only capitalism can save us from having to evacuate Bangladesh. Governments can (and should!) provide incentives, but command-and-control measures wouldn’t work, which is why nobody I know of is proposing any, moron.

  62. #62 Julius
    December 1, 2009

    Well, that’s strangely comforting – with most climate change denialists I know of, I consider them to be clueless twats, paranoid nutters or worse *because* of their views on climate change. However, I’ve considered Eric Raymond a paranoid nutter since well before I ever knew about climate change. I feel vaguely vindicated.
    I’ve certainly noticed his tendency to see communist plots behind everything previously, plus his classic off-the-rails libertarian view (ever so realistic) that it’s every (American) citizen’s duty to own a gun to overthrow the government if that becomes necessary. I’d like to see you try, Eric…

  63. #63 dhogaza
    December 1, 2009

    But Raymond… ffs, he’s a programmer. How can he not know that a single snippet of code is meaningless if you don’t know what the rest of the program does with it?

    It’s the jackboots, guns and libertarian wingnuttery effect.

    However, I’ve considered Eric Raymond a paranoid nutter since well before I ever knew about climate change. I feel vaguely vindicated.

    No need to feel vague about it :)

  64. #64 Statistics sans Frontières
    December 1, 2009

    What about `FOIA\documents\harris-tree\briffa_sep98_e.pro` ?

    The code doesn’t appearr to be commented out in that one and it has the same fudge factor.

    ;
    ; APPLY ARTIFICIAL CORRECTION
    ;
    yearlyadj=interpol(valadj,yrloc,x)
    densall=densall+yearlyadj

  65. #65 ali baba
    December 1, 2009

    How can you possibly accuse anyone of desiring anything as stupid as a command economy!?!

    I accuse denialists of wanting worse: no economy.

  66. #66 MartinM
    December 1, 2009

    The code doesn’t appearr to be commented out in that one and it has the same fudge factor.

    Who cares? We’re talking about a decline which has been described in detail, in the peer-reviewed literature, by the very scientists accused of hiding it.

  67. #67 Ezzthetic
    December 1, 2009

    It’s hard to fathom why the ABC and The Age don’t run a story on ClimateGate if it’s so innocuous.

    In other words:

    It’s hard to fathom why the ABC and The Age don’t focus on a non-issue if they think it’s such a non-issue.

  68. #68 dhogaza
    December 1, 2009

    The code doesn’t appearr to be commented out in that one and it has the same fudge factor.

    Gosh, scientists playing with data when trying to understand certain aspects of the dataset.

    I’m horrified, I say! Horrified!

    (read the first response to the post)

  69. #69 Jody
    December 1, 2009

    Eric Lund, thanks for that. I’m still going to maintain that “The Freddy Krueger Effect” is the most appropriate name for the phenomenon, given the scarring, pain, and fear that results.

  70. #70 jody
    December 1, 2009

    D’Oh! I forgot the smiley face! :-D

  71. #71 Jody
    December 1, 2009

    Produce the data and the algorithms or shut-up.
    The dog ate my data doesn’t cut it. This train is wrecking, I suggest you get off at the next stop or you’ll go tumbling down the gorge with it. There will be a new blacklist, and those who defend too much and too often will be remembered.
    Trust is gone forever, at least during your lifetime.

    Alex, I’m not a scientist and not all that familiar with the details and minutia of the matter, but there is simply no way that a you can honestly read the emails, the context, the history and what’s easily verifiable by google and conclude that this is some mass conspiracy predicated on faulty data.

    As numerous people have shown, the data has been available for years, the code explicitly reveals what is the original temps and what is adjusted temps and the, excuse my French, every fucking paper any of these guys have written makes note of the fucking adjustments made to and the fucking limitations of these data.

    Would you leave the science fiction conspiracy writing to the experts?

  72. #72 TrueSceptic
    December 1, 2009

    15 mb,

    You have taken time off from Denial Depot, I see. :)

  73. #73 Nick Barnes
    December 1, 2009

    Emacs colour codes the comments

    Old Emacs hands tend to turn off such namby-pamby modern nonsense. Certainly my Emacs never colour-codes anything, thank you very much.

  74. #74 el gordo
    December 1, 2009

    Ezzthetic

    If the Wall Street Journal can talk about ClimateGate I don’t see how this is a non-story.

  75. #75 TrueSceptic
    December 1, 2009

    30 caerbannog,

    I could’ve saved you a lot of time. Those of us who are disciples of [STR](http://ShelleyTheRepublican.com/category/education/technical/linux) have known this for years.

  76. #76 Joseph
    December 1, 2009

    If the Wall Street Journal can talk about ClimateGate I don’t see how this is a non-story.

    It’s a story alright, but unless you can show there’s been actual falsification of data, it’s irrelevant and it will blow over.

  77. #77 William Wallace
    December 1, 2009

    William Wallace didn’t seem to realize that I was making fun of Raymond’s quote mining, and not Raymond’s code.

    Good grief. Who let you on the internet.

  78. #78 TrueSceptic
    December 1, 2009

    60 Bruce,

    Because he’s a Denydiot. Is it really necessary to say any more?

  79. #79 Marion Delgado
    December 1, 2009

    Compare ESR and RMS – one gave us a couple of mail tools. The other gave us most of what people mistakenly think of as Linux. You also can’t compare Raymond to, e.g., a Dave Winer. ESR likes to pretend to guruhood. When he went on about how Python was suddenly the true faith, it made me immediately go back to my HOP, OOP and Camel books. Clearly, Perl had hidden virtues if swaggering conceits were abandoning it.

  80. #80 Edward I
    December 1, 2009

    77 WW,

    And you? How much hammering can you take?

  81. #81 Ezzthetic
    December 1, 2009

    If the Wall Street Journal can talk about ClimateGate I don’t see how this is a non-story.

    Well … if the Wall Street Journal takes it seriously, then maybe … maybe I’m wrong about this whole thing …

  82. #82 Bruce Sharp
    December 1, 2009

    Re caerbannog at 30:

    Mother of Zeus, I didn’t see your post before. I’d encourage everyone who is down here at the bottom of the thread to scroll back up for that one.

    I hope no one ever greps through any of my code. If anyone sees how ugly it is, all my public comments over the past fifteen years will be completely discredited. And then Billy Wallace won’t give me permission to use the Intertubes anymore.

  83. #83 P. Lewis
    December 1, 2009

    Jody, I think you underdid the expletives just a tad.

  84. #84 krusty
    December 1, 2009

    Seldom do web-weevils like Betula, Dave Andrews and William Wallace appear in any one place on de intertoobs for very long without a conspiracy of loud stupidity growing among them. Of course pointing this out doesn’t help – nothing would. They’re not even funny like David Kane.

  85. #85 Steve Chamberlain
    December 1, 2009

    el gordo: a little Join-the-Dots game for you…

    1) The WSJ, The Times, The NY Post, The Oz, the (Sydney) Daily Telegraph, Fox – lots and lotsa “ClimateGate” stories.

    2) The Age, The Sydney Morning Herald, the ABC – virtually none.

    Seeing any patterns yet?

  86. #86 MartinM
    December 1, 2009

    See, this is why I never comment my code.

  87. #87 lumpy
    December 1, 2009

    Your analogy might be stronger if ssh were an organization that all but lobbies for command and control economies.

    True, true. When you find that organisation, let me know.

  88. #88 Ezzthetic
    December 1, 2009

    Your analogy might be stronger if ssh were an organization that all but lobbies for command and control economies.

    Personally, I’d be pretty suspicious of anything that called itself “ssh”.

    Sounds like it’s got something to hide.

  89. #89 luminous beauty
    December 1, 2009

    >Well … if the Wall Street Journal takes it seriously, then maybe … maybe I’m wrong about this whole thing …

    I’m waiting to hear from George Will.

    In other news:

    [WINGNUTS FLEE SINKING SHIP](http://littlegreenfootballs.com/article/35233_Did_Climate_Scientists_Destroy_Data_A-_No)

  90. #90 Jody
    December 1, 2009

    P. Lewis, I’m fucking sorry. :-)

    I’m not sure there’s an answer to this other than “wait and see,” but the question that keeps popping up in my head about all of this is are we in any danger of the “anti-AGW” meme becoming dominant? I look at the crappy reporting on the issue, our short sighted nature and the complexity of the whole AGW case and remain concerned.

    Tim’s breakdown of the code is simple and illustrative, but it has one problem: it requires people to think. The “fraud” meme only requires people to feel. For my own site, I’m thinking about writing a post about that dichotomy. There’s a lot of smart people here, so I’ll raise it here, first: Is there a way to make the reply to the hack-crap as much about feeling as it is about thinking?

  91. #91 Janet Akerman
    December 1, 2009

    IF the WSJ reports on it, then it must be credible!

    Like [this](http://www.realclimate.org/index.php/archives/2007/10/global-warming-delusions-at-the-wall-street-journal/), and [this.](http://www.huffingtonpost.com/peter-a-ubel/the-wall-street-journal-a_2_b_45445.html)

    [Who owns](http://www.globalissues.org/article/159/media-conglomerates-mergers-concentration-of-ownership) the media?
    And which self-licking-icecreams do they have an interest in?

    [Speaking from one of Australia's, several one-paper-states.]

  92. #92 David Delony
    December 1, 2009

    Wait, socialism’s a bad thing?

  93. #93 luther blissett
    December 1, 2009

    Clearly, Eric S[extips] Raymond has never kludged around with code for his own personal use. Either that, or he’s a big ol’ hypocrite.

  94. #94 Vince Whirlwind
    December 1, 2009

    Steve Chamberlain @85:
    The Oz has also very recently published a story about how sea levels dropped 20cm in the 1970s. Their “authority” was one Nils-Axel Mörner, dowser and famous kook.
    They’ve also recently trumpeted Plimer’s erroneous “Volcanoes emit more CO2 than humans”.

    So I don’t see why you would use the Oz to judge the merits of a global warming story.

  95. #95 David Kane
    December 1, 2009

    Tim:

    1) Thanks for starting this thread. I think that the best way to make progress on this issue, like most others, is for the smartest proponents of both sides to argue with each other. It is much more productive for you to take on Raymond than it is for you to spend time beating up the stupid skeptics. (For the same reason, I think that your posts on Stephen McIntyre are quite useful.)

    2) I hope that you will also do a new thread on Willis Eschenbach for the same reason.

    3) I think that you make some excellent points in this post. I look forward to reading Raymond’s response and/or further comments. Whatever your commentators think about Raymond, there is little doubt that he had the personality and background (I recommend his book “The Art if UNIX Programming”) for diving into the details. If people like Raymond can’t find significant faults in the code, then there are probably no meaningful faults to find.

  96. #96 Oroboros
    December 1, 2009

    @Jody 90:

    Is there a way to make the reply to the hack-crap as much about feeling as it is about thinking?

    I’ve become convinced that everyone who ever hopes to succeed on Madison Ave. must read the chapter about war propaganda in Mein Kampf.

    I realize some will invoke Godwin here, but I’m dead serious. If you want to counter the feeling of fraud, you probably have to pander to people’s fears of being manipulated with propaganda, which ultimately requires getting them to think a little too.

  97. #97 Steve Chamberlain
    December 1, 2009

    Vince (94): “So I don’t see why you would use the Oz to judge the merits of a global warming story.”

    I wasn’t (well, not in any favourable sense), I was just trying to point el gordo to something, i.e. that the WSJ is part of the same media empire …

  98. #98 ali baba
    December 1, 2009

    If people like Raymond can’t find significant faults in the code, then there are probably no meaningful faults to find.

    Anyone who understood the code would be a better choice. Sheesh, the guy’s a unix programmer, not an effing polymath, the meaning of the code is outside of his domain. If you specified the code formally in detail then he might be able to write a neater implementation. If he resisted his instinct to know better, it might even be a correct implementation.

  99. #99 Chris O'Neill
    December 1, 2009

    David Kane:

    If people like Raymond can’t find significant faults in the code, then there are probably no meaningful faults to find.

    Who needs Raymond, any old compile and run will tell you if there are “faults in the code”, if that’s all that matters.

  100. #100 David Kane
    December 1, 2009

    Chris: That is not what I mean by “faults.”

    Tim: Derek Lowe is another smart scientist whose concerns merit a thread of their own.

    I’m not actually going to comment on the climate-change aspect of all this, though. I have my own opinions, and God knows everyone else has one, too, but what I feel needs to be looked at is the scientific conduct. I’m no climatologist, but I am an experienced working scientist – so, is there a problem here?

    I’ll give you the short answer: yes. I have to say that there appears to be several, as shown by many troubling features in the documents that have come out.

    Read the whole thing.

    Also, climate scientist Eduardo Zorita:

    Why I think that Michael Mann, Phil Jones and Stefan Rahmstorf should be barred from the IPCC process

    Short answer: because the scientific assessments in which they may take part are not credible anymore.

    Also climate scientist Judy Curry:

    Transparency. Climate data needs to be publicly available and well documented. This includes metadata that explains how the data were treated and manipulated, what assumptions were made in assembling the data sets, and what data was omitted and why. This would seem to be an obvious and simple requirement, but the need for such transparency has only been voiced recently as the policy relevance of climate data has increased. The HADCRU surface climate dataset and the paleoclimate dataset that has gone into the various “hockeystick” analyses stand out as lacking such transparency.

    If Judy Curry thinks that there are problems with the various “hockeystick” analyses, don’t you think we should listen to her?

    Again: I think that science works best when the smartest proponents of alternative views in engage in open debate. A thread at Deltoid on each of these scientists would be a great way to move forward.