Reed Cartwright just forwarded me (and a few others) an email that was just sent out to an evolutionary biology mailing list. I'm going to quote it in full below. Don't worry if you don't understand the technical terms in there - you don't need to know what Bayesian methods are, or how they're used in phylogenetics, or even what phylogenetics is to understand why this email is important, and why all concerned should be proud of themselves.
Dear Colleagues,This to inform you that we must retract Hall, B.G. and S.
Salipante. 2007. Measures of clade confidence do not correlate with
accuracy of phylogenetic Trees. PLoS Comp. Biol 3: (3) e51 .As a result of a bug in the Perl script used to compare
estimated trees with true trees, the clade confidence measures were
sometimes associated with the incorrect clades. The error was
detected by the sharp eye of Professor Sarah P. Otto of the
University of British Columbia. She noticed a discrepancy between
the example tree in Figure 1B and the results reported for the gene
nuoK in Table 1. At her request I sent her all ten nuoK Bayesian
trees. She painstakingly did a manual comparison of those trees with
the true trees and concluded that for that data set there was a
strong correlation between clade confidence and the probability of a
clade being true. She suggested to me the possibility of a bug in
the Perl script. Dr. Otto put in considerable effort, and I want to
acknowledge the generosity of that effort.I have now corrected the script and re-analyzed the trees
in Tables 1-6. The results show that there are strong correlations
between clade confidence and the probability that a clade is valid
for Bayesian posterior probabilities and for Maximum Likelihood
bootstrap percentages and weaker correlations for Maximumn Likelihood
aLRT values.The major conclusion of our paper, as given in its title,
is therefore invalid and the paper must be retracted. Because of the
importance of measures such as bootstrap percentages and posterior
probabilities to the field of phylogenetics, it is important to
retract that conclusion as quickly and publicly as possible.It is important to stress that the responsibility for the
necessity of retracting our paper is entirely mine, and that my co-
author Stephen J. Salipante bears none of the responsibility. I
wrote the Perl script and failed to check its accuracy sufficiently.
Sincerely,Barry G. Hall
Here's the short version. Dr. Hall made a mistake in the software he wrote to do something. Another scientist, Dr. Otto, saw the mistake, contacted Dr. Hall and told him about it, and the two of them worked together to confirm that it was in fact a mistake. Recognizing the error, Dr. Hall has now retracted the paper and is working to ensure that people quickly learn that the conclusions are in error. Knowing that will keep others from using Dr. Hall's original conclusions in their own work, which means that they won't be starting from a position that's wrong.
And that, ladies and gents, is exactly how this thing called "science" is supposed to work.
- Log in to post comments
And that, boys and girls, is why you don't use Perl!
Sorry. It had to be said. ;-)
Hi Mike,
Just wanted to let you know that your link from Panda's Thumb to this post is not working.
That is one of those things I like about science. It is through exchanges among colleagues that you advance knowledge. BTW, the link in PT was restored and is functional. Thanks to Reed.
Let me also say that Dr. Hall is a fine scientist with a pot load of great publications to his name. It takes a big man and a lot of integrity to stand and admit you were wrong, publicly. Kudos to Barry for doing the right thing, quickly.
Michael Buratovich
And that, boys and girls, is why you don't use Perl
No, that's why all code should be thoroughly tested and proof-read, regardless of the language used.
Kudos to both Dr. Hall and Dr. Otto. Hall for having the integrity to publicly retract his paper the way he did, i.e. very visibly and prompt. Otto for a level of scientific dedication that is way beyond most of us. As much as it sucks finding this type of result altering error in your publication it is of of surprised, however, that she the one who found it. She is by far the smartest person I have ever been fortunate to know.
Oops, found a small but result altering typo in the previous post. Of course what I meant to say was that "...it is of *no* surprise" that Dr. Otto was the one finding the error.
You know, while you hate to see this happen, you love to see this happen.
This is a great example of scientific integrity.
While it is certainly the case that computer programs need to be checked, reviewed and tested, irrespective of the language employed, it is also the case that some programming languages are more likely to lead one astray than others. In general, modern functional programming languages with static typing help avoid common errors which other types of programming languages allow one to commit.
I personally prefer Standard ML in the New Jersey version.
not on topic here, but
When I was at the U. of Connecticut in 1982 or so, Dr. Hall gave a short talk called "Proving Evolution". He said he didn't approve of the title, but it did get an audience. It was organized by some microbiology majors, and attended by some evolution deniers. His lecture was excellent, especially for this biology undergrad.
It was my first experience with people who hated evolution. The evo deniers were all born-again christians, big surprise. Nothing has changed. The religion is the same, and the creationist arguments from 25 years ago are the same.
I remember doing my projects late at night in the Life Sciences building and wondering who that dude was wandering around in the middle of the night. Until I attended the lecture I thought he was a janitor. Yep, it was Dr. Hall.
This history really illuminates another far more serious problem in contemporary academic research practices, namely the lack of transparency in source code developed and used in research. There is a more lengthy discussion of this in the post Show me the code on my blog.
And that, boys and girls, is why you don't use plaintext.
No, that's why all code should be thoroughly tested and proof-read, regardless of the language used.
Too bad you didn't thoroughly test or proof-read your logic ... it's a blatant false dichotomy. Perl shouldn't be used because it has the highest bug to lines of code ratio of any common programming language due to its numerous violations of the principle of least surprise. That's true regardless of the need for testing and proof-reading, which can never be guaranteed to be "thorough".
Hmmm, seeing as this post is about the way science should be, shouldn't you at least provide some collaborating evidence? Further, since perl typically can accomplish much more in a few lines than many other languages, wouldn't that mean it's still possible to write code with fewer bugs overall?
What language is it you use that allows you to write bug free code without the need of testing or proof-reading?
Without knowing the exact nature of the bug, and the expertise of the programmer in that language, it's impossible to state if the language was the cause.
Using this as exemplary of "don't use Perl" or anything of that ilk is absurd. Code contains bugs. The idea that Perl caused this error to happen is ridiculous. Industry suffers from a constant slew of bugs, but they can release patches and no one flies off the handle about how it shows you shouldn't write in C++. What is possible is that coding standards in university labs aren't up to the standards of industry. I am a biologist turned programmer at a university and there is no QA department I can go to. We test the software by using it. And often we are using it to test problems that we can't test in any other way. You can't go out in nature and grab a phylogenetic tree, you have to construct them using some algorithm. So it is very difficult to detect errors since you can only compare it to some other set of results created with some other algorithm. Dr. Hall's students' only failing in this case was coming up with enough positive and negative controls to adequately detect errors. And that has everything to do with science and nothing to do with Perl.
My supervisor had forwarded this e-mail on to several of us after it first came out and I give huge applause to Dr. Hall for his handling of the error which is exemplary.
As for the "don't use Perl" crowd all I can say is that it has been my experience as a pragmatic programmer that virtually all programmers have ingrained likes and dislikes for various languages. Perl has its issues, but I use it rather extensively. Testing and proofing your code is key no matter what language you use. I also use Python and C depending on the task at hand.
All languages have their strengths and weaknesses, much of it just comes down to personal preference. Perl tends to be heavily used by biologists for various reasons, and is heavily used in bioinformatics and computational biology. I don't foresee that changing anytime soon which is fine with me.
"We made a mistake."
"We wanted to let you know we're wrong, and we're sorry."
£µ©λ.
Shouldn't someone tell the President how it's done?
So if somehow a Creationist quotes that paper's conclusions, do we all get to giggle to ourselves? Though I doubt the average Creationist would read a paper and be able to understand it, I just want a good laugh.
The first time I read Karl Popper as a student was a real eye-opener. The idea that we would expect mistakes and that everything is just a theory and we keep error-checking was so foreign to what I'd learned earlier (avoiding mistakes at all costs!).
Notice that, in the two largest religious groups (Christian, Muslim), it has been/is a punishable sin to question anything said by certain individuals. People are still tortured and put to death for disagreeing with church leaders in some areas of the world.
Science is truly a marvelous gift.
Too funny!
Despite not being involved, the simple thought of acting in this manner makes me feel mature and responsible, makes me feel "grown up", strong and proud.
This absolutely is what we need to teach our children as opposed to political wrangling and nonsensical theistic arguments like the intellectually moronic debate what happens to unbaptized children when they die?
Most scientists are honest, I think, and Hall sounds like an all-around good person. But the main lesson here is the importance of checking, not just our own work, but that of colleagues, at Otto did. Peer review of submitted papers has often caught mistakes I had missed, even after I thought I had double-checked everything. Once, a grad student caught an small error in one of my published papers. I had a related paper in revision at the time, so put a note about the problem in that paper. A really important result will be checked by lots of people, including repeating the experiment, so scientific fraud and error rarely cause lasting harm to society, even if individuals are sometimes hurt. Science is a system that keeps improving our understanding of the universe, even if the components (we scientists) are imperfect.
Some years ago I was showing a friend the SECOND EDITION of a book chapter I wrote. For the first edition I had scrutinized it, as had about six other people. About six more people scrutinized the revised version.
Well, within maybe five minutes my friend, who is one of the best debuggers I know, pointed to one of my code examples: "Matt, what's this!?" I said, "Ooops" and promptly emailed the publishers so they could correct later printings...
Fortunately this was just a piece of example code, not a key part of some item of scientific research!
What happens when errors aren't run to ground quickly:
http://www.the-scientist.com/news/display/39805/
Of course similar principles of openness apply more widely than just science.
Some years ago I was pulled on to a flagship programme of a large company as an external project manager. Six departments were involved, and I was handling the work of one of them. In our first programme-wide meeting, the programme manager asked each department to report on progress and problems. Four departments replied deadpan that everything was running to schedule and that they had no problems whatever. I, however, said "well, we haven't quite got everything together yet, and do have a few problems". The "old hand" on my right prodded me and muttered "never admit to problems - that's not the way to do things here". I muttered back "I'll handle it my way, thank you" and proceeded with the issues I wanted to discuss. The sixth department also admitted to something less than perfection in the way of progress.
Six months later, guess which were the only two departments to deliver on time ...
"What happens when errors aren't run to ground quickly:
http://www.the-scientist.com/news/display/39805/"
This is a reference to the Chang fiasco, where a protein crystallographer had been using flawed software for over five years to generate structural models that were completely bogus. This research resulted in papers in Science, PNAS, and other high-profile journals, all of which had to be retracted.
Bizarrely, Chang still has citations to some of these papers on his faculty Web page:
http://www.scripps.edu/research/faculty.php?rec_id=7431
That may be true today - it certainly wasn't true in the 1990s. From about 1990 to 2000 or so, C++ was the FP community's favorite language to hate. All sorts of bugs - even bugs in apps not written in C++ - were blamed on C++ . Why perl has replaced C++ as their favorite language to hate I've no idea, but it has. Before C++ it was C, and before C they hated fortran. It's an old tradition. The FP community may be geniuses at designing languages, but they are far better at sneering at and despising those who use non-FP languages.
The trouble with the way papers and webpages are now captured almost permanently is that even though it was retracted the retraction may not rank above the original. Once something is out there it is hard to take it back. This is why copyright infringement and blogging may make things worse rather than better.
Where is the peer-review? Sure, comments can be posted, but they can also be removed. The confidence level of any written piece is quickly evaporating. Like so much Chinese toothpaste, we cannot tell if what is really inside is good.
I am going to hold my breath until someone from the DI makes a similar declaration about his own, or another "Fellow"'s, errors.
It was nice knowing all of you.
[...] Three important words: 'I was wrong' [...]
http://blog.whybird.net/post/3476899
I have almost the same experience as Dan Gaston w/r/t programs I use when I've helped researchers with programs. For one thing, sometimes there is pre-set stuff that plugs into Perl, so your choice would be to write something yourself in an approved language or plug in to R or whatever, as a molecular biologist friend of mine is doing.
My job now is doing news programming. When I write a program to help an analysis story, I always make sure the reporter keeps a copy of the source code, and I write it with a lot of comments and using good practices. Now that we have a web presence, we can put that source up if needed and point to the data source and people know how we got what we got.
I have never heard the most bugs per line of any language thing about Perl.
Discussions of Verification, Validation, Software Quality Assurance, and related issues, in scientific and engineering software are the subjects of this blog. All corrections and comments are welcomed.
Don't you all know that the amount of mistakes in any progamming language is proportional to the number of programming lines. That is why you have to comment your code.
The final step is to move all mistakes from the executable code into the comments
;-)
There's an important point to recognize about the Hall/Otto exchange. The reason the bug was found so quickly was that Barry Hall made a strong statement arguing against what most people in the field thought was correct. Sally Otto examined Hall's results because they were important to her research (and went against her deep understanding of the system).
There are plenty of papers that do not take such strong positions, and so escape intense scrutiny.
Barry Hall should be commended for taking a strong and unambiguous stance on his original results. That kind of commitment to one's work is risky, but better for science over all.
The combination of fearless commitment to one's results, coupled with a willingness to instantly abandon them when they're shown to be wrong is the ideal behavior of a scientist.