DeSoto and Hitlan Revisted

About a month ago, I posted about a paper in Child Neurology that was correcting a previous paper that looked at the relationship between mercury and autism. The original paper, Ip et al. 2004, was a case control study that compared the levels of mercury in hair samples from children with autism as opposed to children without. The 2004 paper showed that there was no statistically significant difference in mercury content.

DeSoto and Hitlan showed that the p-value for the 2004 paper was improperly computed. Further, they argue that using new corrected data -- published in an erratum by the original studies authors -- the data also shows a significant difference.

However, the DeSoto and Hitlan paper was sharply criticized by several blogs, notably Autism Street. In an effort to address these criticisms, Dr. DeSoto has published a FAQ about her paper.

(Full disclosure: I have been in email contact with Dr. DeSoto prior to the publication of this FAQ, and I advised her on style and on criticisms she should address.)

Now the criticisms of the DeSoto and Hitlan paper were many and various. Many, but not all, had to do with their selection of a one-tailed t-test in one case. I don't have time to talk about all of those criticism; you should just read the FAQ. I think that the FAQ was helpful primarily because it helps clarify several issues about which I think we were all confused.

Here is a summary of her points:

• The original Ip, 2004 paper -- using the data reported then -- incorrectly computed the p-value. The p-value was in fact significant.
• In addition to incorrectly calculating the p-value, DeSoto and Hitlan's inquiry into the Ip, 2004 paper revealed that it had incorrect data including incorrectly calculated means and standard deviations. This resulted in new data being published in a 2007 Erratum (see bottom) in the same issues as DeSoto and Hitlan's paper.
• If one were to perform a one-tailed t-test on the new data, the p-value for this data is .056. This is not significant as a two-tailed t-test, but it is significant as a one-tailed t-test. Dr. DeSoto argues why this would be appropriate in this case.
• DeSoto and Hitlan argue that the corrected data set includes extreme outliers, and they show that when those outliers are removed the correct data set shows a statistically significant results regardless of whether a one- or two-tailed t-test is used.
• Most importantly, Dr. DeSoto reiterates that the important issue here is that a widely circulated piece of evidence used to debunk the link between mercury and autism misreported their data. This is important regardless of the issue of statistical analysis.

While avoiding taking sides on the scientific issues, I have several comments on this FAQ. I think that in my original two posts on the paper I didn't understand the issue well, and I appreciate it that Dr. DeSoto has clarified her points. I apologize if I contributed confusion over this substance of this paper.

My first comment is to reiterate what I said before: if scientists are to be taken seriously as honest brokers we must always be forthcoming when our data is wrong. Regardless of the issue of data reanalysis -- which I think was the primary objection to the DeSoto and Hitlan paper -- their work is important because it revealed an error in a widely cited paper, and this is the most important take-home.

If someone published a paper tomorrow saying that one of the primary pieces of evidence for global warming was dead wrong, it probably wouldn't change the consensus. But that finding would need to be publicized because any attempt to suppress it or play it down would only add ammunition to those trying to prove that global warming doesn't exist. The same thing is happening with autism and vaccines. A lot of people out there want to prove that there is a link, and if they even smell a cover-up we will only be helping their argument. My point is that if scientists are going to stand up and say mercury does not cause autism, then our data needs to be as good as data can be.

My second comment is that the revelation that the original Ip, 2004 paper misstated its results does not in any way imply that mercury generally or mercury in vaccines specifically cause autism. Ip, 2004 was part of the evidence that mercury does not cause autism, but it is hardly the only piece of evidence. Fundamentally, the issue with DeSoto and Hitlan is for me one of scientific honesty rather than overturning the prevailing view. It is unfortunate that this fact was misrepresented in Congressional testimony that cited the DeSoto and Hitlan paper, but the misrepresentations of Congressmen are hardly the authors' fault. Orac reported that Representative Dan Burton cited DeSoto and Hitlan in a Congressional hearing as evidence for a link between mercury and autism. This is not what their paper says, and Burton should be ashamed of himself. While in an ideal world we police people citing our work for accuracy, in the real world this is nearly impossible to do. People you don't know run with your results in ways you hadn't anticipated, and especially for politically charged issues this can lead to some negative results.

My third comment is that with respect to substantive scientific issues, scientists can -- I hope -- politely disagree. (This would be what distinguishes them from politicians.) For example, take the issue of whether two- or one- tails was appropriate for the t-test. Dr. DeSoto makes a valid argument that no textbook I have read advocates never using a one-tailed t-test. The core issue is one of whether you think directionality is satisfied in this particular case, and -- though I still fall on the side of avoiding one-tailed tests wherever possible and in this case -- I think there are good arguments either way. Likewise, the issue of outliers is one about which you could have a reasonable debate. Throwing outliers out to prove your case is a bad practice, but leaving them in also makes it so that you might miss an otherwise significant effect. I am no statistician, but this sounds to me like it is more appropriate, particularly -- as it was in this case -- when those outliers are more than 3 SDs from the mean.

However, these are all issues that have to be balanced during a well-conducted statistical analysis, and they are issues about which learned people can disagree. When learned people can disagree about a subject, it is best in my experience to try to maintain a civil discussion about it. One of the things that I love about scientific blogging is that the authors of papers can get in on the action. It allows the public access to substantive discussions with the authors that are usually reserved for the corridors of universities. However, blogging -- like everything else on the Internet -- can also cause considerable acrimony. I am not pointing fingers with respect to this particular case, but it saddens me when people impugn the motives of those who differ from them only in their interpretation of the evidence. That is unscientific and profoundly unfair.

While DeSoto and Hitlan seem to have become associated with those who believe mercury causes autism, neither of them has to my knowledge explicitly advocated that position. Unless they do, it is unfair to paint them with that broad brush. Remember that there are careers and reputations at stake here. The Internet may be impersonal, but what we say here does have professional consequences. Please practice courtesy and reticence.

Let me make my position unequivocal, again without pointing fingers. Ad hominem attacks and guilt by association are things that are practiced by junk scientists, Holocaust deniers, televangelists, and other scoundrels -- not practicing scientists. They are a method of last resort employed by the sellers of swamp land and magic beans -- not of those who claim to live by facts. They do not and should not form a legitimate part of a scientific debate.

To the end of having a legitimate debate, I encourage you to read Dr. DeSoto's FAQ as well as the paper so that you can make up your own mind. For my piece, does this change my opinion of whether mercury causes autism? No. That ship has sailed in my opinion. On the other hand, should we publicize contradictory evidence in the name of scientific integrity? Absolutely. It is always our duty to do so.

I encourage you to read and comment below; however, I will caution you that if the debate wonders off topic or degenerates into a shouting match, I will start deleting comments. Play nice, everybody.