I wrote earlier today about mercury and autism, and how I thought a criticism of an earlier paper on statistical grounds was fair. Some of the commentors including Orac took me to task saying that the original analysis was indeed better. After thinking about it for most of the day, I changed my mind. The more I think about the original study and the re-analysis, the more I think that the original study got a fair conclusion and performed fair statistical analysis. On a second look, I think the re-analysis may have been nitpicking unfairly.

Marginal results or not, it is barely ever OK to say that, "If we did a one-tailed t-test it would be significant, so this is a result worth following up." I almost feel like that would result in two many false positives, hence the two-tail standard in the first place.

This is one of the problem with blogging. Sometimes you change your mind. It is sort of like science that if you are honest you can come clean, but in hindsight I am sorry I posted that.

In any case, this brings me a to another question: when is it fair to say, "OK, the facts are in and we are done with it." I have been rereading SciBling Chris Mooney's The Republican War on Science, so I am sensitive to the ways in which the Right has tried to limit regulations by subsidizing endless debate to find the "facts" when the facts are basically in already. I have many conservative friends who question global warming or that the debate can ever be over for that topic.

Of course a debate on a scientific subject can eventually be over. We all agree on gravity. Certainly there is a point where no more debate is necessary, and I would hope that it is generally before hundreds of years have elapsed.

My question is: what is a reasonable standard for ending scientific debate? I think that it has been satisfied in the case of mercury and autism, but we need to be more explicit than "I know it when I see it." With respect to mercury and autism, it fulfills two criterion that I think are necessary for ending debate:

• 1) There is a large amount of evidence.
• 2) At least a majority (if not all) points in a consistent direction.

This amounts to almost a weight of the evidence standard for ending debate -- as well as a weight of the evidence standard for what is correct. It also discourages scientist's using their professional judgment in the place of evidence. (This is point on which I disagree with Chris with respect to his book. Rereading it, I agree entirely on matters of fact and disagree only slightly on matters of emphasis, but he cites the example of the Klamath fish kill about when scientists in government should use their discretion. I remain unconvinced that professional judgment is something we should give scientists free rein with, but I have always been a contrarian that way.)

The point is that if we want to say to the Right or to people who advocate the autism/mercury link the debate is over, we should at least be able to formulate a clear standard for when debates should be over.

Maybe some of the commentors can help me with their opinions on what that standard should be...

UPDATE: Autism Street has a must read critique of the DeSoto and Hitlan re-analysis. They are very skeptical of this re-analysis particularly because they cite evidence from non-peer reviewed journals.