I wrote earlier today about mercury and autism, and how I thought a criticism of an earlier paper on statistical grounds was fair. Some of the commentors including Orac took me to task saying that the original analysis was indeed better. After thinking about it for most of the day, I changed my mind. The more I think about the original study and the re-analysis, the more I think that the original study got a fair conclusion and performed fair statistical analysis. On a second look, I think the re-analysis may have been nitpicking unfairly.
Marginal results or not, it is barely ever OK to say that, "If we did a one-tailed t-test it would be significant, so this is a result worth following up." I almost feel like that would result in two many false positives, hence the two-tail standard in the first place.
This is one of the problem with blogging. Sometimes you change your mind. It is sort of like science that if you are honest you can come clean, but in hindsight I am sorry I posted that.
In any case, this brings me a to another question: when is it fair to say, "OK, the facts are in and we are done with it." I have been rereading SciBling Chris Mooney's The Republican War on Science, so I am sensitive to the ways in which the Right has tried to limit regulations by subsidizing endless debate to find the "facts" when the facts are basically in already. I have many conservative friends who question global warming or that the debate can ever be over for that topic.
Of course a debate on a scientific subject can eventually be over. We all agree on gravity. Certainly there is a point where no more debate is necessary, and I would hope that it is generally before hundreds of years have elapsed.
My question is: what is a reasonable standard for ending scientific debate? I think that it has been satisfied in the case of mercury and autism, but we need to be more explicit than "I know it when I see it." With respect to mercury and autism, it fulfills two criterion that I think are necessary for ending debate:
- 1) There is a large amount of evidence.
- 2) At least a majority (if not all) points in a consistent direction.
This amounts to almost a weight of the evidence standard for ending debate -- as well as a weight of the evidence standard for what is correct. It also discourages scientist's using their professional judgment in the place of evidence. (This is point on which I disagree with Chris with respect to his book. Rereading it, I agree entirely on matters of fact and disagree only slightly on matters of emphasis, but he cites the example of the Klamath fish kill about when scientists in government should use their discretion. I remain unconvinced that professional judgment is something we should give scientists free rein with, but I have always been a contrarian that way.)
The point is that if we want to say to the Right or to people who advocate the autism/mercury link the debate is over, we should at least be able to formulate a clear standard for when debates should be over.
Maybe some of the commentors can help me with their opinions on what that standard should be...
UPDATE: Autism Street has a must read critique of the DeSoto and Hitlan re-analysis. They are very skeptical of this re-analysis particularly because they cite evidence from non-peer reviewed journals.
- Log in to post comments
With correlation analysis, if you have enough evidence, "2) At least a majority ..." (not all) "... points in a consistent direction." would be the most appropriate situation. Given enough studies/evidence that have statistical noise in them, you should have some that point the other way. After all, the .05 t test level means 5 out of a hundred of your tests would indicate statistical significance for the opposite.
I think you need at least, another decimal point on correlation significance, totally independent methods reaching the same conclusion, and a good theoretical framework that has been tested on other predictions, to start putting things to bed.
"The point is that if we want to say to the Right or to people who advocate the autism/mercury link the debate is over, we should at least be able to formulate a clear standard for when debates should be over.
Maybe some of the commentors can help me with their opinions on what that standard should be..."
Here's one idea that builds on yours. You opined that a "weight of the evidence" standard should be used, and I agree. But why? What is the valid reason for ending a debate, aside from academic reasons? I would offer that in a world of limited resources (read: funding for research), there comes a point where decisions must be made regarding future research direction. In the case of Hg/Autism, I would submit that the body of evidence will never be enough to convince the anti-vaxxers and conspiracy theorists who will keep calling for 'more studies', but that the moral high ground exists in a place that says, "No. We must now spend resources in areas that lead to supporting an improved quality of life for those affected by autism."
To encapsulate the idea, I think that once the weight of evidence reaches a certain point, an ethical standard must be applied that makes clear that further research in a failed direction, at least in the absence of entirely new evidence, is a waste of valuable resources.
In the case of autism, it is not just an academic debate. There are some compelling ethics involved as well.