When cluelessness about clinical research attacks

If you leave aside the problem with the Autism Omnibus trial, which has just entered its second week, that annoys me the most, namely a hypothesis so poorly supported by science and so badly argued by a panoply of nonexperts could make it so far in our legal system and possibly even endanger the Vaccine Injury Compensation System with 4,800 almost certainly frivolous claims that vaccines or the mercury in the thimerosal preservative in vaccines, you're left with the more minor annoyances that this whole trial brings. Foremost among these lesser annoyances, which, let's be frank, do not present big questions about big medical questions or address flaws in our legal system when it comes to adjudicating scientific claims, is the resurgence of David Kirby, the opportunistic journalist who, through his book Evidence of Harm: Mercury in Vaccines and the Autism Epidemic: A Medical Mystery and his media appearances, bears a large degree of responsibility, along with Robert F. Kennedy, Jr. and his incredibly dishonest article in Salon.com, for stoking the fears of parents about vaccines with pseudoscience and logical fallacies blaming them for autism, all while piously assuring those who criticize his blather that--really, really--he is not "antivaccine."

Let's face it; before the Omnibus, David Kirby had had his moment in the sun. It had been nearly two years since his book had stoked the fires of antivaccination hysteria, and he was mostly reduced to making the circuit of mercury militia events and spewing the periodic bit of conspiracy-mongering to that repository of antivaccination misinformation, The Huffington Post. Meanwhile, earlier this year, his stock was reduced even further as the end of 2006 passed, and autism diagnoses in California had not decreased, despite being four years out from the expiration date of the last lots of thimerosal-containing vaccines and five years out from a CDC study showing only 2% of vaccines still containing thimerosal. Moreover, after foolishly making a testable claim (namely that the number of cases in the California Department of Developmental Services should decline by the end of 2005) he had been shifting the goalposts to 2007, apparently to shift them further as the 3-5 year old cohort in the CDDS database shows no sign of decreasing in number. Most recently, he was reduced to making such ludicrous claim that mercury from Chinese industry and from the mercury amalgams in the teeth of cadavers being burned in crematoria in California were contributing to this continued increase. I had hoped that that would have put the final nail in the coffin of his misinformation and ushered him into the obscurity that he so richly deserves.

Then came the Autism Omnibus.

Last week, David Kirby sank the lowest that I've ever seen him sink when he basically wrapped the case in the specter of Osama bin Laden, in essence saying that, if the parents win we had better do something or Osama would be pissed off at us for sending vaccines with mercury and "products derived from pigs" to Muslim nations. Yes, indeed, Kirby is becoming like Michael Egnor in a way, the Energizer Bunny of the mercury militia. No matter how many times his pseudoscience and logical fallacies are slapped down, he just keeps going and going and going and going.

Now he's revealing such a profoundly risible misunderstanding of science and how clinical research is conducted that, try as I might, I just couldn't leave his post alone. I wish I could, and I understand if you don't want to read any further, but I couldn't. Kirby starts out with a seemingly reasonable question and then rapidly devolves into rank crankery. Here's the seemingly reasonable request:

Most people (save for a handful of fringe parents who believe that autism is some altered state of being, worthy of celebration) are probably just plain tired of autism and the fight over its cause. They really want to settle this debate and move on.

I know I do.

The irony is that the multi-million-dollar court battles, the melodramatic headlines and the alarm over parents retreating from vaccinations are all so terribly unnecessary.

All we need do is conduct a thorough study of vaccinated and unvaccinated children, and see if there is any difference in their rates of autism spectrum disorders.

Why didn't I think of that? Why didn't scientists think of that? Why didn't the CDC think of that? Why didn't vaccine manufacturers think of that? After all, it sounds so reasonable, doesn't it? Why don't we just do a study?

Would that it were that easy! This is where David Kirby reveals his utter cluelessness about clinical research:

It could be a year or more before we get a decision on the first autism "test case" being heard in federal vaccine court right now, and years more before all 4,800 pending cases are settled. Meanwhile, the Gonzales Justice Department has earmarked millions in taxpayer dollars to fight the autism parents and their attorneys tooth-and-nail in these supposedly "non-adversarial" administrative proceedings. And regardless of the court rulings, neither side is going to back down, period.

But one good study by a respected team of investigators could probably settle this mess by Christmas.

There's only one thing that Kirby is correct about here, and that is his observation that neither side is likely to back down. This of course begs the question of why on earth Kirby thinks that yet another study would convince the antivaxers that vaccines don't cause autism, nor does the mercury in the thimerosal preservative in vaccines. This calling for "just one more study" ploy is classic crankery, and, even if the study were done and produced results consistent with all the other epidemiological studies that have failed to find a link between mercury in vaccines and autism (or between vaccines, thimerosal or no, and autism, for that matter), it'd be a fair bet that it wouldn't convince Kirby or anyone in the mercury militia that vaccines are safe. (Mark points this out quite well.) But let's leave that obvious problem aside for the moment. Why shouldn't we do such a study?

Actually, I'm not saying that we shouldn't. However, it wouldn't be the panacea that Kirby--or Rep. Carolyn Maloney (D-NY)--seems to think that it would be. Particularly ignorant is Kirby's claim that a "respected team of investigators could probably settle this mess by Christmas."

Wrong, wrong, wrong.

Kirby clearly has zero idea of what such a study would take. Even if the finances were made available tomorrow, it would probably take until Christmas just to get the study up and running. First of all, a protocol would have to be written. If it's a retrospective study (which, presumably it would have to be), then to come up with a protocol and statistical analysis that controls for as many confounding factors that can confuse correlation with causation is by no means a trivial task. It's very difficult and would need the input of many clinical investigators with expertise in autism and vaccines. That would only be the first problem faced by investigators, and to do it right and produce as well-designed a study as possible would take at least a few months.

After that, the study would have to be submitted for approval by an Institutional Review Board (IRB) in order to be sure that human subjects are protected. That could easily take another couple of months, possibly more. If researchers are using a legitimate IRB, unlike the bogus IRB that mercury militia "scientists" Mark and David Geier used to rubberstamp their pseudoscientific Lupron trail, a protocol will almost never make it through without the IRB requiring changes to the protocol and/or the consent form. Investigators could choose to examine only already existing medical records, rather than examining vaccinated and unvaccinated populations and interviewing parents, which might be able to produce a protocol that could merit one of the allowed exemptions from full IRB review. Unfortunately, such a study would have much less power and be far more likely to fall victim of confounding factors. Such is the nature of retrospective chart reviews. Such a study would be much less likely to be the "definitive" study that Kirby seems to envision than a prospective study, in which investigators followed subjects over time, rather than just reviewing their records.

Perhaps the most difficult problem would be subject selection, which is yet another reason why it's ludicrously unrealistic to think that such a study could be completed by the end of the year. The gold standard study would be a double-blinded randomized clinical trial, where one group receives thimerosal-containing vaccines, a second group receives vaccines without thimerosal, and a third group receives no vaccines. Such a truly definitive study is clearly impossible on ethical grounds alone, given that withholding routine medical care (vaccines) from children in the last group is unethical. Even if it were, there's no way that parents would agree to be randomized between such groups; such a trial would never accrue enough patients to come to any conclusion. Even if it could, it would take a minimum of several years to produce any meaningful results, given that most diagnoses of autism or autism spectrum disorders (ASDs) occur between three and six years of age.

Of course, in medicine, when you can't do a definitive randomized, double-blinded clinical trial, you have to consider less rigorous trial designs. The problem with such designs is that they are much more prone to confounding factors. Moreover, large numbers of patients would be needed. Indeed, Kirby seems to recognize this--sort of :

The CDC, which has the conflicted task of boosting vaccination rates while also monitoring vaccine safety, dismissed the Maloney bill, saying it would be impossible to locate large enough numbers of unvaccinated American children needed for an accurate comparison.

But that simply isn't true. Dan Olmsted, author of the "Age of Autism" column at UPI, wrote last year about a large medical group in the Chicago suburbs called Homefirst Health Services, which is largely geared toward parents who home birth their kids, and who tend not to vaccinate.

Dr. Mayer Eisenstein, who founded the practice in 1973, told Olmsted that, of the 35,000 children given care at Homefirst, very few have autism, and those who do were all vaccinated. "I don't think we have a single case of autism in children delivered by us who never received vaccines," he said. (Cases of childhood diabetes and asthma were also reportedly very low.)

Kirby truly reveals his idiocy when it comes to evaluating clinical evidence with this statement. This story is the very same one that I dealt with about a year and a half ago when right wing "Christian Libertarian" wingnut Vox Day swallowed Olmsted's antiscientific speculations whole as "evidence" of a link between vaccines and autism. It's anything but. All it really represents is the speculation and confirmation bias of Dr. Eisenstein and colleagues. Indeed, Dr. Eisenstein freely admitted:

Eisenstein stresses his observations are not scientific. "The trouble is this is just anecdotal in a sense, because what if every autistic child goes somewhere else and (their family) never calls us or they moved out of state?"

What Olmsted, in his bias and anxiousness to "prove" a link between vaccines and autism, neglects to mention is that Dr. Eisenstein has zero evidence other than his memory, which, given the favoring of alternative medicine in his group and what seems to be a general antivaccination bias among both doctors and patients, seems to be nothing more than a big case of selective thinking. Certainly, Olmsted did not report any objective evidence to support his speculation. The history of medicine is littered with beliefs based on no rigorous observation that were later shown not to hold water.Dr. Eisenstein has a bachelor's degree in statistics and a master's degree in public health. He should know that. It's odd, then, that he apparently never saw fit to look at the actual numbers in his practice, rather than simply speculating based on his anecdotal observations, which are prone to many confounding biases. It's possible that his recollections may be correct, but without some sort of clinical evidence to back them up, they remain just that--recollections, prone to all the foibles and biases that human memory, even that of trained physicians, manifest. Kirby's other proposal, to study the Amish or Christian Scientists, would be prone to all sorts of confounding factors based on the differences in genetics and lifestyle differences based on their religion, so much so as to make it very tricky indeed to separate them out.

But if you really want to see how simplistic Kirby's thinking is, get a load of his proposed trial design:

But a study of vaccinated-vs.-unvaccinated kids would, in its powerful, elegant simplicity, compel at least one side to finally fold up shop and go away.

When doing autism surveillance studies, The CDC usually looks at eight-year-olds, to ensure that all late-diagnosed kids are counted. It seems reasonable, then, to randomly select 1,000 (or 5,000, or whatever number is needed for statistical significance) unvaccinated eight-year-olds, and compare them to vaccinated children of the same age (born in 1999, by the way, at the height of mercury exposures from vaccines).

First off, even if this study could be designed to account adequately for confounding factors, it would not be looking at whether thimerosal is correlated with autism. It would look at whether vaccines in general are correlated. (Remember, the MMR, for example, never contained thimerosal.) Even if an adequate number of completely unvaccinated children could be found (very difficult because there are few groups that refuse all vaccines), they would consist primarily of two groups: Children whose parents chose not to vaccinate them and children whose parents were advised not to vaccinate for medical reasons. The latter group of children would introduce a confounding factor of medical conditions, the most prominent of which would be some kind of immune suppression. The former group would almost certainly have a number of confounding factors that would be very difficult to disentangle from the lack of vaccination. After all, parents usually refuse to vaccinate their children for very few reasons indeed. Most prominent of these is either religion (the aforementioned Christian Scientists, for example, whose refusal to use standard medical care would introduce a very pernicious confounding factor, or the Amish, who are highly genetically inbred--an even more difficult to separate confounding factor--and who also do get at least some vaccines) or an ideology tending towards alternative medicine, which is frequently associated with other confounding factors of not using other forms of conventional medicine in the same way as populations that are vaccinated, consuming different diets (organic, for example), using supplements, and a number of other associated beliefs and behaviors, including confounding factors that might not even be identifiable before the study commenced.

I'm not opposed on principle to a study of the sort that David Kirby proposes; it's the rationale and reasons behind such a study that give me pause. To justify such a study, there must be a scientific question that hasn't already been adequately addressed by existing studies that is worth the expense of a new study. There is a strong argument to be made that the existing epidemiological evidence is sufficient to conclude that there is almost certainly no link between thimerosal and autism or vaccines in general and autism. Another epidemiological study of the sort that Kirby proposes is unlikely to add much to the existing data and would be expensive, so much so that it would risk sucking up resources that might be better spent on studies into more promising avenues of research. However, the reasons for doing scientific studies are not always strictly scientific; sometimes there's a great deal of politics involved, and this is a perfect example. Consequently, although scientifically of marginal value, such a study might be worth doing if we had some reasonable assurance that a negative result would reassure the public and silence conspiracy-mongers like David Kirby.

Unfortunately, even if the result of the proposed study were to be a stronger than ever resounding reaffirmation of data that we already have that has failed to find a link between vaccines and autism, it would be unlikely to serve such a public policy purpose. Indeed, Kirby's utter ignorance over the logistics of doing such a study, in which he envisions it being completed by Christmas, leads me to believe that his call for "just another study" is nothing more than yet another delaying action, a.k.a. shifting the goalposts. Remember, this is the same man who said that, if the number of autism diagnoses in the California CDDS database didn't start to fall by 2007, that would "deal a severe blow to the autism-thimerosal hypothesis"; yet, now that autism diagnoses haven't declined is now handwaving about mercury from cadavers with amalgam fillings being burned in crematoria and Chinese industry. Actually, I suspect that Kirby is very much aware of how long such a study would take and how difficult it would be. The delay would allow him and the mercury militia to point to keep the myth of thimerosal as a cause of autism alive during the time that it takes to do the study. Then, when the study is done, if it too fails to find a link between vaccines and autism, they will pounce on any of its flaws. They will cling to the uncertainty in any study to definitively prove there is no correlation.

And then they will call for another study.