Data Dredging at VaxGen

By David Michaels

It came as no surprise to some observers that VaxGen (a biotech company in Brisbane, California) failed to meet the specifications of its contract to provide the US government with 75 million doses of a new anthrax vaccine. VaxGen has been playing fast and loose for quite some time â most notably with a famous instance of data dredging in the analysis of the clinical trials for AIDSVAX, its failed AIDS vaccine. Iâll come back to that below.

On Tuesday, the Department of Health and Human Services announced was ending its sole-supplier contract with VaxGen, which would have been worth up to $877.5 had the company been able to produce vaccines that worked. HHS officials would not discuss the reasons of the cancellation, but, according to Renae Merleâs piece in todayâs Washington Post, âHHS evidently canceled the contract after VaxGen missed its deadline Monday to begin human testing because of concerns at the Food and Drug Administration about the product's reliability.â

Ken Silverstein at Harper.org has more of the details:

VaxGen received the anthrax-vaccine contract two years ago, in spite of the fact that the decade-old firm had produced no marketable products, had demonstrated disturbing ethical failures in the area of human-subject drug testing, and had been delisted by Nasdaq because of accounting irregularities. An earlier, similarly hyped-up effort to develop an AIDS vaccineâfor which the company received millions from the National Institutes for Healthâended in total failure and led to a number of lawsuits by investors who charged that VaxGen had fraudulently hyped the vaccine's prospects for success.

VaxGen's competitors suspected that the firm, in winning the anthrax contract, may have received a helping hand from Dr. Phil Russell, formerly a top contracting official at HHS who was involved in the BioShield project. Forbes describes Russell as a âlong time acquaintanceâ of VaxGen CEO Lance Gordon; Russell, for his part, has denied that he helped out VaxGen.

Whatever the reasons for picking VaxGen, the choice has proved a disaster. VaxGen pledged to produce 75 million doses of the vaccine, with deliveries slated to begin early in 2006. But late last year, the company announced that unforseen âproduct refinementsâ would force it to postpone delivery of its vaccine until late 2006. Further problems ensued, and now the earliest delivery date would be late 2008 or 2009.

What about the data dredging? Epidemiologists know that with a sufficiently large data base, a good researcher can often come up with something deemed statistically significant. Dredge deep enough, wide enough, long enough, and then shock the world with your new finding. VaxGen did some data dredging in its clinical trial for AIDSVAX, which resulted in lots of false hopes, and longer trials of a vaccine that simply didnât work.

As reported in JAMA, when the results of the first important trial were analysed, the vaccine was found not to reduce infection rates in the entire group studied, but there was a statistically significant protective effect among the African-Americans in the study. The vaccine did not work for Asians, but there were so few Asians in the trial that they could be added to blacks and yield a âBlacks and Asiansâ category that was still technically significant. In fact, âothersâ could be added without ruining the count, so âblacks and Asians and othersâ was still statistically significant. Ideally, Hispanics could have been added to the group and the vaccine might then have been touted for ânon-whites,â but the study showed no benefit for Hispanics, of whom there were many in the trial, so the ânon-whiteâ category was no good. Hence the post hoc category "Blacks and Asians"

Of course, itâs possible that the test had turned up something of interest, but there was only one way to know for sure: conduct a new clinical trial to test if this group really did benefit from the vaccine. But since âBlacks, Asians, and othersâ was the best VaxGen company could do, thatâs what it went with. When the vaccine received its next real test, a large clinical trial in Thailand, it failed miserably.

David Michaels heads the Project on Scientific Knowledge and Public Policy (SKAPP) and is Professor and Associate Chairman in the Department of Environmental and Occupational Health, the George Washington University School of Public Health and Health Services.