I asked Mary Losch (chair of AAPOR’s Standards Committee) to comment on my previous post
I have read your entry and would note that the links you provided did
not supply the questionnaire items but rather a simple template (as
noted in the heading). The Johns Hopkins report provides only
superficial information about methods and significantly more detail
would be needed to determine the scientific integrity of those methods
– hence our formal request to Dr. Burnham. The Hopkins website
refers to data release but, in fact, no data were provided in response
to our formal requests. Included in our request were full sampling
information, full protocols regarding household selection, and full
case dispositions — Dr. Burnham explicitly refused to provide that
information for review.
We do not provide public reports of the investigations but if there
are other specific questions that I could answer, I would be happy to
try to do so.
It is more than a little ironic that AAPOR has censured Burnham for not fully disclosing the data behind his study, while themselves failing to fully disclose the basis for their serious charge.
In response to a question from Mark Blumenthal she added:
we requested the survey instrument (including consent information) and it
was not provided. The template did not appear to be much beyond an outline and
certainly was not the instrument in its entirety.
I don’t think that Losch or the AAPOR have disclosed, specifically, what is missing from the information that Burnham has provided. I stand by my earlier statement that their press release was misleading. I checked their website and they’ve now put their press release up, but provided no supporting information. They’ve also linked to a paper by David Marker published in their journal which implicitly contradicts their press release. Marker clearly felt that Burnham had published enough information for him to evaluate their mthodology, writing:
Burnham et al. attempt to estimate the number of excess Iraqi war dead throughout the country using fairly standard survey methodology, for which they are to be commended. We have examined four specific methodological factors: coverage errors, correct probabilities of selection, migration, and training and control of interviewers. Coverage provided by the first stage of sample selection (the sample of administrative units) appears to have been complete (other than the exclusion of area considered as too violent to allow household interviews as well as the exclusion of the three sampled clusters for the reasons the authors indicated). The coverage at the second stage appears less than complete. In addition to the exclusion of some deaths mentioned earlier (short term household members), there may have been systematic exclusion of some types of households or housing units. The extraordinarily high response rates reported suggest this as a possibility. It should be noted that for the two governorates where data were not collected, they used an underestimate of no excess deaths, choosing to err on the side of understating mortality in these unknown cases. This level of coverage represents a major improvement over other reported estimates.
Tim Parsons responded to my query on behalf on the Johns Hopkins Bloomberg School of Public Health:
The level of civilian mortality in Iraq is a controversial subject. Questions have been raised regarding the findings and methodology of the 2006 Iraq mortality study conducted by Dr. Gilbert Burnham and published in The Lancet. The Johns Hopkins Bloomberg School of Public Health takes any allegation of scientific or professional misconduct very seriously. It believes that the correct forum for discussing the reported findings of the Lancet study and the general methodology that led to those findings is in the regular exchange of views in the scientific literature. The Bloomberg School of Public Health has undertaken a review of the study to determine if any violation of the School’s rules or guidelines for the conduct of research occurred in the conduct of the study. That review is nearing completion and the School is unable to discuss the results at this time.
The American Association for Public Opinion Researchers (AAPOR) chose to criticize Dr. Burnham for failure to fully cooperate with the organization’s review of his 2006 study. The Johns Hopkins Bloomberg School of Public Health is not a member of the organization and does not know what procedures or standards were followed in reaching the decision regarding this study and therefore is not in a position to comment on the decision.
I guess we’ll have to wait for the Johns Hopkins report.
I happened to be writing my story just as the World Health Organization study that was highly critical of Burnham, et. al. was released. Les Roberts, who had become the public face of the team, was making frankly lunatic claims on the radio that the two studies basically agreed, even though the introduction to the WHO study specifically said that their results made it very unlikely that Burnham et. al. had been correct. This claim was so unusual that when I asked neutral conflict epidemiologists, they patiently explained that I couldn’t possibly have heard Roberts correctly, because no one with half a brain would ever have said that.
McArdle does not seem to have understood what Roberts was saying was in agreement: the excess deaths in the Lancet study (about 650,000) and in the IFHS study (about 400,000).