Among other Lancet critics: Paul Bolton, a professor of international health at Boston University; Stephen Apfelroth, professor of pathology at the Albert Einstein College of Medicine in New York City; and mortality studies expert Richard Garfield.
Of the three, the only one who is an expert in mortality studies is Richard Garfield. What does he say?
The Lancet study cited two sources for the 5.5 pre-war mortality rate: the 2003 CIA Factbook entry for Iraq and a 2002 profile from the U.S. Agency for International Health. (By comparison, the mortality rate for the United States is, 8.3; the United Kingdom, 10.1; Russia, 14.6; and South Africa, 22.) But those sources, according to Bolton, are not rock solid. In the mid-1990s, for instance, the United Nations Population Division reported an Iraqi mortality rate of 10 per 1,000.
“I have no sense as to why they (the Lancet authors) are so keen to believe the data” he said. “We know people had reduced access to food, medication and health care (during the time of United Nations sanctions from 1991-2002). People tried to make mortality estimates, but access to Iraq was so limited.”
The last reliable census done in Iraq dates to 1987, says Richard Garfield, a professor of nursing at Columbia University and an expert in mortality studies, adding, “All pre-war estimates are guesses — it’s all projections.”
Garfield isn’t criticising the Lancet study — he’s defending it. The CIA, USAIH and UN numbers are all guesses. The Lancet study actually measured the pre-war mortality rate. Measurements should usually be prefered to guesses. And in this case it’s not even possible for the Lancet study to agree with all of the guesses since they contradict each other, so the best it could do is agree with two out of three. Which it does. And the new Lancet study does agree with the only other measurement of pre-war Iraqi mortality since 1987, the 5.0 rate from the first Lancet study. Garfield is probably aware of this because he was one of the authors of the study.
So much for Bolton’s concerns, what about Apelroth?
“Cluster sampling is typically used for something like presidential polls, which involve a list of every household and you can choose your sample truly randomly,” said Stephen Apfelroth, professor of pathology at the Albert Einstein College of Medicine in New York City. “But (Iraq) is entirely atypical. Violence occurs in very restricted areas and can vary block to block. The number of clusters sampled is inadequate given the wide geographic variation to be expected with violent death.”
Apfelroth admits he is not an expert on cluster sampling, but believes the Lancet’s estimate is so extreme it does more to disprove the method than support it. “The statistical community has not come to a consensus on the best method for estimating violent deaths,” he said.
“Apelroth admits he is not an expert on cluster sampling”. Says it all really, doesn’t it? Apelroth is entitled to his opinion, but why is it in this story? Why didn’t the reporter stop him there and go and and find an expert on cluster sampling and get an informed opinion on the matter?
Yes, violence tends to cluster, but then so does voting intentions (that’s why gerrymandering works). This does affect the number of clusters you need for a good estimate. So if you are designing such a study, it’s probably a good idea to consult an expert on cluster sampling, not Apelroth, to see how many you need. Which they did. And if the expert makes a mistake and you don’t have enough, it shows up in your results as a confidence interval that is too wide. Which didn’t happen.