Effect Measure

The Lancet paper on the Iraq War toll

One of the headlines made by Bob Woodward’s new book on the Bush administration, State of Denial, is that the violence in Iraq is much worse than we have been told. Told by the Bush administration, anyway. In fact we have been on notice for two years that the level of violence in Iraq is horrendus. In 2004 researchers at Johns Hopkins School of Hygiene and Public Health, Columbia and in Iraq produced the best evidence up to that time of the death toll from the 2003 US invasion. The study was published in the British medical journal, The Lancet. Its publication was expedited so it could appear before the 2004 November elections and was condemned on that basis as a “political document.” It was also subjected to a great deal of misinformed, if not ignorant technical critique. We blogged it many times over on the old Effect Measure site (e.g., here), but by far the best defence of the technical issues was produced by our (now) Science Blogs sibling, Tim Lambert at Deltoid. Tim didn’t just knock down the false technical criticisms, he flattened them (an example, here).

Now an update of the 2004 study has been published using the same methodology, with some of the same team of researchers from Johns Hopkins.

It underlines the Woodward thesis in triplicate. The current best estimate of excess deaths from the 2003 invasion stands at just over 650,000. NOt opnly far worse than we have been told. Far worse than anyone imagined. The response from the Republicans will be the same as last time: technical smoke screen and the claim that it’s “just politics.” We’re already seeing both responses. Dear Leader Bush has given his scientific assessment today by saying at a news conference he doesn’t consider the work credible (as in, “I choose not to believe it.”) (CNN). CNN talking head (“defense analyst”) Anthony Cordesman, whose track record on punditry in the Iraq war has been dismal, gave his opinion before the paper was even available for review: it’s politics. That was in an early CNN story now replaced by a version without Cordesman’s opinion, but we know it will be a prominent version of the Republican response, so let’s deal with it first.

Yes, the objective is, in part, political. Voters should have the best possible information about candidates and their policies before voting. The Iraq war is a major issue in the upcoming US midterm elections. So it makes sense to provide the best information possible. You can call that politics if you want. It’s not a problem for the science. We’ll come back to this at the end of this post.

What about the science? It doesn’t take much reflection to see that estimating mortality from an invasion with a disastrous and chaotic aftermath is no simple matter. The Iraqi Ministry of Health is entirely unequipped to make such an estimate. It doesn’t have the expertise and cannot operate freely in most of the country. It is also constrained by sectarian politics, as are all Iraqi ministries. Most other estimates are based on hospital and mortuary records or media reports. They are neither systematic nor reliable and are known to underestimate the toll:

The US Department of Defence keeps some records of Iraqi deaths, despite initially denying that they did. Recently, Iraqi casualty data from the Multi-National Corps-Iraq (MNC-I) Significant Activities database were released. These data estimated the civilian casuality rate at 117 deaths per day between May, 2005, and June, 2006, on the basis of deaths that occurred in events to which the coalition responded. There also have been several surveys that assessed the burden of conflict on the population. These surveys have predictably produced substantially higher estimates than the passive surveillance reports. (Burnham et al. The Lancet [.pdf])

Given the security situation, a population based random sample is not feasible, but this still leaves some scientific methods available, one of which, cluster sampling, is a well accepted and conventional technique. In this case it also produced data that is consistent with data from other sources. For example, the paper estimates pre-invasion mortality of 5.5 deaths/1000 population (Crude Mortality Rate), in line with independent estimates from the CIA and the US Census Bureau. I haven’t had a chance to read the paper in detail, yet, but the authors are extremely competent and their statistical consultant (Scott Zeger), chair of the Biostatistics Dept. at Hopkins is someone I have been on committees with. He was just elected to the Institute of Medicine, part of the National Academies of Science. These folks are not scientific lightweights. They know what they are doing and The Lancet is one of the world’s premier medical journals. There are many reasons to have confidence in these results, even without a detailed examination of the paper (I am still reading it). That confidence is shared by others:

Ronald Waldman, an epidemiologist at Columbia University who worked at the Centers for Disease Control and Prevention for many years, called the survey method “tried and true,” and added that “this is the best estimate of mortality we have.”

This viewed was echoed by Sarah Leah Whitson, an official of Human Rights Watch in New York, who said, “We have no reason to question the findings or the accuracy” of the survey.

“I expect that people will be surprised by these figures,” she said. “I think it is very important that, rather than questioning them, people realize there is very, very little reliable data coming out of Iraq.” (WaPo)

And surely, that is the main point. We have little reliable information. This is the best we have, by far. It has a wide confidence interval (400,000 to 800,000), not a surprise. But it would be a bad mistake to consider these estimates flawed. They are telling us something, something terrible. Here are some additional details, from the excellent summary by David Brown in The Washington Post:

The survey was conducted between May 20 and July 10 by eight Iraqi physicians organized through Mustansiriya University in Baghdad. They visited 1,849 randomly selected households that had an average of seven members each. One person in each household was asked about deaths in the 14 months before the invasion and in the period after.

The interviewers asked for death certificates 87 percent of the time; when they did, more than 90 percent of households produced certificates.

According to the survey results, Iraq’s mortality rate in the year before the invasion was 5.5 deaths per 1,000 people; in the post-invasion period it was 13.3 deaths per 1,000 people per year. The difference between these rates was used to calculate “excess deaths.”

Of the 629 deaths reported, 87 percent occurred after the invasion. A little more than 75 percent of the dead were men, with a greater male preponderance after the invasion. For violent post-invasion deaths, the male-to-female ratio was 10-to-1, with most victims between 15 and 44 years old.

Gunshot wounds caused 56 percent of violent deaths, with car bombs and other explosions causing 14 percent, according to the survey results. Of the violent deaths that occurred after the invasion, 31 percent were caused by coalition forces or airstrikes, the respondents said. (WaPo))

Is this paper political? Of course. If that bothers you, we’ll let one of the most famous pathologists of the 19th century explain it:

Medicine is a social science and politics is nothing else but medicine on a large scale. Medicine as a social science, as the science of human beings, has the obligation to point out problems and to attempt their theoretical solution; the politician, the practical anthropologist, must find the means for their actual solution.

Americans will have an opportunity on November 7 to choose politicians they will entrust with cleaning up the lethal mess that is the Iraq war. This paper is information to help them choose. That’s politics, for sure.

May they choose wisely.

Addendum: See also excellent posts on this by Cervantes and Lindsay at Majikthise.

Comments

  1. #1 Marissa
    October 12, 2006

    The paper was reviewed by 4 referees before acceptance and the method is a good one. We can argue about the exact numbers, but the results demonstrate that the number of deaths is vastly greater than we have been told by the Bush War Machine. Yes, the timing is political, but so what?

  2. #2 Darin
    October 12, 2006

    Worstall apparently knows humility. That, I can respect.

    From
    http://timworstall.typepad.com/timworstall/2004/11/eating_crow_and.html

    Worstall writes:
    “Further to my article of Friday on this subject. I’m afraid I mangled the statistical argument. My inadequate knowledge of the subject led me to make an argument that is incorrect. I stand by my contention that there is something fishy about this study (leaving aside the politically motivated timing of its publication, something the author has been clear about himself) yet have to admit that I have not found it, leaving me with nothing but personal prejudice upon which to stand my argument. I would also like to make clear that this subject was not “assigned” to me, the idea, research, argument and errors were all my own, as was my request for this clarification. Just in case you are wondering, being fact checked by the Pajamahaddin and being found in error does hurt and I hope that future writings will be, where necessary, so corrected.”

  3. #3 Steinn Sigurdsson
    October 12, 2006

    There is an interesting consistency check in the entry.
    The MNC-I estimate is about 40,000 per year from coalition response events, which presumably include a large fraction of coalition strikes.
    This extrapolates to little over 100,000 deaths since the invasion.
    Which is consistent with 1/3 of the lower bound of the Lancet paper.
    Round numbers, of course, but clearly in the right ballpark.

    So we conclude that, in round numbers, the coalition has killed directly about 100,000 Iraqis since the invasion, and that another few hundred thousand (say 3-500,000) have died from indirect consequences of the occupation, including intra-Iraqi fighting.

    That is not good.

  4. #4 O'Leary
    October 12, 2006

    They can nitpick as much as they want about these very serioous Lancett studies, there is one fact where the margin of error is extremely small and that is that Bush and Co. are solid hypocrites and liars when they insist on their ridiculously low deathcounts.

    I am also quite wary of the death count of U.S. casualties in the Iraq war. Is there a count of the soldiers who have died from their wounds after being evacuated or is the death count only those who are directly killed while on duty in Iraq?

  5. #5 revere
    October 12, 2006

    O’Leary: My understanding is that those who die directly of wounds on the battlefield are counted in the death count. What has been striking in this war is the relatively low mortality of US soldiers, which most speculate comes from tremendous imporvementrs in battlefield treatment. Those are soldiers who would have died in past wars, including Gulf I. So they live on, with serious wounds: missing limbs, quadriplegics and paraplegics, head injuries (the signal wound of this war), etc. So this is a shell game for death and injury, the dead being moved to the seriousy injured category. Then there’s the toll on the Iraqis, which we now see to be horrendus.

    These are the best data we have. They are not perfect but they are many times better any other figure we have. If Dear Leader Bush thinks they aren’t credible, he can use the full power of the US government to do a better one. Unless he’d prefer not to know.

    We know the truth of this is irrelevant to him, just as the truth of what he says is irrelevant to him. As you say, that is one area where the margin of error is small.

    During Vietnam we used to talk about the LBJ Credibility Test, the signs you could use just by watching him whether he was telling the truth or not. When he smoothed back his hair, he was tellilng the truth. When he scratched his nose, he was telling the truth. But when he moved his lips, he was lying.

  6. #6 Lopakhin
    October 14, 2006

    For example, the paper estimates pre-invasion mortality of 5.5 deaths/1000 population (Crude Mortality Rate), in line with independent estimates from the CIA and the US Census Bureau.

    I hope you know that there are problems with relying on the CIA World Factbook (the CIA source cited by the survey). It did indeed give about 5.5/1000 for 2002. However, if you look at the 2006 figures here, you will see that their estimate is 5.37/1000. i.e. even less; and certainly nothing like the 13/1000 average post-war death rate claimed by the survey. Taken at face value, this would give the strange conclusion that lives were actually saved by the war. Now, reasonable explanations have been put forward for this elsewhere, namely, that the CIA hasn’t been bothering to update its statistics. But that still casts a little doubt, IMHO, on the reliability of the 2002 figures. Also bear in mind that the CIA didn’t exactly have many demographic researchers in pre-war Iraq; nor, to my knowledge, did the US Census Bureau. The World Health Organisation, which did have some people thereabouts due to its involvement with Oil-for-Food, gives higher figures.

  7. #7 Lopakhin
    October 14, 2006

    Stein Sigurdsson: The MNC-I estimate is about 40,000 per year from coalition response events, which presumably include a large fraction of coalition strikes.
    This extrapolates to little over 100,000 deaths since the invasion.

    Hate to nitpick. But could I possibly ask where you get that figure from? I’m assuming it’s from here, on p.1 of the Lancet survey:

    The US Department of Defence keeps some records of Iraqi deaths, despite initially denying that they did.4 Recently, Iraqi casualty data from the Multi-National Corps-Iraq (MNC-I) Significant Activities database were released.5 These data estimated the civilian casuality rate at 117 deaths per day between May, 2005, and June, 2006, on the basis of deaths that occurred in events to which the coalition responded.

    Now, if there were 117 deaths per day, that would work out as 42,700 per year, which looks like the figure you cite. But the Lancet only claims that this was the figure between May 2005 and June 2006.

    (By the way, if you go to the US DoD document which the survey cites as evidence for this, which is here, I think you may find that they’ve misrepresented that. But that’s another story, and I don’t want to get into the details just now.)

    The main reason I’m interested in this is that if the Multi-national Coalition Iraq really was saying that there had been 40,000 deaths each year since the invasion in incidents to which they’d responded, then they would be flatly contradicting President Bush. Wasn’t it late last year-ish when, asked how many people had died in Iraq, he said ‘maybe 30,000′?

  8. #8 tom500k
    October 16, 2006

    1) I am engaged in an email discussion with a BBC journalist re the Lancet study.
    My points are these:
    1)providing balance requires citing people with knowledge of the relevant issues. I checked the nay sayers he cited and all seem to have degrees and experience in fields tangental to the study, e.g, MBAs, Ph.D. in political science, books on the atom bomb, Ph.D. in pschology with specialization in the pscycholog of music, etc.
    I raise this issue only because even the BBC’s updated report seems seems to give equal or greater weight to these folks rather than those qualified to comments by training and experience.
    Further the journalist himself seems to lack the skill to do the analysis and it strikes me as odd that he was not, from the byline, aided by BBC experts in medicine or biostat or epi.

    This technique of dismissing the Lancet study seems even more extreme in the 2 AP articles which were picked up by many of its subscribers.

    2) I found a Radio Free Europe article which seems to contain the (mis)framing of the study that has become the orthodoxy among even the better (more respectable?) media. Obviously this could be a coincidence.

    3) Wonder if anyone on this blog is interested in joining me in trying to keep the study in the public eye. Sadly it seems to have been misrepresented and is quickly disapearing from public discourse.

    Eager to hear from you all,

    Tom, ex-Hopkin School of Public Health postdoc