The Iraq Math War

Robin Meija writes about the Lancet studies:

In any case, such problems are common in war zones, according to nearly a dozen leading survey statisticians and epidemiologists I spoke with. “Iraq is not an ideal condition in which to conduct a survey, so to expect them to do the same things that you would do in a survey in the United States is really not reasonable,” says David Marker, a senior statistician with the research corporation Westat. Even if the outdated population data led the researchers to a 20 percent overestimate, Marker explains, the revised death toll would still be at least a couple hundred thousand. “These methodological concerns don’t change the basic message.”

The White House struck back with its own basic message: The study was bunk. Never mind that Roberts and Burnham had used methods similar to those employed for the Kosovo survey and others approvingly cited by the Bush administration. With the notable exception of This American Life producer Alex Blumberg, most reporters dutifully slapped Roberts’ research with the “controversial” label. And when asked about the study directly, President Bush declared that it had been “pretty well discredited.”

“By whom? By him and his political staff?” snaps Bradley Woodruff, who retired last year from his job as a senior cdc epidemiologist. Woodruff has conducted mortality surveys himself, and considers Roberts’ research solid. But when cbs’s 60 Minutes sought to interview Woodruff about the Lancet study in 2007, the cdc wouldn’t allow it. And when Rep. Dennis Kucinich invited Woodruff to Washington to discuss the study, his bosses nixed that, too. “I never had this kind of censorship under previous administrations,” he says.

I guess it shouldn’t have been a surprise that the Bush adminstration’s war on science extend to the Lancet studies.

Meija continues:

More than two years later, the Iraq study remains mired in controversy. But other recent findings suggest that Roberts and Burnham were on the right track. In the summer of 2006, the World Health Organization conducted a large family health survey along with Iraq’s Ministry of Health, interviewing about five times as many people as Roberts and Burnham had, and in a more distributed fashion. In August, Mohamed Ali, a WHO statistician, reported his preliminary results to colleagues at a Denver statistics conference: Nearly 397,000 Iraqis had died because of the war as of July 2006.

That’s the IFHS study which also found 151,000 violent deaths. I used an estimate of 430,000 excess deaths via the IFHS study to make this table.

Meija:

That number falls at the low end of Roberts and Burnham’s confidence interval, which ranges from roughly 393,000 to 943,000. But while epidemiologists and statisticians are still pondering questions raised by differences between the two surveys, there’s no longer much doubt among them that Iraq’s civilian casualties number in the hundreds of thousands.

This grim statistic continues to elude most Americans. According to a February 2007 AP poll, Americans’ median estimate of the number of Iraqis killed since the invasion was just 9,890. And while the Pentagon has presented limited estimates of civilian casualties, it has yet to release any numbers for the total toll since the invasion.

Roberts had set out to provide a legitimate number that might be used to inform public policy. For now, at least, that policy has been to keep the truth buried in academic journals–and beneath the sands of Iraq.

Comments

#1 Crust
November 13, 2008

Americans’ median estimate of the number of Iraqis killed since the invasion was just 9,890

Incredible. What an indictment of our (the US) press corps. That’s reminiscent of the polling data showing majorities of Americans thinking that Saddam Hussein was involved in the 9/11 attacks.
#2 phisrow
November 13, 2008

@Crust: It’s even worse than that. There have been a little over 4,000 American deaths, a number that people ought to be fairly familiar with, and the bulk of those have been from explosions or gunfights in urban areas. The idea that Iraqi civilians are dying at a rate only slightly over twice as high as American soldiers isn’t even remotely plausible. I’m not sure that there is a single war in modern history with a ratio like that, and it certainly doesn’t apply to one being fought largely within urban areas.

The media certainly has done a magnificent job of rolling over and playing dead on this one; but there are basic bounds of plausibility figures that you shouldn’t need to be told to work out.
#3 NoAstronomer
November 13, 2008

@Crust and phisrow: actually it’s *even* worse than that !!!

9,890 is the median estimate. So half the population thinks it’s less than that. Probably a lot less. Like in the hundreds.
#4 Crust
November 13, 2008

@phisrow, good point I agree that ratio of fatalities is wildly implausible given the nature of the conflict.

But I wouldn’t say it’s outside the range of modern history. In WWI, the ratio of civilian to military deaths was around one to one (about 10 million each) and in WWII around two to one (about 47 million vs. 25 million).
#5 Eli Rabett
November 13, 2008

Crust, the number of Iraqi army and police deaths is much higher than 4000. You are comparing sauce to spaghetti
#6 jodyaberdein@doctors.net.uk
November 13, 2008

Funnily enough I was thinking about this issue just today, and lo here it is. My thoughts were along the lines that as Iraq becomes more ‘stable’ presumably it will become easier to estimate the casualties. Does anyone know of groups currently trying to do this or planning the further studies so vehemently demanded previously?
#7 Barry
November 13, 2008

Crust, you might want to use guerrilla wars to come up with plausible ranges of death ratios in a guerrilla war.
#8 Crust
November 14, 2008

Barry, that was kind of my point in the first para of #4.

Eli Rabett, you make a good point in #5. I was just replying to phisrow’s comment in #2 that “I’m not sure that there is a single war in modern history with a ratio [a little over two] like that”. I read (and — as you implicitly point out — quite possibly misread) that as a claim about the ratio of civilian to combatant deaths in modern wars.

Anyways, the main point I wanted to make in this thread was #1, i.e. this is a sorry statement about the (US) media.
#9 David Kane
November 14, 2008

Sad to see that Meija, who was a charming dinner companion in Denver and who (I hope) enjoyed our panel at which Dr. Ali presented his results, would write such a misleading article. Alas, if Mother Jones is paying the bills, you write what the subscribers of Mother Jones want to read.

This is part is just not true.

But while epidemiologists and statisticians are still pondering questions raised by differences between the two surveys, there’s no longer much doubt among them that Iraq’s civilian casualties number in the hundreds of thousands.

And Meija knows it from the panel that she attended. IFHS (like Lancet) provides no estimates of civilian casualties. Conflating civilian and total casualties is one of Les Roberts’ favorite rhetorical tricks.

But thanks, as always, to Tim for bringing these articles to our attention. But why not bring articles that take the opposite side (like this one) to the attention of Deltoid readers? And why not include all (high quality) estimates in your table (what about this one?) rather than cherry-picking?
#10 Crust
November 14, 2008

David, is there really much room for doubt that the Iraqi civilian death toll is over 100,000 at this point? Iraq Body Count is not that far off 100,000 documented civilian deaths (currently 89,000 – 97,000). And of course this is going to be a material underestimate since not all civilian fatalities will be reported in the media (and not all reports will meet IBC’s filter), far from it. Now we can disagree on the magnitude of the effect, but surely even you’d agree it’s got to be significant enough that the real number of non-combatant violent deaths is pretty surely over 100,000 at this point, no?
#11 David Kane
November 14, 2008

I think that the IBC does fine work and it would be perfectly reasonable to round up their estimate to 100,000. But “hundreds of thousands” is the quote and that means 200,000+. Could that many civilians have died? Of course! But the problem with the article is the claim that everyone agrees with that estimate, that 200,000 is some sort of floor. You can’t seriously claim that there is “no longer much doubt” when, at the very panel Meija attended, much doubt was expressed.
#12 sod
November 14, 2008

And Meija knows it from the panel that she attended. IFHS (like Lancet) provides no estimates of civilian casualties. Conflating civilian and total casualties is one of Les Roberts’ favorite rhetorical tricks.

what are you talking about?
a civilian is a person, who is not a member of the armed (or at least para-military) forces.

the difference between total and civilian casualties in Iraq is tiny.

But thanks, as always, to Tim for bringing these articles to our attention. But why not bring articles that take the opposite side (like this one) to the attention of Deltoid readers?

David at his best. why am i not surprised, that the link is to one of his own articles?

I think that the IBC does fine work and it would be perfectly reasonable to round up their estimate to 100,000. But “hundreds of thousands” is the quote and that means 200,000+.

IBC relies on newspaper reports. newspapers NEVER update articles to include follow up casualties after a couple of days.

it is simple not reasonable, to believe in the IBC numbers and at the same time estimate the real number of casualties to be below 200000.
#13 Marion Delgado
November 14, 2008

Kane is innumerate, he’s shifty, he uses sophistry to substitute for explanations, he’s tendentious, he pretends not to understand the difference between shaving off error sources one-sidedly and actually correcting errors* … AND he AUTO-CITES?? And not just autocites, but does so aggressively as if not citing his ridiculous BS was some kind of black mark. Welcome to the status of full-on crank and troll, David Kane.

*IF I spent 20 years “fact-checking,” say, the test achievement records of two schools, and I only “fact-checked” the times when high grades for kids in school 1 had been overstated, and ignored all the other errors, I would not be improving the data for purposes of comparison. As an example. I would likely end up with school 2 doing better even if their actual records, within the margin of error, were identical.

This simple idea is the most blithely ignored one for the trolls. It’s the sole raison d’etre for Anthony Watts and Stephen McIntyre
#14 Tim Lambert
November 14, 2008

David, Obermeyer is not a high quality estimate and does not belong in my table. Daponte’s argument about per-war mortality rates has been already been flogged to death here.
#15 Sortition
November 14, 2008

Kane’s chief disappointment is, very likely, that Meija, despite being “a charming dinner companion”, did not include any reference in her article to Kane’s own valiant efforts in minimizing US atrocities.

A little attention by Deltoid readers is a small consolation prize indeed, but it will have to do.
#16 David Kane
November 16, 2008

1) As Tim is smart enough to figure out (but Delgado and sod are not), I am not auto-citing in the above links. I am referencing the papers by Obermeyer and Daponte (and proving a handy summary).

2) sod claims that:

it is simple not reasonable, to believe in the IBC numbers and at the same time estimate the real number of casualties to be below 200000.

Huh? Have you ever talked to the IBC folks? I have. And, the last time I checked, this is precisely what they believe.

3) Tim: Daponte makes a series of arguments about why we should (mostly) ignore than Lancet papers. (But she’s at Yale, so let’s completely ignore her!) Only one of them has to do with pre-war mortality. First, has this argument really been “flogged to death here?” (I might have missed it, but please provide a link.) Second, Deltoid would be a lot more interesting if you linked to the best criticisms of the Lancet critics rather than solely beating up on the stupid critics. Why don’t you devote a new thread to Daponte? People like sod and Sortition would enjoy it!

4) Tim: You view Obermeyer as “not a high quality estimate.” Can you provide some evidence/discussion for that claim? Also, you realize that it has been published in a high quality, peer reviewed journal? What do you know that the editors of the British Medical Journal don’t?

The point is the truth-in-advertising of your claim that the table lists “surveys of deaths in Iraq,” with the implication that it includes all the highest quality surveys. If you don’t include Obermeyer, then what you really have is a table with “surveys of deaths in Iraq that Tim Lambert likes.” Nothing wrong with such a table, but please label it as such.

You really claim that the not-peer-reviewed ORB survey is “high quality?” On what basis (other than you agree with the results)?
#17 Tim Lambert
November 16, 2008

David, Obermeyer is not a survey of deaths in Iraq at all.

Daponte has been making the same arguments since 2004. See [here](http://crookedtimber.org/2004/11/01/talking-rubbish-about-epidemiology/) and [here](http://scienceblogs.com/deltoid/2006/10/tim_blair_claims_richard_garfi.php).
#18 David Kane
November 16, 2008

Tim,

1) First you claim that “Obermeyer is not a high quality estimate and does not belong in my table.” I then point out that this is, obviously, absurd (unless you want to accuse the British Medical Journal of incompetence). Then, realizing that this a tough claim to defend, you fall back on “Obermeyer is not a survey of deaths in Iraq at all.” Says who? Here is the abstract of the paper:

Analysis of survey data on mortality, adjusted for sampling bias and censoring, from nationally representative surveys designed to measure population health. Estimated deaths compared with estimates in database of passive reports.

Sure sounds like a survey to me! What aspect of the phrase “[a]nalysis of survey data on mortality” do you disagree with?

Now, in your defense, it is true that Obermeyer et do not go to Iraq. They are merely using IBC data (and other analysis) to come up with a better (in their view) estimate. And, following the comments made by joshd on my post, I am not even claiming that theirs is the world’s best estimate.

Again, there is nothing wrong with maintaining a table of your favorite surveys, but to include ORB (not peer reviewed) while excluding Obermeyer (because it is a survey that is not a survey) is the worst sort of cherry picking. When, say, AGW skeptics pull these sorts of tricks, you are (rightly!) all over them.

2) You write: “Daponte has been making the same arguments since 2004. See here and here.”

I think that this is incredibly misleading. Do you not expect us to check the links? Daponte’s name is not mentioned in either of the links you provide. If you have evidence that she was making the same arguments in 2004, then provide links to where she made those arguments.

And the issue is not: When was the first time Daponte made those arguments? The issue is: Why don’t you devote a thread to the most serious arguments made against Lancet by the most serious and credentialed critics? There is nothing wrong with providing links to articles cheering Les Roberts, as above, but is merely being a propogandist for one side the role you seek to embrace?

You act as if Daponte, a credentialed and experienced scholar at Yale, has been dealt with at Deltoid when, in fact, her name has never appeared before this thread. Why don’t you start a new thread with a link to her article? You don’t have to read it yourself or criticize it if you are too busy. But to the extent that you see your role as being an honest broker on the latest information to come out on this important topic, it is wrong for you to act as if Daponte was dealt with 4 years ago.

As always, it is your blog and you can do as you like, but, a year or two ago, I could count on Deltoid for all my Lancet news. Should I not count on you for this going forward?
#19 Tim Lambert
November 16, 2008

David, Obermeyer is not a high quality estimate AND is not a survey of deaths in Iraq. I find it bizarre that you think because I made the second statement, I was somehow withdrawing the first one. I find it even more more bizarre that you think that because Obermeyer was a survey of PLACES OTHER THAN IRAQ, this contradicts my statement that it was not a survey of deaths in Iraq.

As for Daponte, follow the link from Daniel Davies post to Kaplan nad you’ll see she making the same arguments in 2004.
#20 Mark Schaffer
November 16, 2008

Does David Kane add any content or just noise? I vote for the later.
#21 Donald Johnson
November 16, 2008

David, you linked to Daponte in some earlier thread–I remember reading the article and being disappointed with it, because she never acknowledged the fact that Lancet1 and the IFHS survey, gave almost identical midrange estimates for the violent death rate during the first 18 months. If she were interested in being fair she’d have mentioned this, but she didn’t.
#22 Kevin Donoghue
November 16, 2008

David Kane: You act as if Daponte, a credentialed and experienced scholar at Yale, has been dealt with at Deltoid when, in fact, her name has never appeared before this thread.

Mind how you google, David. Tim linked to a Nature article which included her comments in October 2006. Her name also came up in comments in February 2007. I haven’t bothered linking to the thread Donald mentions, because as he says that added nothing.
#23 Donald Johnson
November 16, 2008

IFHS says the violent death rate in Iraq was about 100 per day in the first 18 months–I’d like to know if any Lancet1 critic writing in 2004-2006 ever acknowledged that Lancet1′s midrange estimate of 60,000 violent deaths by Sept 2004 was reasonable. As I recall the whole point of the Lancet1 critics was that the violent death rate couldn’t have been anywhere near that high–Kaplan made the point that IBC’s number at the time was around 15,000, acknowledging that maybe it was really 20, 25, or 30,000, but Lancet1′s excess death toll of 100,000 (civilian and insurgent, violent and nonviolent) was apparently unbelievable to him.
#24 Crust
November 17, 2008

When I do a Google search for Daponte on this site, I get six hits (of which one is from this thread).
#25 Eli Rabett
November 17, 2008

Stick a fork in it. Lancet threads are so yesterday
#26 Robert
November 18, 2008

Sortition wrote:

Kane’s chief disappointment is, very likely, that Meija, despite being “a charming dinner companion”, did not include any reference in her article to [David Kane].

Perhaps she decided it’d be better to quote only people who, um, you know, actually knew how to calculate a crude mortality rate?
#27 Sortition
November 18, 2008

> Perhaps she decided it’d be better to quote only people who, um, you know, actually knew how to calculate a crude mortality rate?

Robert, that can’t be true – Kaplan’s “dartboard” made it into the story. I’d say Meija just figured out that despite trying to push himself into the field, Kane is not a player.
#28 Robert
November 18, 2008

Sortition wrote:

that can’t be true – Kaplan’s “dartboard” made it into the story.

Ah, good point. I’m almost willing to concede but, first, are we absolutely certain that Kaplan doesn’t know how to calculate a crude mortality rate? ‘Cuz, you know, it’s really not such a high bar.
#29 Sortition
November 18, 2008

Heh…
#30 dalazal
November 19, 2008

David Kane @ 18: “_Why don’t you devote a thread to the most serious arguments made against Lancet by the most serious and credentialed critics? (…) Why don’t you start a new thread with a link to her [Daponte's] article?_”

David Kane, what do you think that Daponte’s points would add to what has already been discussed several times? I read her paper and the section she devotes to the Lancet studies criticize 4 things, only 3 of which are relevant for the interpretation of the results (the fourth one is that she does not think L1 and L2 were ethically conducted, since they put the interviewers in harm’s way):

* (1) The interpretation and the width of the 95% CI obtained in L1:

> “_The authors misinterpreted the analysis of the data they had interviewers collect when they reported ”at least 100,000” – the 95 per cent confidence interval reflects that the accurate statement should have been ”we can say with 95 per cent certainty that between 8,000 and 194,000 excess deaths occurred to Iraqis during the period”. Such a wide confidence interval makes one question the usefulness of the information._” (p. 950)

* (2) The use of Crude Death Rates in L2:

> “_Demographers rarely use crude death rates because these rates are affected by the age structure of a population, and thus CDRs do not accurately represent the mortality schedule in a population. Instead, demographers think in terms of age- and sex-specific mortality rates, usually summarized in terms of ”life expectancy”._” (p. 950)

* (3) The pre-war CDRs obtained in L1 and L2 don’t jive with the CDR obtained by UNDP/ILCS:

> “_The pre-war CDR that the two Lancet studies yield seems too low. That is not to say that it is wrong, but the authors should provide a credible explanation as to why their pre-war CDR is nearly half that of what the UN Population Division estimates for pre-war Iraq. Since Burnham et al. arrive at their estimate of Iraqi ”excess deaths” by taking the difference in the pre-war and wartime crude death rates and applying it to a population, if the pre-war mortality rate was too low and/or if the population estimates are too high (e.g., do not take into account the refugee movement out of Iraq), then the resulting number of ”excess deaths” would be too high, yielding inflated estimates. Unfortunately, the authors have not adequately addressed these issues._” (p. 951)

Now, as far as I can tell, all these issues have received considerable attention here at Deltoid. Do you think Daponte’s criticisms bring something new to the discussion? If so, what?

The only thing I can see, so far, is that she is a professional demographer from Yale, instead of Kaplan from Slate.
#31 jb
November 24, 2008

David Kane says:

Why don’t you devote a thread to the most serious arguments made against Lancet by the most serious and credentialed critics?

That leaves David Kane out.
#32 David Kane
November 24, 2008

Apologies for the delay in replying.

1) I am sorry that I assumed that the search box that Tim (or Science Blogs/Google) provides in the upper lefthand corner does not work. My mistake!

2) dalazal asks:

David Kane, what do you think that Daponte’s points would add to what has already been discussed several times?

For the same reason that Tim links, again and again, to the same arguments made by the global warming skeptics. There are two visions for Deltoid: First, once argument X is dealt with, don’t bring it up again. Second, feel free to revisit argument X over and over again. I am sad that Tim chooses the first option for Lancet while (correctly!) going with the second for AGW. It’s his blog and he can do what he wants, but it is a shame that there isn’t some place on the Web that would do for Lancet what Tim does for AGW.

Do you think Daponte’s criticisms bring something new to the discussion? If so, what?

The only thing I can see, so far, is that she is a professional demographer from Yale, instead of Kaplan from Slate.

Her affiliation with Yale is key, I think. There was a stage in the Lancet debate when folks like, say, Robert and Sortition, tried to pretend that all the critics of the Lancet were hacks. I may very well be a hack, but when someone at Yale is making the same arguments, it become harder to dismiss those arguments out of hand.

But it is easy for Tim to ignore them, to not even devote a new thread to Daponte’s article. It is almost like Tim is scared that the Lancet critics are correct . . .

As to Daponte’s article, here is the key part of what I wrote:

Daponte’s article is fair and professional. If you only have time to read 15 pages about the debate over Iraqi mortality, this is the paper for you. Bottom line:

Perhaps the best that the public can be given is exactly what IBC provides – a running tally of deaths derived from knowledge about incidents. While imperfect, that knowledge, supplemented by the wealth of data of the Iraq Living Conditions Survey and Iraq Family Health Survey (which have their own limitations), provides enough information in the light of the circumstances. At a later date, additional surveys can be conducted to determine the impact and/or do demographic analysis. But for now, the Iraq Body Count’s imperfect figures combined with the date of the ILCS and IFHS may suffice.

Exactly right. No survey is perfect, but combining the information from IBC, ILCS and IFHS is the best way to get a handle on Iraqi mortality. But what about those Lancet surveys? Why does Daponte not even mention them in her conclusion? Because she thinks that they are highly suspect.

You quotes from Daponte are correct but, I think, leave out the most damning one.

The estimates from these [Lancet] students [DK --- I think that this should have been "studies"] have been lauded but also questioned, partially because the researchers have misinterpreted their own figures but also because of fundamental questions about the representativeness of the achieved survey sample.

And that is the key. Daponte does not think that the sample was representative. And, if she is correct, then the Lancet estimate is useless.

Whether or not you think that this criticism has “received considerable attention here at Deltoid” depends on your point of view.
#33 sod
November 24, 2008

sorry, but i seriously struggle to accept any “analysis” that values (english!) newspaper reports in Iraq over a scientific poll.

did you count letters to the editor, before the last US election? looking at japanese ones only?

but this one simply is absurd:

also because of fundamental questions about the representativeness of the achieved survey sample.

now there are obvious question about the representativeness of any sample. but let us look at those questions:

Appropriately, the authors at the end of the article discuss research processes that may have yielded a non-random sample, including: (1) not following up with
households where no one was home when an interviewer went to it; (2) not including households where everyone in it was killed or where all the members of the household had migrated out of Iraq; (3) and the misreporting of the number of deaths, the cause and circumstances of death, and the combatant status of the deceased.
(numbers added by me)

(1) is an obvious problem in a war region. it is causing enormous problems during polls over here as well. most polls will struggle to call back, many simply ignore it.
hardly a serious problem in representativeness

(2) would INCREASE the numbers. this point is contradicting Davids claims!

(3) lancet 2 asked for death certificates. cause of death will always be the result of a question in a poll. and combatant status is irrelevant for the answer of the question, that lancet asked!

sorry, but if there is a real question, i must have missed it. please enlighten me, david!

http://tinyurl.com/48mq63
(sorry, i shortcut the route via your blog david. the link is directly to the pdf..)