IBC's campaign against Lancet study

"the IBC numbers were usually misreported as the number of deaths and the IBC maximum was often reported as an upper bound on the number of deaths."

... which has much to do with IBC's extremely poor choice of terms.

Maximum and minimum are pretty unambiguous terms and the IBC's use of these in the context of a statistical estimate of mortality is not only innacurate, but quite misleading -- whether intentional or not.

Perhaps the time has arrived to collectively help IBC's getting used to life without the spotlight. They played a valuable and unique role in calling attention to the consequences of the war in the early times. Reality has forcefully surpassed their capabilities in a way that they refuse to accept. Let's leave the attention to IBC methods and arguments to the Bush Administration and the unrepentant supporters of the criminal war in Iraq.

Did something go wrong when you pasted his letter? The fourth paragraph is a mess. Also your final link.

I gather Medialens started this whole spat, but apart from revenge, what does Sloboda want? The JHU guys acknowledge that there is a role for passive counts. Why can't IBC just accept that there is a role for surveys? It's not as if policy makers are much interested in any kind of body count, active or passive.

Someone at the medialens message board just linked to this article--it's more evidence for the notion that official Iraqi bodycounts aren't worth much.

http://news.bbc.co.uk/2/hi/middle_east/6045708.stm

As for Sloboda, yeah, people like me drove him bonkers last January or February, I think. I wrote IBC a couple of harsh emails and mine were probably a lot nicer than some. I felt bad later, but I think they still get a lot of flack.

As for medialens starting it, it depends on the point of view. I was very irritated by the triumphalist tone of the IBC two year analysis (which may not be available for downloading anymore), where they calculated percentages of people killed by this and that to 3 digits and barely touched on the possibility that the media sample might not be truly representative. So when medialens launched the attack of the leftist internet hordes, I joined in.

They're social scientists defending their methodology--Sloboda had a slideshow up on the IBC site early this year where it was very clear that he thought this media counting notion was a new and revolutionary way to study the impact of war on civilians. If the Lancet II paper is right, their methodology is seriously flawed and may actually function better as a way of studying the biases of the press and their effectiveness in covering wars like Iraq. It misses most of the deaths and doesn't present a valid breakdown of who is doing most of the killing and when, because if you believe IBC's attributions, the vast majority of coalition-caused deaths occurred in March and April 2003.

"their methodology" means "IBC's methodology". Will stop cluttering up the message boards today.

Oddly enough I've just been watching the BBC documentary Donald linked to; I posted a comment about it at Obsidian Wings but I may as well inflict it on Tim as well. Dammit, I had to look at the blood, so why shouldn't you suffer?

It should be required viewing for the guys (including IBC) who responded to the Lancet study by asking "where are all the wounded?" Sit down and watch them being treated, guys - often without anaesthetics. A doctor said that 90% of all the injuries he sees are war wounds. It's amazing how people cope. They had a blood shortage, so now, when they give a transfusion, they take the patient's ID card. When his relatives come in and donate as much blood as they have given him, he gets the ID card back. I guess if you've lived through wars and economic sanctions you figure out how to keep the system working.

Relevant to Donald's current concern: a remarkable number of people said "No Iraqi would do this!" even though it was pretty obvious that some Iraqi did, unless the Marines are using car-bombs these days. Denial, denial; I think that's at least part of the story with the Lancet figures. I doubt that the respondents invented deaths, but I suspect they blamed the foreigners any time they could.

Dopnald said: "If the Lancet II paper is right, their methodology is seriously flawed and may actually function better as a way of studying the biases of the press and their effectiveness in covering wars like Iraq."

..which is precisely why many in the mainstream media are now attacking the Lancet study.

Of all the failures related to the Iraq war, the most devastating is the complete failure of the media in America to ask questions -- either before the war started (about the rationale(s) for invading) or since (about Iraqi deaths, coalition deaths, corruption, etc).

Without the abject failure of the American media, a very large number of people would still be alive.

In a very real sense, the American media have blood on their hands and their current denial of Lancet is a denial of their own complicity in this whole affair.

, there would never have been a war to begin with.

"The answer from the IBC folks was that they didn't have the time to do that."

Tim, you're a bald-faced liar. The issue of time neccessary to correct real or imagined (as many were) errors by every journalist in a sound-byte, was only one of many disagreements over that issue, as anyone could see even by reading the thread that followed your (error-filled) categorizing of IBC-related sound-bytes from the media.

After your lying and distortions on that point, you then engage in dishonest evasion at the end when you write:

"The UNDP number only covered the first year of the war and only war-related deaths. The new Lancet study is obviously much more comprehensive and gives a better idea of deaths in the war so far."

The point is, when you compare the two studies, one of the two has to be _wrong_, and wrong in a big way.

The fact that Lancet 2 is covers a longer time period is meaningless if it is wrong. So how have these experts determined that Lancet 2 is right and ILCS is wrong. So, what grounds do they have for dismissing the findings of ILCS (assuming they'd even heard of it before endorsing Lancet)?

Tim, do you really need to be such a monumental fraud and liar to puff up this Lancet study? Well, I guess so.

Not too charming, josh. What's your own best guess though at the most (roughly) likely number of excess deaths in Iraq as a consequence of the US attack?

Josh.

I suggest that before you call Tim a liar, you get your own facts straight.

Please explain your statement that "when you compare the two studies, one of the two has to be wrong, and wrong in a big way".

You must account for the following facts which are the main findings of the studies upon which the cumulative excess mortality numbers are based:

Results to the latest Lancet study:
Pre-invasion crude mortality rate: 5.5/1000 per year.
Post-invasion crude mortality rate: 13.3/1000 per year.

Results to first Lancet study
Pre-invasion crude mortality rate: 5.0/1000 per year.
Post-invasion crude mortality rate: 12.3/1000 per year.

And by the way, in case you may not have noticed, things have gotten even more violent in Iarq since the first study, which explains the rise in the post invasion mortality rate from the first study to the last.

JB, the 12.3 mortality rate from Lancet1 includes Falluja and is for a different period, so it's not really comparable.

Josh, if I have misunderstood the reasons you have given for not contacting the media who have misrepresented what the IBC says, I'd like to correct the record.

Your are definately aware of what the media have said about the IBC number -- you maintain a list of those references. Can you tell us then, if you have the time to correct them, why you have not done so?

Given that Lancet II found 87% of reported deaths were issued death certificates, if the study was accurate, there would be records of about 570,000 death certificates. There aren't.

There are so many flaws in Lancet II that I would fail the authors in a first-year quant methods class.

Combat deaths tend to cluster by neighborhood. Therefore, selecting successive houses on a single street does not give you independent variables. Effectively, instead of 1800+ data points, they only effectively had 55 or so. This is too small a sample size to extrapolate from.

[I]f the study was accurate, there would be records of about 570,000 death certificates. There aren't.

And how do you know there aren't?

I still feel that the curious incident of the non-violent death rate deserves addressing. The fact that the 2006 study finds that the non-violent death rate did not rise is surprising for two reasons: because the devastation in Iraq should be associated with increased health risks, and because the 2004 survey did find an increase in the non-violent death rate.

Can Tim get the study's author to comment on this issue?

r81x, because of the clustering of deaths the effective sample was smaller than 1849 households. But it wasn't 55. The design effect was 1.6, so the effective sample size was about 1150.

"The fact that the 2006 study finds that the non-violent death rate did not rise ...and because the 2004 survey did find an increase in the non-violent death rate. "

More precisely, I think, the results weren't statistically significant in either case. As has been pointed out at length, there are big standard errors, so it takes a fairly clear-cut effect to show up.

Burnham et al make a direct comparison between the results of the first survey and the second in "The Human Cost
of the War in Iraq, A Mortality Study, 2002-2006":

http://www.jhsph.edu/refugee/research/iraq/Human_Cost_of_WarFORMATTED.p…

"The 2004 survey compared with the 2006 survey
Since the 2006 survey included the period of time contained in the 2004 survey, we could compare
these two results for the time frame from January 2002 through August 2004. In 2004 we estimated that
somewhere in excess of 100,000 deaths had occurred from the time of the invasion until August 2004.
Using data from the 2006 survey to look at the time included in the 2004 survey, we estimate that the
number of excess deaths during that time were about 112,000.
That these two surveys were carried out in different locations and two years apart from each other yet
yielded results that were very similar to each other, is strong validation of both surveys."

John Quiggin,

The confidence intervals around the non-violent death estimates are smaller than those in the first Lancet study.

I get it as about 140,000 non-violent deaths per year before the war, increasing by 16,000 +/-42,000 per year. Which is to say, zero is much more likely than, say, 40,000 per year.

The statement that "the 2006 study finds that the non-violent death rate did not rise" is not a completely accurate characterization:

Here's what Burnham et al say about it in The Human Cost of the War in Iraq, A Mortality Study, 2002-2006":

http://www.jhsph.edu/refugee/research/iraq/HumanCostof_WarFORMATTED.pdf

"Non-violent death rates
The deaths recorded for the pre-invasion period in both the 2004 and the 2006 surveys were almost
entirely non-violent deaths. (We define non-violent deaths as not due to intentional violence--that is,
our non-violent deaths include deaths in "accidents," such a traffic fatalities.) Immediately post-invasion,
the death rate due to non-violent causes dropped slightly, then stayed level for the next period, but began
to rise in the period from June 2005 until June 2006. The excess death rate due to non-violent causes is
estimated to be 1.2 deaths/1,000/year for this most recent period of time, and 2.0 deaths/1,000/year for
the first six months of 2006. It is not possible to say that this number is a statistically significant increase
over the pre-invasion baseline death rate. However, this may represent the beginning of a trend toward
increasing deaths from deterioration in the health services and stagnation in efforts to improve environ-mental
health in Iraq. (emphasis added)

r8ix: "Given that Lancet II found 87% of reported deaths were issued death certificates, if the study was accurate, there would be records of about 570,000 death certificates. There aren't."

Yes there are. It's just that the central government is unable to recieve and tabulate them all even if they want to. Many have probably just been handed out off the record as well. Bureucracies _break down_ during a war, and comprehensive data gathering like that is demanding under the best of terms.

I've repeated this dozens of times now in the comment sections here, and no one has explained to me why they think that I am wrong, that the Iraqi government should be able (and willing!) to do comprehensive mortality data gathering during a war. Even if you denialists did have an objection, the honest thing to do would be to state that one, and not the original "devastating critique".

Josh,

What is the official soundbyte to deal with the very REAL misrepresentation of the IBC data in the media. The IBC figure is the absolute minimum number of excess deaths, why isn't this even clearly labelled on the IBC website?

You criticise the Lancet 2 report by comparing it to the UNDP report. Surely you're aware that is like comparing apples and oranges. The two studies have very different objectives. The UNDP study focuses on deaths related to the official war. The Lancet study is examining the crude mortality rate with an examination of the causes of death there as well.

If you're going to rudely accuse people of lying and fraud make sure you've got some evidence, or even an argument that makes sense.

After so many blatant Lancet distortions, the reality in Iraq seems more acceptable. The COtW should thank Lancet profusely.

I am among the ones that find Tim Lambert credible, reasonable an fair, and the insolent comments from Josh only reflect on him and on the desperate need of IBC for self-preservation.

That's right Graham. For some people death in the thousands is more acceptable if the ones calling attention to it don't agree with you. In your world that might even be call "moral clarity"

I have no illusions that my drive by critique will move mountains but the gaping holes in analysis bother me enough to write.

1. There is a design problem here. There are people who would wish for grossly high casualty figures to go in circulation. They are called the insurgency. There has been very little commentary here, or elsewhere that I've seen addressing the problem that somewhere, very quietly, an intelligence agent or two is laughing at you. It serves enemy purposes for this data to come out and fighting penetration of the local mortuaries is probably very low on the priority of our counter-spies.
2. If 80% of the deaths in the Lancet study had death certificates, one would expect that 80% of the deaths would have been caught by the official system. Instead, Iraqi government officials have been very loud about how bad/how wrong the Lancet study data is. Asking "how do you know" the government death figures aren't in line is a bit thick. We know because the Iraqi government is telling us so. That's not definitive in a war (see problem 1 above) but it certainly carries more weight than peer review in the Lancet.
3. The pre-war and war deaths trend 3/4 male in the Lancet study. The Iraqi population is not sex skewed demographically anywhere near that figure. This should have been a sign that something was very wrong with the pre-war baseline.
4. One cannot be but be disappointed in the tenor of false choice on offer here and elsewhere. Just because IBC is wrong does not mean Lancet is right or vice-versa. They are independent variables. It is possible that both are wrong (see problem 1 above).

r8ix: "Given that Lancet II found 87% of reported deaths were issued death certificates, if the study was accurate, there would be records of about 570,000 death certificates. There aren't."

r8ix: "Yes there are. It's just that the central government is unable to recieve and tabulate them all even if they want to. Many have probably just been handed out off the record as well. Bureucracies break down during a war, and comprehensive data gathering like that is demanding under the best of terms."

Official Iraqi records show about 49,000 death certificates, IIRC, which at 87% puts the total at pretty close to IDCs. Granted, there are problems with record keeping, but use Occam's Razor. It is more likely that the estimate is off than that the government is missing 93% of the death certificates, especially given the other problems with the study.

Oh, and the study does not appear to have been peer reviewed. Some of the references have dates that are so close to the publishing date that there is no way an academic peer review could possibly have taken place.

r8ix: "Some of the references have dates that are so close to the publishing date that there is no way an academic peer review could possibly have taken place."

That's funny! r8ix apparently has never had a paper peer reviewed. In my experience one of the most common types of reviewer feedback is to add more references, particularly to their papers.

R8ix,

You don't think the study has been peer reviewed? That is a pretty bold assertion based on your own perceptions for how Lancet peer review actually works. For instance, those final references could easily have been included after the 1st proof had been produced shortly before its publication, and this is having read about their submission policy.

Occam's Razor is usually only best applied once empiricism has been exhausted, in this case it hasn't. There are plenty more pieces of information we could find out, and that we should find out, before we blindly reach for Occam's Razor. How degraded is the Iraqi Bureacratic system in the face of widespread sectarian violence?

If you're so keen on Occam's Razor, how about using it to assess the IBC estimates, their assumptions are really stretching belief and reality. Is the media's eye really covering everything that happens in Iraq? But again that would be unfair, know one has actively measured the media's ability to do this yet.

My point, stick with empiricism. Instead of making assertions, and having commentators say this feels wrong or this is about right lets go and have more surveys or lets get some checks done at the graveyards, just like Les Roberts said.

TM, Re your #3, the Lancet study recorded 67 adult deaths in the prewar period, of which 20 were identified as male, 6 as female, and the remainder as unidentified seniors over the age of 65. Given that women live longer than men in nearly every culture, these numbers are not particularly surprising or discrediting.

As for the postwar figures, a higher death rate among non-elderly adult men over women would be what one would expect in civil war conditions. The fact that the Lancet study found that supports, rather than refutes, its overall conclusions.

TM Lutas, some comments on your critique:

(1) and (2) contradict each other - if we should a piori dismiss claims based on assumed political bias, then claims made by the Iraqi government should also be judged with a disbelieving eye. Please note that Sardr's militia guys are running the Ministry of Health. I'll trust their statistics if and only if - well, never.

(3) is rather dim; I'd expect deaths during the war to skew heavily male, until and unless starvation became the primary killer. This is due to both men who are combatants, men who are primary targets of combat, and men being the ones most likely to leave the home when it's really dangerous outside.

(4) As *******anybody who has read the study would know******, the IBC figures actually support the Lancet study. The IBC tally is based on english-language media reported deaths only. This, by references in the Lancet article, should be multiplied by at least a factor of 5. This would yield an estimate of 250,000 *civilian deaths caused by violence* (not all deaths, from all causes). ********If you had read the Lancet article, you'd have known this; if you knew the basis of the IBC project's tally, you'd have known that it was a small subset of actual deaths.*****

Now, you're not stupid; I've read your blog before. You actually have the rare refreshingly sane conservative viewpoint. So why did you put together a very, very 'Red State' level critique?

Tim -

Iraq Body Counts' attempts to impugn the Lancet study commenced just days after publication of the original paper, and several months before the first MediaLens alert

John Sloboda, Iraq Body Count Co-founder
November 5, 2004

"I think you're going to find, in the weeks and months to follow, that there's going to be very, very serious debates and criticism of the [Lancet] study, and maybe, at the end of the day, the figure will be retracted or modified. And one of the lasting problems of this is that then, somehow, everybody who's trying to do estimates of civilian casualties in Iraq might be tarred with the same brush, and the whole enterprise kind of written off."

Not much of an endorsement, is it?

It is more likely that the estimate is off than that the government is missing 93% of the death certificates, especially given the other problems with the study.

I find the idea that a government built from scratch over the past two years, in the middle of a civil war, and run in parts by militia is accurately collating and tabulating 570,000 death certificates to be the most implausible part of your entire formulation.

And, IIRC, the Health Ministry building in Baghdad was looted and burned. That had to help with the record-keeping. Not.

Some of the references have dates that are so close to the publishing date that there is no way an academic peer review could possibly have taken place

It is not at all uncommon for forthcoming papers to be circulated by the authors in draft or proof form, or to be published online well in advance of journal publication date. I've sometimes added final reference information in the proof of a manuscript, not long before publication.

Iraq Biody Count's Sloboda said: "And one of the lasting problems of this is that then, somehow, everybody who's trying to do estimates of civilian casualties in Iraq might be tarred with the same brush, and the whole enterprise kind of written off."

Funny, I thought the whole emterprise of counting bodies was written off long ago -- after the American army realized counting bodies had caused them so many "problems" in the Vietnam war.

Let us not forget General Tommy Frank's famous "We don't do body counts" remark at the beginning of the Iraq war.

John Sloboda is just kidding himself if he thinks anyone in the Bush administration gives a damn about the number provided by Iraq Body Count (or by anyone else, for that matter).

They don't do body counts because they know that the American public can not stomach the thought that their policies may be killing lots of people (even though the leadership in Washington could not care less about this).

Burnham et al. has attracted considerable controversy. How does it compare with other tallies?

The Burnham et al. survey surveyed 47 clusters and about 1800 households. This resulted in a fairly broad range of estimates for either excess deaths or deaths by violence since the invasion. They estimate somewhere between 425,000 and 800,000 deaths by violence since the invasion

Comparison is tricky, because there have been several attempts to measure the number of deaths, each using different methods over different periods and using different criteria. The Iraq Living Condition Survey (ILCS) was also a survey based estimate: 22,000 households in 220 clusters were interviewed. As well as looking at a large number of other questions, surveyors asked the number of household members that died in war-related violence both in the year following the invasion and the year before. The surveyors estimated 24,00 Iraqis had been killed in this way, almost all of them post invasion, and estimated with 95% confidence that the number was between a 18,000 and 29,000.

Crudely adjusting Burnham et al.'s estimate for the 14 month postwar period to 12 months, it would appear that they believe that during the same period about 34,000 violent Iraqi deaths were directly attributable to coalition or "other" combatant groups. Another 43,000 were attributable to "unknown" killers. The unknown category included normal criminal murders, but also deaths where bystanders were killed as a result of combat between the coalition and its opponents but the respondent couldn't or wouldn't say which side was responsible. It would also include some deaths caused by killers the respondent couldn't or wouldn't identify, but that they might have classified as "war-related violence" if the question had been framed in that way.

In addition, the Burnham et al. survey seems to have been designed to exclude deaths of household members who didn't live in the household for three months before their death. This would appear to exclude soldiers living in barracks. There a various estimates of how many Iraqi soldiers were killed in the invasion itself, but 10,000 combatants (http://www.comw.org/pda/0310rm8.html) seems to be a conservative estimate. Probably no less than half of these were regular soldiers not living at home.

To make Burnham et al deaths equivalent to those measured by ILCS in their survey period, it is necessary to make some adjustment for both military deaths and deaths that ILCS respondents would have considered something other than "war-related". One approach would be to compare the ILCS "war-related" total, (ca. 24,000) with the Burnham et al total for deaths caused by "Coalition and others" in the same period (34,000) plus some addition for the above categories. If 1/3 of the 77,000 violent deaths (and more than half of the unknown deaths) are ascribed to crime and other non-war related violence, and 5,000 Iraqi military deaths are added then Burnham et al. implies some 56,000 war related deaths in the ILCS period. This is 233% of the ILCS estimate: a discrepancy difficult to explain by ILCS respondents failing to mention relatives killed in the previous year.

Crudely adjusting Burnham et al.'s estimate for the 14 month postwar period to 12 months, it would appear that they believe that during the same period about 34,000 violent Iraqi deaths were directly attributable to coalition or "other" combatant groups. Another 43,000 were attributable to "unknown" killers.

Will, can you explain how you get this please? You seem to be coming up with a violent mortality rate in the region of 3/1000. The study actually gives 1.5 (CI 1.1-2.0) for March 03 - April 04.

Will, apologies - I was looking at the risk ratio in Table 4. The violent mortality rate was 3.2 (1.8-4.9). I'm still puzzled by your figures, though. Table 4 shows 45 deaths in 03/04 but the breakdown doesn't seem to match your's.

Kevin write:

"Will, apologies - I was looking at the risk ratio in Table 4. The violent mortality rate was 3.2 (1.8-4.9). I'm still puzzled by your figures, though. Table 4 shows 45 deaths in 03/04 but the breakdown doesn't seem to match your's."

That may be a result of cumulative rounding error on my part. Doing the calculation another way, I get 83,000 violent deaths in the relevant period from Burnham et al, based on 3.2 deaths per 1,000 per year on 26 million Iraqi population quoted in the paper. That would break down to 37,000 from "Coalition and Other" and 46,000 "unknown". Assigning 18,000 of the "unknown" category to crime and other non war-related violent death, that makes war related death from Burnham et al. 65,000 plus at least 5,000 soldiers living in barracks. So the gap between Burnham and ILCS becomes even greater.

Will McLean

Several of us have pointed out the discrepancy between the ILCS and the Lancet 2 paper--part of the difference could come from criminal violence. One could also take the low end of the Lancet 2 confidence interval (1.8), which gives about 50,000 deaths.

Someone here (Robert, I think) pointed out that the ILCS survey was 60 pages in English translation and the median interview time was 80 minutes or so, and someone at another blog (in the comments section) said that the war-related violence question was on page 49. Those two facts (assuming the second one is right), along with the fact that the head of the ILCS survey admitted so some doubts about the professionalism of his surveyors (I read a quote to that effect in Carl Bialik's piece, but don't have the cite handy) makes me think we shouldn't necessarily be using the ILCS survey as the gold standard for comparison purposes. One could just as easily use Lancet 2 to cast doubt on the ILCS survey.

Not that I know which is right. But if we're going to pick on the Lancet 2 survey then every survey should get the same skeptical treatment.

Will,

Thanks for your response. I'm quite puzzled by the ILCS, which I didn't study when it came out. What's a war-related death? Judging by Table 39 of the Analytical Report, it's something that was much more likely to happen in the South. Here are the figures, central estimates of numbers of deaths (with deaths per thousand in brackets):

South: 12,044 (1.24)
Baghdad: 7,547 (1.15)
Centre: 3,686 (0.51)
North: 466 (0.13)
Total: 23,743 (0.87)

Considering that the Centre includes Anbar it seems to have got off pretty lightly. Of course there are some trouble spots in the South too, notably Kerbala and Najaf. But looking at those figure I'm inclined think they must reflect the shock-and-awe phase of the war - major combat operations - rather than the long drawn out agony of the insurgency. 0.5/1000 looks way too low for the Centre otherwise.

Another thing that puzzles me is the lack of information on other causes of death. The questionnaire asked if anyone in the household had died or gone missing and the causes were listed as disease, traffic accident, war-related, pregnancy-related and Other. But the only table I can see is the one for war. It would be useful to have the others, just to get a sense of how complete the figures are.

Carl Bialik wrote in 2005:

"Lead researcher Jon Pederson, FAFO's deputy managing director, also told me that he had reservations about the Iraqi statistical workers who carried out the research -- not their impartiality, but their techniques. Statistical workers who have worked under dictatorships "tend to develop a fairly loose relation to data because they know that things will be changed by the government, and that leads to a sloppiness in field work," Mr. Pederson says, though he adds that the statistical chief, installed after the removal of Saddam Hussein, was committed to accuracy."

Note that the same concern might be raised with respect to the Burnham et al. surveyors. They worked under the previous regime, and misplacing more than one cluster suggests less than optimal technique.

"We urge open and constructive debate..."

Sorry, I don't debate with traitors like Les Roberts.

I prefer the "try, and execute or imprison" method.

My compliments to the person who created the Dave Surls parody character. Nice.

Best,

D