More on the ORB poll

I must confess I'm a bit disappointed in the wingnuts. When Will pointed out the anomaly I thought they would appear in droves, scenting another Rathergate: poring over the tables and finding umpteen irrefutable proofs that the poll is a fabrication; "Stick a fork in it, it's done", etc. But even Seixon doesn't seem to have his heart in it any more.

Nicely done by Will.

More or less on topic: Via Robert Naiman check out the Washington Post's strange Iraq casualties widget (go to the bottom of the page).

According to WaPo, the "maximum count" on Iraqi civilian casualties is 79,999. If you click through their "About these figures" link, you'll find out this number comes from IBC. They even mention the 655,000 Lancet number, but somehow the cognitive dissonance with 80,000 or so as a maximum doesn't occur to them. (Even the Lancet reference is a little misleading, as they don't extrapolate it to the present day which would get you a number in the vicinity of the ORB number.)

Also, WaPo cites 159 U.S. civilian / contractor deaths; which seems low. Indeed, the New York Times says that "the total number of contractors killed in Iraq [is] at least 917, along with more than 12,000 wounded in battle or injured on the job, according to government figures and dozens of interviews."

Ah, the state of journalism today. Also check out the Robert Naiman link for some shenanigans at the Cleveland Plain Dealer.

It was indeed well spotted by Will ... it shows how being sceptical and examining the data can actually improve things.

Note that no polling took place in Anbar and Karbala, two of the most violent provinces, for security reasons. Hence Baghdad seems to the most violent place in Iraq in this poll.

The gender demographics for Baghdad still seem to conflict with the reported violent death rate. 421 households, the weighted number for Baghdad, would have had about 2750 members in 2003, if their size was typical of Iraq, with about 700 adult males and a like number of adult females. According to both media reports collected by Iraq Body Count and those polls that asked the question, adult males represent about 90% of the violent deaths in Iraq since the invasion. It is plausible to estimate that of the 295 violent deaths reported in the weighted ORB poll, about 250 were adult males (18 or older), 15 were adult females, and the rest were children. The adult survivors would then be about 450 males and 685 females: 40% male and 60% female. Children coming of age since 2003 might reduce the imbalance slightly: to 42%/58%, assuming no sexual disparity in deaths among those who were 14-17 at the time of the invasion. This last is not necessarily a plausible assumption: 14-17 year old males are probably at higher risk than females of the same age.

ORB reported that their adult Baghdad respondents were 49% male. Plausible extrapolation of the reported violent death rate above, and the age and gender of reported victims of violent death in Iraq, suggest that given that death rate Baghdad adults should only be 40-42% male. That's a significant conflict

There's another reason to distrust the Baghdad death rate in the September ORB poll. In a poll they released in March, they asked a similar but broader question: had the respondent had a relative murdered in the past three years? 26% said yes: 31% said so in Baghdad, and 24 % in the rest of the country. In the September poll, 12% outside Baghdad has lost at least one household member. If we assume that the Iraq Body Count was a fairly consistent undercount during the preceding six months and use it to estimate the change in cumulative violent deaths, that would have been 10% at the time of the March survey. Outside Baghdad, the two polls seem in reasonable agreement: it seems entirely plausible that an extended family in Iraq is about 2.5 times the size of the immediate household.

But extrapolating the March Baghdad responses forward on the same basis predicts a household death rate of 15%: higher than the rest of Iraq, but less than a third of that reported for Baghdad in the September poll. Alternatively, projecting the September Baghdad results backwards implies that Baghdad extended families are only 25% larger than the immediate household living under one roof. That seems highly implausible.

Will McLean wrote:

ORB reported that their adult Baghdad respondents were 49% male. Plausible extrapolation of the reported violent death rate above, and the age and gender of reported victims of violent death in Iraq, suggest that given that death rate Baghdad adults should only be 40-42% male. That's a significant conflict

Will, you shoulda stopped while you were ahead. Now you're getting scary. Respondents often don't reflect the overall adult sex ratio -- sometimes that's by design (as when you want to query the head of the household, or when you're trying to get specific types of information (such as child health status)), sometimes that's a function of the time of day or the day of the week that you contact that household, sometimes it's a function of the household composition and size (single person households are more likely to be male than female).

Robert:

If there was a survey bias towards interviewing males, wouldn't we see it in the overall numbers? One could imagine timing or cultural factors that would make the respondents disproportionately female relative to the actual household, such as the interview being done while the men were at work. However, to mask the demographic impact of the reported deaths you'd need to interview a dispoportionate number of males, and that is harder to explain, assuming no survey bias.

Even if you can explain the apparent reported gender/reported deaths conflict, there's still the striking disparity between the September and March results for Baghdad

A bit of a nitpick, but I don't think it's safe to use IBC as a consistent measure of the level of violence, even though people tend to do this (including, I think, the Lancet authors). If the level of violence goes up, it seems quite possible that the percentage of reported deaths might go down. Or it might not--in any event, I don't think the IBC numbers can be trusted as anything other than as a minimum figure for the number of deaths.

Will McLean asked:

If there was a survey bias towards interviewing males, wouldn't we see it in the overall numbers?

Not necessarily. Sometimes, yes, sometimes, no, because survey bias doesn't always translate into response bias.

There are two deeper issues, one statistical and one substantive. The statistical issue is that in examining dozens (or hundreds) of crosstabs, one is bound to find oddities. You may be familiar with multi-comparison corrections, which are (crudely and not always successfully) intended to handle this kind of problem. In any event, the substantive issue is rather more interesting: what's the mechanism by which we'd expect that an unusual distribution in respondents' characteristics leads to biased response? This is quite a difficult problem, and I don't know the generalized answer. For a concrete example, women turn out to give fuller and more precise responses about family event histories than men -- that's why all of the Demographic and Health Surveys that focus on child health have protocols to select a (particular) female respondent; however, while male respondents give less precise answers, that doesn't (always) mean they give biased answers. One way to think of it is that men aren't as good in remembering birthdays and ages and illnesses and vaccination records of their kids, but they're just as good as women at remembering how many. IOW, the big question in this case isn't "is there systematic bias in the characteristics of respondents" but "even if there is systematic bias in the characteristics of respondents, will the answers be biased; and, if so, in which direction and by how much?" Note that some of the questions on that ORB survey had to do with availability and affordability of food. It's easy for me to think that the answers to those questions would differ between male and female respondents, but I'm not sure which way the bias would run. It's the same for number of deaths within the household.

Slightly off-topic, but here's a post by a reporter in Iraq who admits she has no way of knowing what the true death toll is.

http://www.sacbee.com/110/story/392583.html

[I don't think it's safe to use IBC as a consistent measure of the level of violence, even though people tend to do this (including, I think, the Lancet authors). ]

Donald is right about this; Burnham et al were challenged on this in the review issue of the Lancet and agreed that it was a mistake and they shouldn't have done it.

Donald Johnson:

I think it's quite plausible that the level of undercount by IBC may have changed over time, but I think this is less of an issue when the polls are fairly close together in timing. And it's clear that comparing polls at different times requires some adjustment for changes in the cumulative death toll. The IBC cumulative measure is an imperfect way to do that, but I'm not sure there's a better one.

dsquared,

What are you referring to in #11? It is certainly true that Burnham et al were challenged on their ridiculous graphic which plotted levels and changes as if they were the same thing. They did, indeed, latter admit that this was an error, one not caught by the Lancet peer review process. I am unaware of any other "mistake" that they have admitted to.

But if this the incident you are referencing than I don't think that it has anything to do with Donald Johnson's point. The Lancet authors still claim, as far as I know, that the IBC data supports their estimates, once you account for the undercount. That is, I think, IBC data shows more deaths in the 2006 than 2004, consistent with the Lancet surveys. Have the Lancet authors ever argued that it is not the case that IBC is "a consistent measure of the level of violence?" Not that I have seen.

Now, just because they haven't argued the contrary doesn't mean that they believe it, but if you have a relevant citation, please share it.

Donald Johnson, dsquared, Will McLean: I agree with Will that the assumption of a constant IBC undercount is only justifiable because the IBC count is the only consistent, honest, and regular series to go on. As Donald says, the IBC undercount may be growing; the factors here - less physical access by journalists and other sources, a declining chance of a given incident being reported as there are more of them - certainly point that way. The IBC could be getting better at what they do, but this seems a weak effect. So a constant undercount is a conservative assumption. If you drew a trend line between the four surveys you would get a higher growth rate than the IBC's.

In the arena of political judgment, it doesn't matter much whether the mortality is 500,000 or a million. (It matters of course enormously to the families concerned.) The important truth, now proved beyond reasonable doubt, is that the second Iraq war was and continues to be a gigantic humanitarian disaster.

The September ORB poll doesn't mesh well with the one carried out for ABC News, USA Today, the BBC and ARD German TV by D3 Systems in February-March. In that poll, 17% reported that a family member living in their household had been physically harmed by the "violence that is occurring in the country at this time". If we assume that the ORB figures for non-fatal harm and multiple fatality households are in the right ballpark, that suggest 400,000 violent deaths by that time, scaled to about 500,000 by August following the IBC trendline.

I think there is a lot of noise in the poll numbers, particularly the ones that left substantial swaths of the country unsampled: Burnham et al and ORB's September survey. Just quoting the standard error margin for sample size doesn't capture the uncertainty.

While it has been argued that that just means the most volatile areas are unsampled and the violence is undercounted, that may not be true. If you look at the governorates skipped by Burnham et al or ORB's September release, Al Anbar and Karbala were almost certainly above average in violence. But Wasit seems to be somewhat below average, and Irbil, Muthanna and Dahuk were very safe by Iraq standards.

Will,

The phrase you quote, the "violence that is occurring in the country at this time" is hopelessly vague. How far back does "this time" stretch? For example, in an area which experienced more-or-less continuous violence from the time of the invasion onwards, it could easily mean the entire period since March 2003. But in areas where the violence intensified sharply after a more recent event it would not be natural to interpret it that way. Another example: does it include soldiers killed during the "major combat operations" or not? We have no way of knowing.

Kevin, you write:

"The phrase you quote, the "violence that is occurring in the country at this time" is hopelessly vague."

While it isn't expressed precisely, if I was living in Iraq and had lost a household member to the conflict any time after the invasion began in 2003, I would not hesitate to say yes. Probably for the pre-invasion bombing as well, but I don't think that affects the numbers much.

Will McLean

Fair enough Will, so would I. But then if asked whether Iraq was in a state of civil war I would answer yes to that question also. A majority of Iraqis did not. So I'm not persuaded that Iraqis use Arabic the same way I use English. Even when the questions are explicit this can be a problem. When they are not the case is hopeless.

http://www.prospect-magazine.co.uk/article_details.php?id=9777

Read this sad account of a family that should have been among the backbone of a democratic Iraq, now in terrified flight.

It gives a picture of the human story behind each poll figure.

James correctly said the 'second Iraq war was and continues to be a gigantic humanitarian disaster'. But it goes much farther than that. The second Iraq war was and is a massive crime and a violation of international law. It was aggression, pure and simple: the 'supreme international crime' for which the aggressors in the Nazi leadership were tried at the Nuremburg tribunals and subsequently hanged.

Oops. Here is the correct link for the WaPo Iraqi Casualties widget (see #3 above).

USA Today has an article today noting that for the first time, the US has released casualty figures for the insurgents:

http://www.usatoday.com/news/world/iraq/2007-09-26-insurgents_N.htm

The article notes that in addition to the 19,000 "militants" killed in the fighting, "There are 25,000 detainees in U.S. military custody in Iraq, according to the military. The numbers of enemy killed and detained would exceed the estimate given last year of the size of the insurgency."

It's also worth noting that "The numbers do not include enemy killed during the invasion."

Thanks, Bruce. I've been wondering about that number (insurgents killed) for years. Assuming it is accurate, 19,000 is somewhat smaller than I would have guessed, and while I think it supports my suspicion that the number of civilians killed by the US is much greater than what IBC has counted, it also seems a little small to be consistent with the Lancet2 claim of 180,000 or more killed by coalition forces. I doubt the US kills 8 civilians for every militant. I'm leaving out the deaths during the invasion phase, but I think the bulk of the deaths inflicted by the coalition were inflicted post-2003, if we go by Lancet2. I don't have the paper handy, but I suppose that would pull down the civilian/militant ratio somewhat.

Of course, who knows if the figure is accurate? Still, it's more data (valid or not) than we had.

To clarify what I meant, IBC counted about 500 civilian deaths caused by the coalition in the fourth year of the war (March 2006-2007) and about 370 in the year before that. They counted about 7000 civilian deaths in the opening two months of the war and a bit less than 2000 for the 2004 assaults on Fallujah and other than that, they seem to count about 1-2 civilian deaths per day caused by the coalition. Which seems improbably low if the post-invasion militant death toll is 19,000, at least compared to other guerilla wars that I know about. If Lancet2 is correct (31 percent of the deaths attributable to the coalition and many deaths were by unknown actors, so the total could be higher), you'd have 180,000 deaths and if you subtract off the ones in 2003 (whatever that number is), it still seems high for 19,000 militants unless the US forces are being extremely careless (to use an understated word).

I agree that it's very difficult to reconcile 19,000 combatant fatalities with Lancet2's 180,000+ civilian deaths caused by the coalition.

On the more general topic of the ratio of civilian deaths to combatant deaths, I believe that it's extremely difficult to apply guidelines derived from previous conflicts. It appears to me that Iraq is suffering from two concurrent crises: an insurgent war against an occupying power, and a campaign of ethnic (or more accurately, religious) cleansing. What ratios do we apply? The ratio of (violent) civilian deaths to combatant deaths in Vietnam was, I believe, about 1.45 to 1. But what was the ratio of civilian deaths to combatant deaths during the Rwandan genocide? A hundred to one? Two hundred to one? I have no idea... but I'm pretty sure it wasn't 1.45 to 1. Obviously, Iraq is not Rwanda, but the central point still applies: the ratio of deaths in a conventional war is going to be vastly different than the ratio of deaths in an instance where an armed force is deliberately targeting civilians.

I'm not making any point about the overall level of deaths, just the ones inflicted specifically by the US. Both the Lancet poll and (indirectly) the ORB poll suggest much much higher numbers for that subcategory than IBC, of course, but they also seem high compared to the 19,000 figure--I'm guessing militant deaths and "collateral damage" deaths would be in the same range. I think deaths inflicted by the US are the ones we hear the least about (unless IBC is right and virtually all of them are faithfully reported).

ORB says 9 percent from aerial bombardment, which would be in the 100,000 range and presumably refers to US-inflicted deaths. I don't know if they give any error bar on that number.

Donald:

Burnham et al (Lancet2) doesn't measure *civilian* deaths. None of the survey based estimates make any attempt to distinguish between combatant and civilian deaths.

Also, expressing the deaths blamed on the coalition in Lancet2 as 180,000 is way too precise. On sample size alone, the range is probably from 80,000-300,000.

Will: You're right (again). Thanks for the correction.

Donald: I agree that if you restrict the application of ratios to the fatalities caused by the coalition, the comparison with previous conflicts becomes more meaningful. I'm curious about one thing, however: why do you think that deaths inflicted by the US are the ones we are least likely to hear about? I'm not disagreeing; I just don't know what would lead you to make an assumption one way or the other.

I know the 180,000 is too precise--they gave a CI for the percentage attributed to the coalition--26-37 percent and there were many on top of that where responsibility was unknown. Of course there's also a CI for the total violent death toll and I don't know how to piece the two estimates together (or if that's a proper way to calculate it). Also, some of those deaths occurred during the invasion phase, though they also say the level of coalition-caused death increased each year. I know the Lancet2 survey doesn't distinguish between civilians and insurgents--the point is that 19,000 is only a small fraction of what Lancet 2 seems to report for coalition-caused deaths, so if you believe both then the coalition is killing many more civilians than insurgents.

To Bruce--In the stories I read about Haditha it was stated as a defense that what the marines did was standard practice and I also remember reading a John Burns piece in the NYT (early June 2006) who said that civilian casualties were a common occurrence in the fighting that occurred involving US forces. He gave no numbers, but since I rarely read about such stories I suspect we don't hear about most of them. Also, I think that reporters are largely dependent on the US and Iraqi government for information about deaths inflicted by our side--we only now have a number for the killing of insurgents. So if our side kills civilians I'm guessing this won't necessarily be reported. Look at IBC statistics--they show a very high number of US-killed civilians in the opening months and I think those numbers were gathered some months later by reporters who went to various hospitals piecing together the story. Back then it wasn't hard to say who was responsible for the killing. As the war dragged on it's become harder for reporters to travel around Iraq independently gathering information and my guess (it's only that) is that, since they have to depend so much on official sources, they are least likely to hear about US-inflicted civilian deaths. (Though if the Iraq government, such as it is, ever turns against us that might change.)

It is a puzzle that in both the ORB and Lancet 2 studies coalition forces apparently cause a sizeable number of deaths, but you'd never know that from reading the press. Even the 19,000 dead insurgents suggests there are probably a fair number of unreported civilian deaths accompanying those, but maybe not as many as L2 suggests. I'm not being snarky about this--I don't know what's going on over there

Aren't there equally valid reasons to assume that other casualties are underreported? If reporters are only able to travel in the company of coalition forces, wouldn't that make is less likely that they would report casualties in other areas? I would expect that sectarian violence peaks when coalition forces are absent. (After all, plenty of people have been telling us that we'll have a bloodbath on our hands if we're not there. That, of course, is a subject for a different article.) Put another way: if a corpse falls in a charnel house, and there's no one around to hear it, does it make a sound?

I don't mean to criticize IBC, by the way: despite its flaws, their work remains one of the few benchmarks. I wonder, though, if their tasks is becoming increasingly futile. As more and more deaths occur, each individual death becomes less newsworthy, and hence, less likely to be reported.

That's true, Bruce. I don't know which category is most underreported--I would just guess that US-killed civilians is pretty high on the list.

Donald Johnson writes:

"It is a puzzle that in both the ORB and Lancet 2 studies coalition forces apparently cause a sizeable number of deaths, but you'd never know that from reading the press."

In the case of ORB, I don't see a contradiction. The only deaths you can say with confidence were caused by the coalition were the airstrikes, which caused deaths in 2% of the sample households. Since the margin of error for sample size was 2.5%, that poll supports any number between 29 and about 200,000.

20,000 dead militants seems conservative, as does 10,000 dead Iraqi combatants in the invasion phase. IBC also recorded thousands of dead civilians in the invasion phase.

Thousands in the invasion phase, dropping (except for Fallujah) to several hundred per year in the post invasion phase when 19,000 insurgents are killed. Seems implausible.

Donald Johnson writes:

"Thousands in the invasion phase, dropping (except for Fallujah) to several hundred per year in the post invasion phase when 19,000 insurgents are killed. Seems implausible"

Why? During the invasion phase, the US destroyed a large conventional army with paramilitary supporters, something more than 300,000 armed combatants, in about 40 days. It probably had to kill more than 10,000 men to do it. While fighting insurgents after the invasion, it seems to have killed an average of about 5,0000 militants a year.

Given the higher intensity of invasion phase combat, I don't see anything implausible about much lower collateral damage after the invasion.

Your own guess is 10,000 soldiers killed, many of them probably in the desert, and by IBC's figures 7000 civilians died. So if 19,000 insurgents are killed, many in urban fighting, the civilian death toll is likely to be higher than in the invasion phase. Yeah, the intensity is lower per day, but the total number of deaths is high. Guerilla wars in general are bad for civilians, from what I've read, and if X insurgents are killed, usually more than X civilians die in the process. I posted some figures from B'Tselem, the Israeli human rights group, which showed this to be the case in that conflict.

"The only deaths you can say with confidence were caused by the coalition were the airstrikes, which caused deaths in 2% of the sample households. Since the margin of error for sample size was 2.5%, that poll supports any number between 29 and about 200,000."

A couple of questions. First, I think it says 29 deaths from air strikes, (it's hard to read the survey on my computer screen and I can't print it out)and if that occurred in 29 households, that's your 2 percent, I think. Do we know it's 29 separate households or am I missing something in the survey data? (Which is easy for me, given the fine print.) Of course if it is fewer it supports your point better.

Second, I'm not one of the statistics experts here, but does a 2.5 percent margin of error mean you can assume that if 2 percent of households were hit, it might really mean anything from 0 percent to 4 .5 percent? I thought, cluster sampling technicalities aside, if you sampled 1500 households and got 29 positive responses, it's sorta like a Poisson distribution and the variance would be 29 and the margin of error roughly 29 plus/minus twice the sqrt 29, which translates into--well, I'm too sleepy to figure it out, but I think you multiply by about 4000, so that's tens of thousands at the lower end. There's probably some reason why the error bar should be bigger, but I doubt a survey of 1500 households (out of 4 million) finding 29 deaths from air strikes means it's plausible there were only a few dozen deaths from air strikes.

Anyway, I realize I'm just speculating and don't have anything else to say on this.

Yes, the 2.5% margin of error assumes that the proportion found was 50% -- the margin is smaller for smaller proportions. According to this [handy calculator](http://faculty.vassar.edu/lowry/prop1.html), for sampled 2% of 1500, the 95% CI is 1-3%.

Donald #35:

The invasion phase didn't just kill a lot of Iraqis. It also involved a *lot* of airstrikes: about 20 times as many bombs as the occupation to date. And airstrikes are particularly likely to produce collateral damage.

Also, the invasion period in the IBC database is heavily dependent on mass statistics from hospitals, morgues and burial societies. I doubt they had much inclination or ability to record which dead men in civilian cloths were civilians and which were fedayeen.

In contrast, many of the civilians caught in crossfires post-invasion wouldn't leave evidence in the news reports which side fired the bullet. So civilian casualties caused by the coalition would be understated by an unknown amount in the IBC database.

Using Tim's handy calculator gives a minimum of about 40,000 deaths from airstrikes from ORB. Theres probably more noise in the figure than error for sample size, given that there are places they couldn't sample, a lot of people refused to answer the question, and the regional population figures they used to weight their sampling are unreliable. 20,000 dead insurgents, 10,000 Coalition caused civilian casualties from IBC and 10,000 invasion phase totals 40,0000, but each of those counts is probably low.

Thanks for the statistics link, Tim.

Will, I suspect there are also going to be civilians killed by gunfire if there's fighting between the coalition and guerillas in urban areas. And one bomb dropped in an urban area might cause a lot more collateral damage than ten bombs dropped on military targets in the desert. If the US has killed 19,000 insurgents, I'm going to go with historical analogies and assume there's a comparable number of dead civilians to go with them.

But anyway, if you agree that it might be hard to distinguish between insurgents and civilians and you also think the US has killed tens of thousands since the invasion phase, we don't have much to argue about here. Unless you think the US has killed those 20,000 or so insurgents with pinpoint precision. (But even then, there's not much point in arguing, since neither one of us can prove anything.)