Daniel Davies comments on the attempted disproof by incredulity of the Lancet numbers:

I am curious as to why anyone is bothering with this debate any more (in some of the discussion on Dr Madelyn Hsiao-Rei Hick’s comments, it has got parodic, as people discuss the minutiae of the “informed consent” requirements of the questionnaire). Does anyone think at this late date that they are going to come up with a result that proves that the whole war and occupation has been really good for the Iraqis? Have they not noticed that this debate (and the one on global warming too) is a bit like the Berlin Wall – people are only going from one side to the other in one direction?

This prompted a response by Jane Galt who comes up with the macaroni and cheese argument against the study:

But what I wanted to blog about is a somewhat related phenomenon, which is the systematic human tendency to underestimate how long things take. This was driven home to me rather poignantly when I went up against Spencer Ackerman in Blogging Chefs, and tried to estimate just how much I could do in 90 minutes. Then I tested how long it actually took to, say, cook macaroni and cheese.

Well, the best way to find out how long it takes in the field is to do such surveys. From the story in Nature:

each team split into two pairs, a workload that is doable”, says Paul Spiegel, an epidemiologist at the United Nations High Commission for Refugees in Geneva, who carried out similar surveys in Kosovo and Ethiopia.

I commented at the time:

you’d hope that this finally puts the matter to rest.

Apparently not — I had not anticipated the macaroni and cheese argument.

Galt also gives us this:

The more Les Roberts talks, the less confident I get in his results; he doesn’t seem to have the faintest clue how the interviews were conducted. He keeps changing the number of interviewers, the number of houses (which has suddenly and without explanation dropped to 30 in some of the arguments) or who asked what. And somehow now all the interviews were done in 3 hours? At 20 households per team, that’s 6 houses an hour, ten minutes a house including walking, peeing, and informed consent.

The more Galt writes, the less she seems to understand. Roberts hasn’t changed anything. Roberts said that in 2004 (when they used 30 house clusters), they took about three hours, which is about 12 minutes per house. He says that the 2006 interviews took 15-20 minutes. He has not contradicted himself as Galt claims, but is discussing two surveys that took similar (but not the same amount of time per interview).

When Galt misunderstand Roberts, it is wrong to say that he “contradicted” himself. And far from reducing confidence in Roberts, it reduces confidence in Galt.

Need I point out that if Davies is right, and Burnham et. al. are right, then we should be seeing massive floods of refugees?

I guess that Galt took so long the make the macaroni and cheese that she didn’t notice the thousands of news stories about Iraqi refugees.

The UN refugee agency believes that about two million refugees have fled Iraq. On the other hand, macaroni and cheese takes longer than you think to make, so the refugee agency could be wrong.


  1. #1 Donald Johnson
    March 19, 2007

    Here’s an interesting article–http://www.guardian.co.uk/Iraq/Story/0,,2037219,00.html

    Notice the last paragraph, which is why I link to it. A poll found that one person in four had lost a close relative. Too bad we don’t know how many people would fall into the typical Iraqi’s notion of “close relative”. It’s sort of irritating–why not just ask if someone in the household had died?

    Anyone know anything more about this poll?

  2. #2 Kevin Donoghue
    March 19, 2007
  3. #3 Donald Johnson
    March 19, 2007

    Thanks, Kevin. Very frustrating. I could see why they’d ask about relatives being murdered, since it would tie into the level of unhappiness, but why not add just one or two extra questions and ask about whether someone in the household had died? Yeah, someone might change households, but it would pin the number down better. Or they could ask about dead siblings and/or children or parents vs. more distant relations.

  4. #4 Kevin
    March 21, 2007


    You say that Burnham puts the matter of the timeframe to rest but for me it just makes more questions.

    Nature says that an interviewer in 2006 claimed they did not work independently of one another and when asked again by Nature about this particular inconsistency, Roberts replied that he was referring to 2004 and not 2006. Burnham claims that they also went singly in 2006. How is this supposed to help?

    The Lancet study group’s response changes at every single objection.
    1) Too hard for two groups of four to pull this off? The groups of four split into twos.

    2) Too hard for groups of two to pull it off? The groups of two split into singles.

    3) The interviewer for Lancet in Iraq says they didn’t split into singles? I was talking about 2004 and not 2006.

    4) The whole narrative is now inconsistent? Well they “sometimes” split into singles.

    Burnham’s claim about them sometimes going singly, and I still don’t understand why if there is some substantive reason for picking teams of two and specifying one male and one female then going singly is okay, still contradicts the direct report of one of the interviewers, according to the Nature article.

  5. #5 Donald Johnson
    March 21, 2007

    The ABC/USA Today/BBC poll seems to come closest to replicating the Lancet mortality survey. Not very close, but it’s the closest yet.

    The link is here–


    One Iraqi household in six had someone “harmed” by violence. Presuming 6 people in the average household, that would mean at least 1 person in 36 were “harmed”. (Or somewhat higher, since some households would have 2 or more victims.)

    This sounds like a number that is probably lower than Lancet 2, since I’d assume the majority of those “harmed” were wounded, not killed. But 1 out of 36 would be about 700,000 people.

  6. #6 JB
    March 22, 2007

    This sounds like a number that is probably lower than Lancet 2, since I’d assume the majority of those “harmed” were wounded, not killed. But 1 out of 36 would be about 700,000 people.”

    Any way you look at it, one hell of a lot of people have been killed/maimed/wounded/harmed/traumatized/orphaned/affected (pick your adjective) by the war.

    All this debate about what the “actual” number is (which really depends on on your definition, at any rate) is a sideshow (or perhaps sidetrack [or perhaps derailment in some cases]).

    An order of magnitude estimate is certainly sufficient to gauge how bad things are in Iraq, and by most measures (including the IBC number), they are pretty damned bad.

    At some point one has to cease debating what the “actual” number is and start doing something about the problem.

    Unfortunately, some have used the debate to derail that process.

  7. #7 Donald Johnson
    March 22, 2007

    My above estimate is conservative–some “harmed” households could contain more than one harmed individual.

    I agree that the war is a disaster, but I’m sure you agree, JB, that if the death toll is in the hundreds of thousands it is important for Americans (and others) to know this, rather than believe a much lower figure if it is wrong. This new poll suggests the correct figure might be in the hundreds of thousands–whether it agrees with the Lancet2 CI is secondary. What also matters here is that it is clear that people could replicate the Lancet study in all of its details if they wanted to. There shouldn’t be any need to wonder how many “harmed” individuals there were per household, or how large the household was. Apparently no one who commissions polls cares about this.

    One unfortunate side-effect of the IBC approach is that we now have a reinforced double standard on how atrocities are reported. The scale of enemy atrocities is generally given with large dramatic estimates and we almost never know the basis for them. Deaths caused by US policies, on the other hand, have to be actual counts (or at least based on what is purported to be actual counts). This is going to bias discussion in favor of so-called humanitarian interventions. So that’s why the number debate is important. I suspect you agree.

  8. #8 kevin
    March 22, 2007

    Donald: The new poll could suggest many things. A finding about being “harmed by violence” necessarily entails nothing for consequent mortality rates. 100% of members of my household have been harmed by violence in the past 5 years and none have been mortal wounds. It’s a pretty vague question.

    And of course I understand the cash value of polls like this. The point is supposed to be that the invasion caused all of these deaths. Of course this is made in complete ignorance of the fact that the majority of them are due to the acts of radical Islamist foreign infiltrators or displaced Baathist supporters willing to wreck Iraq’s economy and commit mass murder as a form of political discourse.

    Why is the Lancet methodology even worthy of replication? How is it not easier and more accurate to get the records of death certificates being issued at the local level? Even Lancet used those as a check on their polling.

    And yes, the war could have been handled much better from data available at the time of the invasion. I don’t think that the war was handled incompetently is any sign that the effort itself was unjust. The Baath regime didn’t have a swell record on excess mortality; why is the fact that the Baath regime was a mass-murdering, police state irrelevant to whether or not invading was just?

  9. #9 Donald Johnson
    March 23, 2007

    In context, Kevin, when the pollster asks how many people have been “harmed”, it probably means serious injury or death. Much of the poll is about the security situation, and what sorts of violence have the responders seen. One question was whether the respondent had seen various forms of violence in their neighborhood and the highest response (44 percent) was for “unnecessary coalition violence against civilians”. Asked to say what they feared the most, car bombs and suicide attacks were at the top of the list (38 percent). Unnecessary violence from coalition forces was second at 16 percent and various forms of Iraqi-on-Iraqi violence picked up the rest. But to me, the fact that 16 percent fear coalition forces the most (Shiites almost as much as Sunnis–the poll is broken down by sect) and that 44 percent have seen unnecessary violence by coalition forces suggests to me that we aren’t hearing about a lot that goes on over there.

    There are some statistics you could use for the prowar side too–53 percent of Shiites and 59 percent of Kurds but only 7 percent of Sunnis think the country is better off than before the war. But I think the picture painted of violence makes it seem more widespread than one would guess from the usual media statistics and in particular, US violence seems more widespread and a bigger cause of concern for Iraqis than you would ever dream if you went by the US press.

  10. #10 joshd
    March 23, 2007

    Donald you seem to be tweaking things in your direction. First you find the one report that uses the phrase “close relative” for one of the questions while the poll just said ‘relative’, which would mean anyone related to them, uncles, cousins etc. (which is probably around 50+ people).

    Then you interpret harmed to mean “serious injury or death”. Why “serious”? The question casts the widest possible net over injuries, not just “serious” ones. If anything it would possibly include some violent events that impacted household members, but which didn’t necessarily cause injuries at all, like kidnappings, which the poll suggests are rampant in many areas.

    I don’t think any firm casualty numbers can be derived from this poll, but this does look like another piece of evidence suggesting L2 is an overestimate. Roberts has said that his study suggests “one in seven” (14%) of households have had someone killed. And this poll finds only 3% more when the net is widened to include anyone “harmed”. And this is after seven more months of the most intense violence since 2003, more intense than most of the period covered by L2. If L2 was right, widening the net to include any injuries should produce a far, far higher number, not 3% higher. It should probably be 3% higher just for deaths alone given the extra seven months of extreme violence.

    (Shiites almost as much as Sunnis–the poll is broken down by sect)

    Too bad L2 isn’t similarly broken down. Note the huge disparity of answers depending on the sect of the respondent. If a poll is oversampling Sunni Arabs or undersampling Kurds for example, you’re likely to get overestimates of violence related indicators. Yet we have no idea, like most everything else, of the sects of the respondents (or the surveyors) in L2. We do know though that L2 says the surveyors were fluent in “English and Arabic”, while Kurdish is not mentioned.

  11. #11 Kevin Donoghue
    March 23, 2007

    Question 35 actually reads:

    “Have you or an immediate family member – by which I mean someone living in this household – been physically harmed by the violence that is occurring in the country at this time?”

    I don’t see how this can either support or undermine L2 since the period during which the violence took place is unclear and so is the actual harm done. The two polls do give the impression that Iraq is a far worse mess than suggested by official figures and suchlike. But then it must take quite an effort to square the official line with the accounts of reporters on the scene. Anyone who can do that should easily be able to explain these polls away as well.

  12. #12 Donald Johnson
    March 23, 2007

    Try not to be so suspicious, Josh. I’m as biased as the next guy, but I used “close relative” because it happened that was the term used in the report that I first read.

    BTW, there are two polls here–the one that I mistakenly thought said “close relative” and this other one I’m citing in my more recent posts. The second one oversampled Sunnis, though it’s not clear to me if they took this into account in calculating the overall results. It doesn’t make much difference for my purposes–the number of Sunnis who said someone in their household was “physically harmed” was 22 percent, for Shiites it was 17 percent and for Kurds it was 7 percent. If I use the 60, 20, 20 percent composition I usually see claimed for the Shiite, Sunni, Kurd population I get 16 percent physically harmed. The poll says 17 percent overall.

    Since neither of us are statistics gurus, I don’t mind embarrassing myself with back of the envelope calculations in front of you. (The rest of you can avert your eyes.) I’ll assume 7 people per household–that’s what Lancet 2 found. (I thought it was 6 before.) Assume 24 million people (allowing for some refugees having gone). That’s 3.4 million households. Multiply by .16 and you get 550,000 households with at least 1 casualty.

    I’m going to guess the number of casualties per harmed household is between 1 and 2, so that’s 550,000 to 1.1 million casualties. Then assume that the fraction which are deaths is 1/4 to 1/3. Then you get a range of deaths from 140,000 to 370,000. Lower than L2, though the top of my range almost touches the bottom of theirs. Higher than what IBC would allow, though the bottom number might be close to acceptable if it includes insurgent dead. (Something it’d be interesting to have counted from press accounts, btw. I suspect you’d find it difficult to do.)

    One could quarrel with this, of course, but it seems like a reasonable estimate to make based on these numbers.

  13. #13 Donald Johnson
    March 23, 2007

    “These numbers” refers to the poll results, of course–the other numbers are my guesses about household size and casualties per house and deaths /casualties and so forth.

  14. #14 Donald Johnson
    March 23, 2007

    Kevin D, it does suggest a minimum number of casualties in the mid hundreds of thousands and possibly more, or anyway, I can’t see how one could get a number for “physically harmed” which isn’t at least that much. It doesn’t tell us how many of the physically harmed were killed or how many were harmed per household–I supplied my guesses above, to illustrate the point that the poll suggests a death toll well into the hundreds of thousands range.

  15. #15 Kevin Donoghue
    March 23, 2007

    Donald, what are you taking the phrase “at this time” to mean, in the question I quoted above? You seem to be extending it back to 2003 or thereabouts.

  16. #16 joshd
    March 23, 2007

    “But then it must take quite an effort to square the official line with the accounts of reporters on the scene. Anyone who can do that should easily be able to explain these polls away as well.”

    I’m not sure what “official line” or “official figures” you’re fantasizing about, or how you’re able to compare it to this poll when you’ve just finished claiming the poll is all but meaningless in this regard, but “official figures” I’ve seen suggest a very big mess with about 3,000 being killed each month, lots more injured, kidnapped etc., all kinds of attacks up, up, up, and a situation that has been rapidly deteriorating for some time.

    As to “accounts of reporters on the scene”, I doubt you read many of them, and you should recall that you’re speaking to someone who reads them much more closely than you do, and has for the past four years.

    You should try reading some of them, particularly not just anecdotes you cherry pick from blogs, but the ones that have actually explored in detail “official figures” and compared them against local records, which are more relevant, such as:

    BORZOU DARAGAHI: Well, we think — the Los Angeles Times thinks these numbers are too large, depending on the extensive research we’ve done. Earlier this year, around June, the report was published at least in June, but the reporting was done over weeks earlier. We went to morgues, cemeteries, hospitals, health officials, and we gathered as many statistics as we could on the actual dead bodies, and the number we came up with around June was about at least 50,000.

    And that kind of jibed with some of the news report that were out there, the accumulation of news reports, in terms of the numbers kill. The U.N. says that there’s about 3,000 a month being killed; that also fits in with our numbers and with morgue numbers. This number of 600,000 or more killed since the beginning of the war, it’s way off our charts.

  17. #17 Donald Johnson
    March 23, 2007

    To Kevin D— I see your point. I was assuming that violence “at this time” meant all the violence since the March 2003 invasion. Maybe some of the respondents thought in terms of a shorter period.

    Sticking to what I think one can conclude from the poll, allowing for sampling error (which is probably not too large for this big a poll), there are probably several hundred thousand households which have suffered at least one casualty during whatever time period the respondents had in mind.

  18. #18 Kevin Donoghue
    March 23, 2007


    I posted my last question before I saw your response to my earlier comment. I don’t think we disagree. When I say these figures neither support nor undermine L2 what I mean is that they could be consistent with a wide range of deaths. If the true number of violent deaths is inside the 426,000 to 794,000 confidence interval then L2 is vindicated and if not then there is something badly wrong with it. These polls could be consistent with either scenario, depending on your assumptions. I agree that they broadly support the impression that the death toll is in the hundreds of thousands. But Ragout, for example, would say that we had good reason to believe that before L2 came out.

  19. #19 Kevin Donoghue
    March 24, 2007

    Not for the first time, Josh quotes Borzou Daragahi. The report Daragahi refers to is probably by Louise Roug. Josh doesn’t often quote Roug, who says inconvenient things like this: “Obtaining accurate numbers from the Health Ministry or the 18 major hospitals serving Baghdad proved difficult, because officials at all tiers of government routinely inflate or deflate numbers to suit political purposes.” She makes it very clear that the figures she has obtained are at best a lower bound (and even that is only true so long as it suits the MoH to produce low numbers rather than high ones).

    Like Roug, experienced reporters like Patrick Cockburn, John Simpson and many others make it perfectly clear that the figures furnished by officials (yes Josh that’s what I mean by official figures) are only a partial count and that in general there is much more violence taking place than the media coverage would suggest to the unwary. Indeed Daragahi may be aware of this: “about at least 50,000″ – that seems to be an acknowledgement that we are talking about a minimum here.

    It must take a real effort to read reports from Iraq and miss these warnings. Perhaps the trick is to look at them so “closely” that you can’t see the wood for the trees: “only that which can be counted is permitted to count.” Reports of fighting or explosions which don’t mention a specific number of deaths add nothing to the total. That’s fair enough as long as you remember the limitations of the method.

    In short, the statement n > b doesn’t tell us what n is, it only tells us what it isn’t.

  20. #20 joshd
    March 24, 2007

    First it’s that I should read them, then when I read them more closely than you but still don’t agree with you, I read them too closely. Ok.

    That you claim I’m ignoring “warnings” about underreporting is stupid because the “warnings” don’t say anything other than that there’s a lot of unreported stuff, which I already assume and have since the beginning.

    Next you assume Roug would have some different view than Daragahi (who I believe was involved in the same work, and certainly knows the basis of its statements and analysis better then you or I do).

    And it is true that n>b can’t give us an exact figure for n, but that does not mean ANY value for n is reasonable next to the evidence and manner in which b was obtained. What Daragahi is saying is that the comprehensive way they went about obtaining b (“We went to morgues, cemeteries, hospitals, health officials” ..etc.), and checking with as many different sources as they could find, renders some values for n (like 600,000) quite absurd and extreme.

    But then you’ll have no problem disregarding this “warning” from “reporters on the ground”. You’ll also have no problem disregarding that LR claimed his study would be confirmed if someone would only go investigate cemetaries. Daragahi is telling you they did, and is telling you that the Lancet numbers don’t look anything like reality when you do.

    On the comment about “inflating or deflating” numbers for political reasons, what this tells me is that some Iraqi officials in Baghdad, according to Roug, have tweaked numbers up or down for various reasons or at various times. Maybe so. It’s not clear in the reports exactly what she bases this claim on, but far from being “inconvenient”, it can be surmised that she can say this because she and her colleagues had some means to check the veracity of the numbers she got from some sources against those she got from other sources, which was the pont of their efforts. And it tells me that when they looked at multiple sources for the numbers they could see that some accounts didn’t match up with others.

    Given what Daragahi says (Roug has never commented on Lancet numbers to my knowledge), quite obviously the discrepencies they found in different accounts, and which presumably informed such a comment, were not anything remotely like Lancet-size discrepencies. If the discrepencies were huge rather than small, I think they would have written on this, and Daragahi would hardly say what he does (unless he’s also in on the big cover up).

    You choose to take Roug’s comment to mean “they’re hiding hundreds of thousands of deaths”, but that doesn’t seem to be anything like what they mean, and Daragahi is telling you that’s not what it means, and that this interpretation is not reasonable to them.

    Perhaps more importantly, what it tells me is that independent investigators have at least some ways to go about checking the veracity of “official numbers”, by checking many different sources and accounts, and have some means to find fault with claims made and the persons making them, and that’s what they were trying to do.

    Quite the opposite of this would be the numbers reported by the Lancet teams, which neither Louise Roug nor any other independent party have any means to check at all, and which are apparently therefore to be taken as much more reliable.

  21. #21 Donald Johnson
    March 24, 2007

    Josh, it’s because of arguments like yours in the 11:27 comment that I’m not completely in the Lancet2 camp. But it’s not wholly convincing either–there are reporters like Cockburn and Fisk and a couple of others who either support outright the hundreds of thousands of deaths (Rory Stewart did so most recently in a side-comment in his NYT guest column) or who say that the violence is widespread and largely unreported. Maybe the LA Times could talk to people who were fearless and unafraid to tell the truth about the death toll, or maybe they talked to people who thought it was a dangerous subject to be truthful about. Or maybe people in these various places simply don’t keep accurate counts–there are also reports, you know, of the chaotic state of the Iraqi health care system. If methodology is a problem to be discussed with L2, it seems at least as much a problem with the LA Times approach.

    That’s why polls and surveys ought to be an invaluable method of determining whether or not the official statistics are correct, because you bypass officials of all types and go directly to ordinary people (who, of course, might also lie). You and others think the Lancet papers are badly flawed–fine, but now we’ve got another poll which can’t give us an accurate count, but at the very least strongly supports a figure of several hundred thousand households which have suffered at least one casualty. You ought to wonder whether this means the IBC number is much too low, even if the truth isn’t necessarily as high as the Lancet2 confidence interval.

    And what I wonder is why these news agencies couldn’t just have ordered up a poll explicitly intended to determine the number of dead and wounded, given that this controversy over death tolls has made it into the mainstream press to some extent. If they are really serious professional organizations devoted to uncovering the truth and if it occurred to them to commission these polls on Iraqi attitudes, why on earth wouldn’t they try to determine mortality rates?

  22. #22 Kevin Donoghue
    March 25, 2007

    Josh wrote: What Daragahi is saying is that the comprehensive way they went about obtaining [a lower bound] (“We went to morgues, cemeteries, hospitals, health officials” ..etc.), and checking with as many different sources as they could find, renders some values for n (like 600,000) quite absurd and extreme.

    I’ve re-read the report by Louise Roug and Doug Smith (Raheem Salman is credited in the footnote, Daragahi doesn’t get a mention) and there is nothing in it to support the claim that their investigation established any upper bound whatever, nor even that they hoped to do so.

    But then you’ll have no problem disregarding this “warning” from “reporters on the ground”.

    I’ll certainly have no problem whatever ignoring what a reporter says in the course of a radio interview about his colleagues’ work. Likewise, if somebody at JHU claims that the Iraq study established results which the authors themselves do not assert, rest assured I won’t expect you to regard that as persuasive – unless an argument goes with it and the argument is sound.

    But let’s suppose that every LA Times reporter who has ever been to Baghdad is incredulous about the 600,000 figure. What would that tell us? Not a damn thing, until they tell us why they are incredulous. I read reports to find out what the reporters actually see happening. What they see happening is more violence than they can cover, corpses eaten by dogs and various other horrors which are entirely compatible with a very high death toll indeed. But if Daragahi ever tells us how he gets from “I’m incredulous” to “I therefore conclude that X is an upper bound for the death toll”, believe me I’ll study his argument.

    Incidentally, Roug and Smith don’t mention checking cemetaries at all in that particular report. But Roug does mention them in another report: “In the Sunni cemeteries serving Baghdad, a city of 5 million people, demand for tombs is so high that people are buried between old graves or at the edges of the burial grounds.”

    That’s not the sort of test Roberts suggested. But it certainly doesn’t conflict with his reading of the situation. Bear in mind that many people die of natural causes, even in Iraq. Mortality rates would have to rise quite a bit to have a noticeable impact on demand for tombs.

    You choose to take Roug’s comment to mean “they’re hiding hundreds of thousands of deaths”….

    Misrepresentations like this don’t enhance your claim to be a careful reader, of reports from Iraq or anything else.

  23. #23 joshd
    March 26, 2007

    Kevin, i think you’re wasting my time. I’d just say that the Roug quote about Sunni graveyards in Baghdad “doesn’t conflict” with the figures Roug is giving you for Baghdad in the very same report.

  24. #24 Kevin Donoghue
    March 26, 2007

    Likewise, the figures given by Roug do not conflict in any way with the findings of Burnham et al., since she makes it abundantly clear that the figures are an undercount; they “do not present the full picture of the violence in the capital.” A careful reader will also notice words like “at least” and “conservative” at various points in her report.

  25. #25 MikeB
    March 26, 2007

    Tim – you’ve probably seen this http://news.bbc.co.uk/1/hi/uk_politics/6495753.stm already, and are typing up a new post on the Lancet report right now. But if you haven’t, then I think that one or two people here might be interested…