The Washington Post continues its sorry record on the Lancet study with this piece by Sarah Sewall:
The Lancet study relied on a door-to-door survey of Iraqi households in 33 neighborhoods. The surveyors asked for details of deaths in the months before and after the invasion and found a significantly higher death rate after. But the approach was flawed. War is not like a pandemic; it comes in pockets. And the study itself qualified its conclusions, acknowledging that the figure could range enormously between 8,000 and 194,000.
Sewall fails to present any argument, any argument at all, why war "comes in pockets" means that random sampling is flawed. The only thing I can guess she is driving at is the known problem with cluster sampling when the deaths are also clustered. But in that case you are likely to underestimate the number of deaths. And she fails to mention that values in the middle of the confidence interval (like 100,00, say) are much more likely than values at the ends. What is really disappointing here is that Sewall is a lecturer at Harvard and surely has access to the expertise that would let her write a more informed piece if she took advantage of it.
This short piece by Bill Marsh in the New York Times isn't much better. Marsh writes:
A 2004 study estimated that 100,000 Iraqis had died but figures from a group called Iraq Body Count are more often cited because they are based on media reports, not projections. Since many deaths are unreported, that tally is undoubtedly low.
It's good that he points out that the IBC number is undoubtedly low, but the reason he gives for people citing it rather than the Lancet estimate doesn't make sense. Figures for deaths in the Congo or Rwanda or Bosnia that get cited aren't based on media reports but on estimates like the Lancet study. What on earth could be the important difference between Iraq and those other places that leads to a different standard for reporting war-related deaths? What could it be?
But the really bad thing about Marsh's piece is the grossly misleading graph. He presents a graph showing how "the President's number compares with estimated deaths in recent conflicts". But why compare a number that is "undoubtedly low" with estimated deaths in other conflicts? Surely you should compare estimated deaths (ie the Lancet number) with estimated deaths? Or if you really wanted to compare tallied deaths in Iraq, you should compare with tallied deaths in the other conflicts. For example, the graph shows the estimated 200,000 deaths in Bosnia and not the 102,000 Bosnian deaths that have been tallied..
Furthermore, the graph gives an estimate of 180,000 deaths in Iraq Internal Violence 1998-1991 and I just can't find this figure in the source Marsh gives. His source has 150,000 dead Kurds in the period 1961--1993 and 25,000 dead Shia from 1979 to 1998, so things just don't add up. In any case, the basis for these estimates is much less solid than the Lancet number, but oddly enough, you don't see the Lancet skeptics dismissing these numbers.
Finally Andrew Cockburn in the LA Times, in a rare event for a US newspaper, does a good job of covering Iraqi deaths:
Columbia professor Richard Garfield, one of the team members and study authors, told me this week that by now the number of "excess deaths" in Iraq "couldn't possibly be less than 150,000." But, he added, "there's no reason to be guessing. We ought to know better."
Andrew Cockburn is brother of Patrick (Iraq correspondant for the UK Independent) and Alexander (Counterpunch) and son of the legendary Claude Cockburn. He also co-wrote with Patrick a very good book on Saddam's Iraq. So, not unexpected that he would get it right.
'In fact, the amazingly dedicated and courageous Iraqi doctors who actually gathered the data visited 33 "clusters" selected on an entirely random basis. '
Is that what you consider to be a 'good job'?
Cockburn either hasn't read the study, or hasn't understood it, if he thinks the clusters were 'entirely random'.
From the study:
The paired Governorates were:
Basrah and Missan, Dhi Qar and Qadisiyah, Najaf and
Karbala, Salah ad Din and Tamin, Arbil and
Sulaymaniya, and Dehuk and Ninawa.
That list is, as acknowledged in the study, based on travel times, not random selection.
Do you agree that Cockburn is wrong on this?
soru
No, you are. The clusters were randomly chosen and each household was equally likely to be chosen. You might as well complain that by using governorates as the first stage in allocating clusters they were not random.
I've e-mailed Sarah; let's see if I'll get a response.
I also emailed Sarah.
Have you emailed the NYT? I copied your article (giving credit) and added a paragraph of my own and sent it to the Public Editor (i.e., the complaints department) at the NYT.
What on earth could be the important difference between Iraq and those other places that leads to a different standard for reporting war-related deaths? What could it be?
Oh my. The methodology for Congo, Rwanda, etc was not the same as it was in Iraq. The wars in Africa were civil wars where entire villages would be massacred and the violence was a lot more evenly spread. In Iraq, most of the damage was done with bombings during the initial invasion, or isolated engagements such as in Fallujah. The confidence intervals were also much narrower on those other studies vs. the Iraq one.
Don't let those huge differences taint your picture though... ;)
Yes, 100,000 is the "most likely" figure within the range cited, but there is no guarantee that it is correct. The fact remains that it COULD be any of the figures between 8,000 and 194,000. You know, assuming that the underlying data is precise, which the study itself admits that it is not. When you have a narrower confidence interval, you know that it couldn't be outside that and that much. Clinging onto the "most likely" rhetoric pretends that therefore it IS that number. Also, I would suggest producing a distribution graph of the data to see how it looks like. Does it look like a normal distribution, or is it a lot flatter? That matters quite a bit.
You all know my position on the "randomness" of the sample.
Tim,
This post shows the real harm you're doing by shamelessly exaggerating the quality of the Lancet study. You don't mention Sewall's main point, so let me. Sewall says (correctly) that the existing figures on Iraqi civilian deaths are pretty poor, and that better ones should be collected, especially by the military. Her argument is that the military should study these figures in order to learn how to better avoid killing civilians. In other words, she points out that the importance of these figures goes far beyond political point-scoring, and that it's a big problem that nobody is collecting them.
You claim to support collecting better figures. But I think that your exaggeration of the reliability of the Lancet study will mislead readers into believing that better data isn't needed. You continually claim that it's no big deal if the Lancet study underestimates deaths. And you're right, if the only goal is the criticize the war. But if you want to know the truth, a biased estimate is a problem, whichever direction the bias is. And now you bash Sewall, whose whole essay is devoted to discussing the need for better figures.
The Cockburn piece is OK, but falls down in suggesting that the only methods of collecting mortality data is by reading newspapers or conducting surveys. As Sewall points out, the military could be doing civilian body counts. Also, Iraqi hospitals ought to be counting every death and recording the cause. That's the way "vital statistics" are collected in most countries. Iraq appears to have a more or less functioning health care system, and press reports say that such figures are in fact being collected. They just aren't being released, because the US and Iraqi governments want them kept secret.
I emailed Sarah Sewall as well, pointing out the wild and woolly way she treated the Lancet study, like so many in the media, as a kind of 'over' to be dumbly discounted and discarded in a sentence, before moving on to the IBC.
I got a response, but I don't feel at liberty to share it without her permission.
You don't mention Sewall's main point, so let me.
Well, Sewall doesn't do the Lancet study justice either, so let us.
But I think that your exaggeration of the reliability of the Lancet study will mislead readers into believing that better data isn't needed.
You're entitled to think what you like, but you're not entitled to base the source of your thoughts from this site. As opposed to, say, your rear end.
Contrariwise, the typical three-sentence dismissal of the Lancet study, usually accompanied by signs of basic statistical innumeracy, will definitely mislead readers into thinking that the IBC's number is all that needs to be examined.
Of course, that was Bush's point. The soft bigotry of low estimations, to re-coin a phrase.
No, you are. The clusters were randomly chosen and each household was equally likely to be chosen.
Which does not make it 'entirely random'. It means that in your judgement, the non-random elements of the process were performed in a way that would not significantly influence the result (e.g. presumably you would agree that the 2003 Ministry of Health population figures are accurate enough to be used for these purposes).
Whether the result (that all 5 most violent provinces were sampled and 4 of the 5 least violent provinces were not sampled) is a result of random chance, or of mistakes in the handling of the non-random factors, really doesn't change anything.
The survey was a reasonable effort for it's time, given the effort available. But if your goal is understanding of the situation in Iraq, as oppsoed to polemic, I think more emphasis should be placed on the results of all-province surveys, even if none so far ask exactly the same questions as the lancet study.
soru
I think the IBC data is getting a really rotten rep here at this blog, perhaps for obvious reasons.
If you look at their newly released report, you'll see that out of the 24,865 deaths recorded in their report, the primary source of the deaths were:
Mortuaries - 36%
Medics - 19%
Iraqi officials - 18%
Eyewitnesses - 15%
Police - 14%
Relatives - 11%
Coalition - 10%
Journalists - 8%
The clusters between pairs were not randomly distributed, according to the definition of a random sample. That is not a necessary thing to do, but if you don't do that, you have to distribute the clusters via proportionality, which the study also does not do......
I think the IBC data is getting a really rotten rep here at this blog, perhaps for obvious reasons.
Again, your thoughts are entirely your own business. So, playing that game, I think you're imputing such things for obvious reasons of your own.
Sarah Sewall takes her campaign to reduce civialian casualties to the front page of the Washington Post today.
She gets the last word in the article too: "When it comes to the extent to which they are minimizing civilian harm, the question becomes: How do you know?" Sewall said.
But there's no reason to listen to her, after Tim's explained why she has no credibility. She's just a disappointing, poor reporter, carrying on a sorry tradition.
The IBC data implies that the pairing process, on average, moved clusters to less violent Governorates. In other words, it destroys Seixon's argument. I could have left it that, but fairness compelled me to mention that the data is not good enough to draw that conclusion.
Ragout, reducing civilian casualties is a good cause, but that does not give Sewall a license to make erroneous statements about the Lancet study. Statements which you yourself know to be mistaken.
I think Sewall has written quite a good paragraph summarizing the Lancet study, and that it is perfectly accurate. You haven't identified any flaws at all in her summary -- you've just asked for more argument, and more explanations of a confidence interval. Considering the context (an argument for more data collection by the military) Sewall was correct not to discuss the Lancet article at greater length.
Your suggested ammendments, on the other hand, are false. In particular, "values in the middle of the confidence interval (like 100,000, say) are much more likely than values at the ends." In fact, Roberts believes that 100,000 is less likely than, say, 150,000.
Admittedly, one could make a case for your summary, which I guess would be that you totally discount the results that include Falluja. Roberts own one paragraph summary isn't much like either yours or Sewall's. Roberts gives much greater emphasis to the results that include Falluja.
It's hard to summarize a statistical study in a single paragraph, especially one that is as much of a mess as the Lancet study. I much prefer Sewall's summary to yours. It's factually accurate, and right in spirit as well: the Lancet study is fairly weak evidence for anything.
This American Life did a nice piece on this.
Check out episode 300 at www.thislife.org
The IBC data implies that the pairing process, on average, moved clusters to less violent Governorates. In other words, it destroys Seixon's argument. I could have left it that, but fairness compelled me to mention that the data is not good enough to draw that conclusion.
4 out of 6 moved to more violent ones, 2 out of 6 moved to less violent ones. The net effect, as I have detailed roughly on another thread, is that this would overestimate the true numbers. Thus, I cannot see how your statement above holds.
The data is good enough to conclude that 3 of the 6 pairings were off the wall, with another 2 being quite likely to be dissimilar, with the last being ambiguous. I will be trying to collect the ILCS data from the UNDP to get a less biased data set which perhaps even Mr. Lambert might suscribe to.