Tim Lambert (deltoidblog AT gmail.com) is a computer scientist at the University of New South Wales.
If you followed the debate over the first Lancet study you know that it featured numerous attacks on the study from folks who manifestly did not have a clue about statistics. The new study gives us much more of the same.
First up is President Bush who said:
"I don't consider it a credible report. Neither does Gen. (George) Casey, and neither do Iraqi officials."
and (after a lot of waffling)
"The methodology is pretty well discredited"
Cluster sampling is discredited? Not according to any statistics textbook in the world.
Next we have Blue Crab Boulevard:
This would be almost 400 (article says 500 - even higher than my back of the envelope calcs) people PER DAY. Over the normal death rate? This is utter and complete crap. Period. No thinking person can possibly believe these numbers. When have you EVER heard of that many people dying in one day in Iraq, much less every single day since the war began. For heaven's sake, think, people.
Yes, the media hasn't reported that many deaths in Iraq. But they don't have reporters every or even most places, so it's certain that most deaths go unreported in the media.
Jay Redding offers
If the death toll were really that high, there would be massive refugee outflows from Iraq. We're seeing some of that, but nowhere near as much as those figures would suggest. Furthermore, the same group predicted 100,000 dead in the first year of the war (releasing their figures near the 2004 elections, again for political gain) -- now they want to argue that an addition 550,000 have died in the subsequent two years? That argument doesn't even pass the smell test.
Well, there has been massive refugee outflow. And they do argue that the death rate has gone way up since the first study. But this increase seems well supported by all other indicators of violence in Iraq.
The new study released is more than TEN TIMES higher than other liberal estimates!
Other liberal estimates being the Iraq Body Count. The IBC counts deaths reported in the media. Not all or even most deaths are reported in the media. Gateway Pundit also posted Gilbert Burnham's email presumably to encourage his readers to send abusive emails. Gateway Pundit is a scumbag.
Moving right along we have Rick Moran, who cites Kaplan's fallacy and then writes:
As you can see from the above New York Times excerpt, these purveyors of wildly exaggerated mortality have tried the same technique this time around as well: they have "a margin of error that ranged from 426,369 to 793,663 deaths."
Kaplan's argument was to point to the bottom end of the confidence interval (8,000 in the first study) and say that 8,000 was a small number. In the new study the bottom end is 426,000, which is an enormous number of deaths. Does Moran think that 426,000 is an acceptable number of deaths?
Finally it's a welcome relief to come across Sean Bannion. Bannion is completely wrong, but at least he knows enough statistics to be a danger to himself:
The study relies on a stratified, multi-stage cluster sample design. Clustering of households in the sample results in variances that are smaller than you would get with a simple and random sample design. (And if you've ever taken so much as an introductory statistics course you know the word "random" is a mantra to be repeated over and over...)
No, clustering results in larger variances than simple random sampling.
So a failure to take account of this "design effect" when using your basic statistical tests in the analysis will make accurate testing difficult, because standard tests assume independence among observations and a simple, random sample design.
So you should use tests that take account the design effect. Which they did:
The SE for mortality rates were calculated with robust variance estimation that took into account the correlation between rates of death within the same cluster over time.14 The log-linear regression model assumed that the variation in mortality rates across clusters is proportional to the average mortality rate; to assess the effect of this assumption we also obtained non-parametric CIs by use of bootstrapping.13,15 As an additional sensitivity analysis, we assessed the effect of differences across clusters by extending models to allow the baseline mortality rate to vary by cluster.
And if you don't understand that paragraph you probably shouldn't be criticising the study's methodology.
Back to Bannion:
I can tell you from personal experience that polling and sampling in Iraq are notoriously difficult. ... many Iraqis were suspicious of being asked even innocuous questions for two reasons. One, they didn't believe you (the pollster) really cared about what they thought and they would often say anything, quickly, to make you go away. Which brings me to my second reason. Being seen talking to a Westerner, or a local working for a Western firm, was often prima facie proof to the jihadists that you were a "collaborator."
The study employed Iraqi doctors to do the surveying.
The second reason is because Arab culture is in general - oh, how to say this gently - fabulistic. When we would get reports from locals there were not 5 jihadis around the corner, there were "dozens." There weren't 5 or 6 IEDs on the road to the airport, there were "hundreds." You get the picture. This deception even included on some occasions generating bogus "proof" to prove erroneous or outright false assertions. We quickly learned not to trust one-on-one reports from any Arab unless they were backed up be multiple sources and hopefully with independent (read that "Westerner" or technological) means.
The study verified that the deaths were real by checking death certificates:
Survey teams asked for death certificates in 545 (87%) reported deaths and these were present in 501 cases. The pattern of deaths in households without death certificates was no different from those with certificates.
Comments
They have a companion paper here to counter most of the misinformation.
In it they cover the topic that I brought up a few posts ago that the UN had released a study from ministry of health statistics which showed violent civilian mortality peaking in the summer at little over 100 a day. We should note this study didn't include kurdistan and Al-Anbar reported no deaths. I raised the question of MoH statistics accurately got all deaths that recieved death certificates. They argue:
The Ministry of Health in Iraq has published some numbers from time to time, but these are generally considered to be unreliable. The registration of deaths in Iraq has been an organized process for many years. Death certificates have traditionally been obtained for the deaths of all adults and older children. Death certificates are required for insurance claims, compensation, payment of benefits, and for burial. Cemeteries do not take bodies for burial without certificates. If deaths occurred outside of hospital, the bodies would be transported to the general hospital for the certificate to be issued. If there were doubts about the cause of death, a post-mortem examination would be carried out before issuing a certificate. Copies of the death certificates would go to the national offices managing vital registration. This process has continued through the current conflict, with death certificates being required for burial, and with information from certificates being duly recorded. However, the tabulation of data from registration of deaths in Iraq has suffered from the chaos of the current conflict. Beyond this, there is also a suspicion that records of death, particularly related to violent deaths, is being manipulated and only partially being released for various political reasons. Even with the death certificate system, only about one-third of deaths were captured by the government's surveillance system in the years before the current war, according to informed sources in Iraq. At a death rate of 5/1,000/year, in a population of 24 million, the government should have reported 120,000 deaths annually. In 2002, the government documented less than 40,000 from all sources. The ministry's numbers are not likely to be more complete or accurate today."
If thier 1/3 of all deaths are reported then the two would be in the same ballpark. - Andy
Posted by: Andy Barenberg | October 11, 2006 3:17 PM
One thing you're going to find (you may have already) is something I saw on a forum about science and evolution. This was the claim that _ had been "debunked" and was defended by changing the plain meaning of "debunked" to "disputed". So if I claim that Tim Lambert is not at a university in Australia, I have "debunked" the claim that he is there, even though my claim is obviously false. I've disputed that he is, but claim I've debunked that he is.
This seems to work quite well with various types of deniers, so expect it.
Posted by: QrazyQat | October 11, 2006 3:34 PM
This seems to work quite well with various types of deniers, so expect it.
I've expected this for some time. It's a standard denialist tactic: "I've given you any ol' answer, therefore I've debunked your claim. Never mind all that fancy fact stuff you spew."
Best,
D
Posted by: Dano | October 11, 2006 4:48 PM
So, will Joe Barton get his team of crack statisticians to investigate the scientific basis for Bush's proclamation that Cluster analysis is a discredited methodology?
Patiently waiting... Mitch
Posted by: Mitch | October 11, 2006 5:42 PM
Cluster analysis?
I believe Bush must have been thinking of "cluster fucking" (what he and others have engaged in in post invasion-Iraq) when he said "the methodology has been pretty well discredited".
Posted by: JB | October 11, 2006 5:54 PM
I think bush was referring to cluster bombing.
Posted by: Alex | October 11, 2006 5:58 PM
Sorry children, but the last study was debunked. Unfortunately once your credibility is shattered, coming back just in time for the elections with another headline grabber that just coincidentally happens to fit into their loudly proclaimed anti-war agenda may raise suspicions. Numbers, no matter how you cook them up, are excellent propaganda, and for the same reason numbers are more often than not the hostage of an agenda. It hardly bears noting the curious capacity of advocacy groups to produce findings that strangely enough support (and support extremely) the very position they just happened to advocate, as if the result was somehow completely fortuitous. If you start out from an staunch anti-war position than exactly how hard has it ever been to fudge the results, to overlook problems, to circumvent reasonable objections, and simply to let the agenda drive the conclusions derived? Lies, damn lies, and statistics.
Posted by: ian | October 11, 2006 7:05 PM
You know, Ians comment sounds very like the anti-evolutionists comments I am used to reading. It says nothing substantial, just states some broad factoids, (Yes, we know advocacy groups happen to produce information that agrees with their aim) without actually considering the specific case in hand.
So what exactly is wrong with the current study, Ian?
(And I have to get in the obligatory snark about WMD's, and how it seems they dont exist after all)
Posted by: guthrie | October 11, 2006 7:23 PM
Ian, do you have a point? What's wrong with the study?
Posted by: Brian S. | October 11, 2006 7:26 PM
Well done, 'ian', for your substantive critique.
I'm most interested in the extent to which the new survey validates the distribution of the 2004 survey, given the amount of BS poured upon it at the time. Alas, it'll lead to the old 'damned if you do, damned if you don't': had the new study come up with a different number, the denialists would be crowing about its inaccuracy. Now that the two essentially agree, the denialists are pulling out the old canards to claim that the new study is just as flawed.
Perhaps we should offer them a new statistical model, based upon a 'maximum death ickiness': that is, the total number of dead Iraqis that our denialists are prepared to concede. Let's at least systematise their arguments.
Posted by: ahem | October 11, 2006 7:38 PM
Number of empirical field studies the denialists have produced obtaining their own number and/or debunking Lancet studies: zero.
Slope of rate of increase of expected number of studies: zero.
Number of bytes of bandwidth expenditure by denialists "debunking" Lancet studies: bigbig number
Slope of rate of increase of bandwidth expenditure "debunking" Lancet studies as Murrican Election Day approaches: rising asymptotically.
This public service announcement brought to you by: any average thinking person on the street and Michael Moore (TM).
Best,
D
Posted by: Dano | October 11, 2006 7:57 PM
If there were something wrong with the Lancet study, I somehow doubt Ian would be capable of saying what it might be.
Paroting the "Lies, damned lies and statistics" quote reveals that he simply does not understand the nature and purpose of statistics.
Those who know something about science appreciate that statistics play a very important role in the scientifc process and can sometimes reveal information that can not be obtained in any other way.
Posted by: JB | October 11, 2006 8:03 PM
So we meet again.
I challenge you to a duel, Sir Lambert, one that I am sure you cannot refuse.
Posted by: Seixon | October 11, 2006 8:04 PM
Wow. It only took 5 comments for someone to use the disputed means debunked claim I mentioned in my comment.
Posted by: QrazyQat | October 11, 2006 8:07 PM
The report states that 'empty houses or those that refused to participate were passed over until 40 households had been interviewed in all locations.' This is a problem because hundreds of thousands of Iraqi families have fled their homes. Had those families that fled the clusters that were sampled been included in the survey, the estimated death rate would have probably been lower. Also absent from the report is the estimated death rate prior to January, 2002.
Posted by: Iraqi American | October 11, 2006 8:09 PM
As always, the question that counts is "if you don't accept this survey, what numbers do you estimate?" The American government won't make a guess, and the only possible reason for that is that they think it's in their own best interests - i.e. that the actual figures are as high or higher than the lancet figures. If they could make a defensible case that the actual figures were substantially lower, they would.
Posted by: Chris | October 11, 2006 8:21 PM
So, Ian, what is the 'correct' increase in mortality rate in Iraq since the invasion? Don't you want to know? Do you even really care? Would it really change your mind about the righteousness of the invasion if it were proven that it had cost the lives of half-a-million innocent people? Exactly how much death would it take to turn you off of this war? Seriously. I say you should embrace this information, take it in stride, and just stick to "freedom is messy" and "you have to break a few eggs to make an omelet."
Posted by: Charles Winder | October 11, 2006 8:23 PM
You guys are hilarious. So because we can't come up with another number, namely because it's virtually impossible to come up with an accurate one, then anything goes?
Why not just say 10 million and be done with it?
Bush has given a number - 30,000. That was last year. So it might be time to stop pretending that "the American government won't make a guess" - they already did.
As always, the anti-war liberals insert speculation and pure lies to fill a vacuum that cannot be filled with reliable and accurate information. It's like a game.
Posted by: Seixon | October 11, 2006 8:37 PM
It's like a game.
For you, perhaps.
So because we can't come up with another number, namely because it's virtually impossible to come up with an accurate one, then anything goes?
Ah, the Pontius Pilate defence. The 'anything goes' approach is yours, Sickson, because no number satisfies your gut. That's the kind of thing one expects from five-year-olds refusing to eat their dinners. Still, it takes a certain something to keep bullshitting for two years straight.
Onto substance:
The report states that 'empty houses or those that refused to participate were passed over until 40 households had been interviewed in all locations.'
As were households in which all members had been killed.
Posted by: ahem | October 11, 2006 9:22 PM
I'm willing to bet cash money that the number of families that have fled their homes is much, much higher than the number of families in which all members had been killed. I honestly don't know how this affects the results of the study, but one would think that if a large percentage of households in sampled clusters were empty because the families fled, then the survey is not truly representative of those clusters. Also I wonder how many households in each cluster refused to participate and why.
Posted by: Iraqi American | October 11, 2006 9:51 PM
As an actuary I can tell you part of the problem you run into with Joe Layman and real statisical analysis is that he has a hard time with any probability that is not 100% or 0%. In my experience, most people see only two possibilities: certainty and cluelessness, the latter a state with which many sadly have great familiarity. They don't know from confidence intervals and logorithmic fits, and the results of such studies constantly conflict with their "gut", and we all know what happens when facts conflict with the gut: the facts be damned. Just start a conversation at your next dinner party, preferably one that you don't care to get invited back to, about the Monte Hall 3 doors problem, and you'll see what I mean.
Since all statistical studies are gibberish to the Ian's of the world, they are all equally worthless, malleable to any desired agenda. Never mind actually showing the agenda exists or had an effect on the work. The mere possibility of a methodological bias is sufficient to reject the conclusions. It is as if students in statistics classes were told "Do whatever you want".
Posted by: MarkP | October 11, 2006 10:27 PM
"If thier 1/3 of all deaths are reported then the two would be in the same ballpark. - Andy"
Andy, 1/3 of 650,000 is 216,666. In July of this year the MoH & MLI in Baghdad had documented 50,000.
If their "1/3" is accurate, then there have been about 150,000 deaths, which is about a mile short of 650,000 and even a mile short of the lower bound 426,000.
But their 1/3 is not accurate to begin with. The MoH recorded over 80,000 in 2002 (excluding Kurdistan too), not 40,000. And the "should have reported 120,000" is somebody's uncited estimate of the number of deaths in Iraq in 2002, presented misleadingly as if it were a hard fact.
These are not even in the same city, let alone the same ball park.
Posted by: josh | October 11, 2006 10:28 PM
Bush has given a number - 30,000...
As I recall, Bush cherrypicked this from the lowest conservative estimate, perhaps even conflating the IBC and UNDP studies. Both of which are now rather dated, and indeed were so when Bush made his "estimate".
Posted by: Jacob A. Stam | October 11, 2006 10:43 PM
I think the design may have some problems if the data exhibit spatial correlation, which seems a reasonable assumption phenomenologically (violent death in a war zone being inherently spatially correlated). Starting with a randomly chosen hh within a cluster and then going to adjacent hhs doesn't seem to me to produce the kind of within-cluster heterogeneity one looks for in a cluster sampling framework. In wildlife surveys (same application, to get counts data) one often moves a minimum distance from one observation point to another within a cluster.
And to anticipate the WTFAY question, I'm an econometrician. Profs. Flinn, Peracchi, Wolpin and Ramsey might still consider me innumerate, but I passed their classes.
I'm not dismissing the study, but if this were someone's job market paper I'd be asking to see everything. And tomorrow I probably will ask for everything...
And Tim, people can be skeptical of the methodology without being a bunch of fundie nutjob "deniers." The zeal to characterize the other side in such condescending terms doesn't wear well on either side of the debate. Sounds like some of your commenters could use a reminder.
Posted by: DrSteve | October 11, 2006 10:54 PM
One thing I don't quite understand about this survey (and there may be cultural or legal reasons for it), is that the study states that about 92% of all households visited were able to present death certificates. This suggests that if the figure in question is roughly accurate, the Iraqi government should certainly have statistics relating to it, correct? Or do they not gather such data?
Posted by: mike | October 11, 2006 11:09 PM
If the figure of ca. 600,000 is accurate, has anyone noted that both the absolute number of deaths and the percentage of the population killed (ca. 2.5%) would be eerily similar to the figures for the US Civil War?
Posted by: DavidSewell | October 11, 2006 11:22 PM
I haven't seen the actual study, though the supplementary document provided by Andy Barenberg helped answer my last question. (Thanks, Andy.) Does the study in the Lancet break down cause of deaths by gender? Does it say, for example, what the male/female ratio is for those killed by gunshot or by airstrike?
Posted by: mike | October 11, 2006 11:32 PM
"Iraqi American" - the estimate of pre-2002 mortality is 5.5 per thousand per year. This figure matches the CIa's own estiamte for that period. It features prominently in the reprot. The 600,000 figure is the estimated INCREASE over the pre-war death rate.
Also, I would have thought that families that lost members would be more likely to flee, suggesting that this would bias the estimate downwards rather than up as you suggest.
Posted by: Ian Gould | October 11, 2006 11:50 PM
At this point I feel like anyone denying the deathtoll in Iraq is well over 600,000 is exactly the same thing morally as denying the holocaust. Worse even for Americans, because it is our government doing it, and that makes us partially responsible
Posted by: ThinkTank | October 11, 2006 11:54 PM
Does the study in the Lancet break down cause of deaths by gender? Does it say, for example, what the male/female ratio is for those killed by gunshot or by airstrike?
Yes for the former, no for the latter. There's a methodological critique at DKos that raises questions about potential oversampling. And I agree with those who would like to see the dataset. But I also think it's important to weed out the hack critiques in a forceful way to prevent them dominating the debate.
Posted by: ahem | October 12, 2006 12:03 AM
Actually, I'll correct myself: Table 2 in the Lancet paper has the raw death totals by sex and cause of death.
Posted by: ahem | October 12, 2006 12:22 AM
DrSteve wrote:
Well, that would make the estimates inefficient but why would it systematically bias the estimate upwards? In any event, the authors appear to have been aware of it--see the discussion on the left hand column of page 3 of the paper.
Iraqi American wrote:
A total of 15 households of 1849 refused to participate. And, are you saying that the households that fled had lower mortality than those that stayed?
Posted by: Robert | October 12, 2006 12:36 AM
Two things. How many people would have died at the hands of Saddam Hussein if he were still in power? Also, how many of the deaths the Lancet cites are the result of American military action and how many are a result of insurgent attacks and suicide bombings?
The following links may (or may not) fully explain these numbers:
http://www.logictimes.com/body%20count%20Jul%2006%20ed%20note.JPG
From: http://www.logictimes.com/antiwar.htm (I do not necessarily agree with some items on this site, but they do make a few good points here and there.)
and:
"Who's Really Killing Iraqis?
The Real 2006 'Iraq Body Count'
Iraqi civilians killed this year by Islamic Terrorists:
9,812
Iraqi civilians killed collaterally by Americans:
64*
*Source: IraqBodyCount.net (includes civilians caught in crossfire who may have been killed by the terrorists)"
From: http://www.thereligionofpeace.com/ (Caution: contents may offend some viewers.)
Posted by: Stephen Berg | October 12, 2006 1:02 AM
Seixon wrote:
Find the elephant yet?
Posted by: Robert | October 12, 2006 1:33 AM
Ian Gould, the 5.5 per thousand number is for the 14 months prior to the invasion, ie 2002, not pre-2002.
Stephen Berg, the study apparently attributes exactly zero deaths in all the households surveyed to government action or other violent acts in the last 14 months of the Saddam regime, suggesting that, if Saddam was still there, all those people who died violently would still be alive today. A surprising finding, sure, but that is what the survey says.
Posted by: BruceR | October 12, 2006 1:42 AM
Two things. How many people would have died at the hands of Saddam Hussein if he were still in power? Also, how many of the deaths the Lancet cites are the result of American military action and how many are a result of insurgent attacks and suicide bombings?
Impossible to say, as we don't know what the trajectory of his policies would have been. At the rate things were going, roughly this estimate fewer, since this is a count of excess mortality
As to #2: Does it matter? The "insurgent attacks and suicide bombings" are part and parcel of the policy trajectory set in motion by the Bush Administration when they invaded. One doesn't have to claim them to be wrong to have done so, although I do, to point this out. One needs to be a fantacist to deny it.
Posted by: Lettuce | October 12, 2006 1:43 AM
Stephen:
"How many people would have died at the hands of Saddam Hussein if he were still in power?"
&
"Also, how many of the deaths the Lancet cites are the result of American military action and how many are a result of insurgent attacks and suicide bombings?"
Q1: The study estimates the INCREASE in violent deaths since Saddam was overthrown. So your question was already answered by an even cursory read of this post or the Lancet piece itself. Like, just the abstract:)
Q2: American military action has destabilised the country, so both of your categories of violent death are "the result of American military action". Said military action has reduced (to zero) the number of deaths caused by Saddam's regime. This is good. However, the total number of deaths appears to have risen sharply. That is what we are discussing here.
Are you suggesting that this would have happened anyway? That's a pretty crazy assertion.
Posted by: FDB | October 12, 2006 1:59 AM
Ian and his ilk actually don't care how many deaths in excess of the preinvasion death rate have occurred. The point was made with admirable simplicity two years ago by a white House spokesperson-'We don't do Iraqi body counts'. The question to Ian is this-forget about the soft left totalitarian BMA body count guesstimate. How many Iraqi deaths have the CoW noted and accounted for since the invasion-oops-liberation of iraq? If no work has been done on this, why not? Is your position then that Iraqi deaths are not worth recording?
Posted by: stillwaiting | October 12, 2006 1:59 AM
Oops, too slow.
Posted by: FDB | October 12, 2006 2:01 AM
Retreading old ground:
How many people would have died at the hands of Saddam Hussein if he were still in power?
Compared to earlier periods, Saddam wasn't killing that many Iraqis in 2003: that is, he wasn't conducting large-scale repression (the Anfal) or involved in a large-scale war. The distribution of 'died-at-the-hands-of-Saddam' is not rectangular. The best response, courtesy of Daniel Davies, is to note that HRW did not consider invading Iraq a humanitarian intervention -- that is, one which would save more lives under imminent threat than would be lost. Might there have been another Anfal? Very possibly. Would that have created a more credible casus belli? Undoubtedly. But it's fallacious to make a humanitarian case in March 2003 (as opposed to a retributive one) based upon the killings of 1991.
Also, how many of the deaths the Lancet cites are the result of American military action and how many are a result of insurgent attacks and suicide bombings?
Read the companion paper. It's not long. There are piecharts to help you out.
Posted by: ahem | October 12, 2006 2:05 AM
Robert Blendon, director of the Harvard Program on Public Opinion and Health and Social Policy, said interviewing urban dwellers chosen at random was "the best of what you can expect in a war zone."
But he said the number of deaths in the families interviewed -- 547 in the post-invasion period versus 82 in a similar period before the invasion -- was too few to extrapolate up to more than 600,000 deaths across the country.
Donald Berry, chairman of biostatistics at M. D. Anderson Cancer Center in Houston, was even more troubled by the study, which he said had "a tone of accuracy that's just inappropriate." http://www.nytimes.com/2006/10/11/world/middleeast/11casualties.html
??????????
Posted by: wacki | October 12, 2006 2:49 AM
Blendon was badly misquoted on that one; he was talking about the precision of the point estimate. I'll try to get permission from him to post my email correspondence.
Posted by: dsquared | October 12, 2006 4:18 AM
This suggests that if the figure in question is roughly accurate, the Iraqi government should certainly have statistics relating to it, correct? Or do they not gather such data?
See my challenge to Lambert on that, this is an easy way to see how ridiculous the study's conclusions are.
At this point I feel like anyone denying the deathtoll in Iraq is well over 600,000 is exactly the same thing morally as denying the holocaust. Worse even for Americans, because it is our government doing it, and that makes us partially responsible
Here we come full circle in the left-wing circle-jerk. Create a death toll that is completely absurd with a survey that doesn't fit its purpose, then call everyone who denies it akin to a Holocaust-denier.
But go ahead, take me up on my challenge, and let's see who has a problem with facing reality.
Posted by: Seixon | October 12, 2006 4:24 AM
Seixon, if you had read the thread, you would have seen the arguments that even in pre-war Iraq, the bureucracy only managed to report a fracttion of natural deaths up to the top.
This is not "an easy way to see how ridiculous the study's conculsions are". Have you ever worked in a bureucracy, or even in an office? Unless you have an extremely well-managed organization, the top will not have an accurate picture of what the folks on the floor see. Reports aren't filed, documents disappear in the "paper mill", happens to the best. It would be nothing short of a miracle if your average district council or local authority in Iraq managed to present accurate data to the central authorities in the present situation - especially when we know that they weren't able to before the war either.
So could you stop saying that? I'm sure you have lots of other great arguments that you can use, there should be no need to use so poor ones.
Posted by: Harald Korneliussen | October 12, 2006 5:49 AM
It seems the Right knows a thing or two about in-depth analysis of scientific studies if the response to the Lancet paper is anything to go by. We in the reality-based community can learn a lot from this. I've summarised here. There is much wisdom. Oh so much wisdom.
Posted by: Nexus 6 | October 12, 2006 5:50 AM
"Seixon, if you had read the thread, you would have seen the arguments that even in pre-war Iraq, the bureucracy only managed to report a fracttion of natural deaths up to the top."
Harold, if you had read the thread, you would have seen that the "arguments" you refer to are factually incorrect on one hand and claim to test those (incorrect) facts against an unsourced estimate, which is itself presented misleadingly as an established fact.
I look forward, in any case, to somebody taking up Seixon on his challenge. The persistent evasions of it make me think he may have a point.
Posted by: josh | October 12, 2006 6:04 AM
Seixon's challenge, for those who can't be bothered clicking through, is to work out from the study the number of death certificates issued post-war. I don't know why he thinks this is a conclusive refutation though, as I don't think he has the real death-certificate data or even knows if it exists.
Posted by: ekmi | October 12, 2006 6:24 AM
Seixon and Josh,
You both seem to be claiming that the Iraqi central government had the ability, before and after the invasion, to accurately keep track of death rates. This is not the case has Harald has clearly pointed out. Josh, perhaps you just missed it as it's a big thread now, so here is what Harald referred to (straight from the companion paper of the Lancet study):
"Even with the death certificate system, only about one-third of deaths were captured by the government's surveillance system in the years before the current war, according to informed sources in Iraq. At a death rate of 5/1,000/year, in a population of 24 million, the government should have reported 120,000 deaths annually. In 2002, the government documented less than 40,000 from all sources. The ministry's numbers are not likely to be more complete or accurate today."
Note, they've determined this death rate of 5.5/1000 by using the same method as in the main study. This number is verified by the CIA's own estimates. It should be clear that the Iraqi government did not have the ability to determine accurate figures before the war, so why would this ability be improved after the war? So, to repeat to Seixon, will you please stop using this (discredited) argument?
Posted by: Paul | October 12, 2006 7:17 AM
"Well, that would make the estimates inefficient but why would it systematically bias the estimate upwards?"
I haven't suggested it would (reserving the right to object). The effect on the SEEs could be bothersome enough.
"In any event, the authors appear to have been aware of it--see the discussion on the left hand column of page 3 of the paper."
I read that too, Robert. That discussion just notes the garden-variety adjustments one makes to any clustered/stratified/weighted sample. Sandwich estimators, etc. We would always expect some degree of adjustment to clustered samples, because each household member shares characteristics with others in the same household. The discussion also suggests it's within-cluster temporal autocorrelation they're talking about (they cite Diggle's work on longitudinal data in the accompanying note). So I'm not sure the passage you cited addresses my question.
Posted by: DrSteve | October 12, 2006 7:29 AM
It seems certain people have forgotten who was in control of the Iraqi government before the war... Was it because they couldn't count all deaths accurately, or was it because they didn't want to? Why would Saddam want to publicize numbers of all the people he had killed? Doesn't sound like a good thing to do as a dictator.
The media and the Iraqi government have access to all the data on death certificates. This would be easy to track down and compile. Why, you'd almost think that it would be a much better idea than interviewing 1800 families!
But no, that wouldn't have provided the result they wanted. The Iraqi government says the study is absurd, you know, the folks who actually preside over the bureaucracy that has the records to prove how many death certificates have been given out.
That won't stop the usual suspects from claiming they know better, because of a friggin cluster survey. Sheesh. Get back to screaming about the 2004 exit polls, at least you weren't being political opportunists with the lives of Iraqis.
ekmi,
How many morgues and coroners do you think exist in Iraq? Less than 1800? Why didn't the JHU team start there instead?
I know the answer, do you?
Posted by: Seixon | October 12, 2006 7:34 AM
Paul, I know what Harold referred to and I addressed it above. The passage you quote is both false and misleading.
Seixon has a valid "challenge" open, and my previous comments regarding the evasion of it keep getting reinforced.
Posted by: josh | October 12, 2006 7:49 AM
"The media and the Iraqi government have access to all the data on death certificates."
What planet are you on? One with reliable computer systems, sternly conscientous bureucrats, frictionless administration and supreme efficiency? I can't determine the ancestry of all Norwegians, even though the old church books, recording births, marriages and deaths, are accessible to all! If the church books were all we had, even deciding what the population was in 1870 would have been a tremendous task. Now add half a century of war and endemic corruption...
Posted by: Harald Korneliussen | October 12, 2006 8:06 AM
Seixon,
"The media and the Iraqi government have access to all the data on death certificates."
That's as compiled by the Iraqi government. The Iraqi government which can't count birth certificates before the chaos of invasion. As noted, their account is a gross underestimation, not because of a Saddam lead coverup, as you insinuate, but because their counting system was ineffectual. Why else was the study able to uncover a pre-war death rate of 5.5/1000 deaths per annum versus the Iraqi government's figure of roughly 1.5/1000. 1.5/1000 is extremely low compared to a bunch of western countries. The only other coutries with such dubiously low death rates are, you guessed it, other countries with poor central administration and are therefore incapable of counting death certificates. The post war death rate is estimated at 13.2/1000 per annum.
It's funny actually, because you, and others, claim the death rate is high in Iraq without seemingly doing any kind of comparison with any other countries. Did you know the UK has a death rate of 10.3, and France is estimated at 9.1. Put in that perspective I'd say 13.2 doesn't seem too high, and 5.5 seems really low. That makes me doubt how effective the Iraqi's are at issuing death certificates....
Is there anyone here who can comment on the validity of those UK and French death rate figures?
Posted by: Paul | October 12, 2006 8:10 AM
Josh,
"The passage you quote is both false and misleading."
OK, if that passage is false please can you provide a link or some sort of evidence that the number of deaths in 2002 was 80,000 (excluding kurdistan), as you suggest and not 40,000 as the study suggests. I'm only going by what was in the study, so if there is some other information out there it'd be good to see it.
Posted by: Paul | October 12, 2006 8:16 AM
To follow up on myself: there's a reason that goverments and large organisations often prefer statistical sampling to comprehensive data gathering. For one thing, it's cheaper, for another, it will often give more correct results!
(There are also some, especially in government, who prefer statistics because it's easier to mislead people with them. As far as I know, US unemployment numbers are sampled, whereas in most other countries they are comprehensive. I have heard allegations that these samples are made using phone calls, thus eliminating people who don't own a phone, or couldn't afford to pay the phone bill last month. I mention this as an apropos, but I don't doubt that the Seixonians will rather "answer" this paragraph than the former!)
Posted by: Harald Korneliussen | October 12, 2006 8:16 AM
Dr. Steve wrote:
I'm not sure it doesn't. First, I agree those were garden-variety adjustments, but spatial correlation in clusters is a pretty garden-variety concern to adjust for. Second, the sampling unit is the household, so the intra-household variability isn't the issue -- I thought your original comment about spatial correlation was about the correlation between households. Nonetheless, if your concern is that spatial correlation may affect the SEE's and not the estimates, then it's helpful to clarify what that means for the readers who've never taken econometrics: you're not questioning (at least, now) the central estimate of 655K for total excess deaths but rather you're concerned that the stated CI around it is too narrow.
Posted by: Robert | October 12, 2006 8:26 AM
Paul, first things first. What is the source of the 40,000 assertion in the study? I noticed no citation for it in the study, but certainly at least a few folks here have a direct line to Les Roberts or others, and could easily pass along their source.
Posted by: josh | October 12, 2006 8:27 AM
"Seixon has a valid "challenge" open, and my previous comments regarding the evasion of it keep getting reinforced."
That would be this challenge(?):
"If you feel the Iraqi government and I are in error about Iraqi casualties, please have a go at gathering in the 500,000+ death certificates that this study would have to claim exist in Iraq."
Or maybe the challenge is to tell poor Seixon where the elephant is hiding in the living-room.
I'll tell you what, Josh, you go to Iraq, gather up just 10,000 death certificates and a letter from Ayatollah Sistani confirming that there aren't any more.
Do you have a serious query or are you just being a pain in the arse?
Posted by: Kevin Donoghue | October 12, 2006 8:28 AM
The death certificate issue is one I mentioned myself in an earlier thread and there are a few possible resolutions in no particular order--
They could do a better job adding, but they lie about it for political reasons.
We know that there are cases where they have suppressed data and there is always some incompetence. We will each have our own reasons for believing one or another of these factors.
The solution is obvious--pressure from everyone on the US and Iraqi governments to support an independent survey to determine the number of deaths that have occurred. It's outrageous that Bush and Casey quote incomplete Iraqi government reports (which are guaranteed to be too low by some unknown amount) or worse, compilations of media reports as a substitute for their clear responsibilities. I hope Josh's response in this thread is not indicative of the main thrust of Iraq Body Count's attitude when they are finished digesting the Lancet paper. Everyone should get behind what the Lancet authors have repeatedly asked for. Not that I actually expect the government to do this, but if they don't, it will be because they know there is something to hide, even if the numbers aren't really as high as 600,000.
Posted by: Donald Johnson | October 12, 2006 8:32 AM
Josh, the source of this figure seems to be Iraqi government documented figures from all sources. Presumably a data sheet exists somewhere with all the numbers compiled on it.
Have you got the source of the 80,000 number yet?
Seixon, the challenge just seems to be for you produce some credible criticism of cluster sampling under this set of conditions. I haven't seen you do this yet.
Posted by: Paul | October 12, 2006 8:38 AM
Paul asked: