The biggest limitation of the Lancet study is the small sample size. We can be reasonably confident that deaths have increased in Iraq since the invasion, but the 100,000 estimate is a very rough one. The sample from Falluja found an alarming number of deaths from air strikes, but since it was only one sample it is hard to guess how many others have died in similar ways. Fortunately, it is easy to address both these limitations. For the cost of running the Iraq war for about two minutes it would be possible to do a survey with four times the sample size and which oversampled in violent areas. Unfortunately, Tony Blair doesn't want to know the answer:
Yesterday Tony Blair rejected a call from more than 40 diplomats, peers, scientists and religious leaders who pressed for an independent inquiry for a civilian death toll.
"Figures from the Iraqi ministry of health, which are a survey from the hospitals there, are in our view the most accurate survey there is," he told parliament.
The health ministry has produced a figure of 3,853 civilians killed between April and October this year. But it is not clear whether those figures cover the entire country, how they were confirmed or what causes of death. No figures have been produced for the first year since the invasion.
Now, it is a well-established principle of statistics that random sampling will give more accurate estimates than trying to count the entire population. If Blair doesn't know this, his scientific advisors surely do. I think it is obvious that if he felt that the Iraqi death rate had gone down, he would have been in favour of a survey to establish this.
So here's a question that should be put to all those who object to the Lancet study because of the wide confidence interval: Do you agree that a larger survey should be carried out?
Also of interest: An interview with Richard Garfield about the study, and a letter to the Independent by the study's authors (posted by Tribbs in this discussion):
Fallujah is the only insight into those cities experiencing extreme violence (ie Ramadi, Tallafar, Fallujah, Najaf); all the others were passed over in our sample by random chance. If the Fallujah duster is representative, there were about 200,000 excess deaths above the 98,000.
Perhaps Fallujah is so unique that it represents only Fallujah, implying that it represents only 50-70,000 additional deaths. There is a tiny chance that the neighborhood we visited in Fallujah was worse than the average experience, and only corresponds with a couple of tens of thousands of deaths. We also explain why, given study limitations, our estimate is likely to be low. Therefore, when taken in total, we concluded that the civilian death toll was at least around 100,000 and probably higher, not between 8,000 and l94,000 as Mr. Straw states. While far higher than the Iraq Ministry of health surveillance estimates, on 17 August the minister himself described surveillance in Iraq as geographically incomplete, insensitive and missing most health events.
More on this from this briefing by the Prime Minister's Official Spokesman; the exchange at Prime Minister's Questions is here.
this being a war and all, why are the numbers so important? The simple fact is that if the insurgents stop, the dying will end. Insurgents don't stop, dying continues. It's black and white.
I know, why should they trust us? What's the down side of trusting us compared to dying by the hundreds of thousands for not trusting us? It must be clear to anyone with any sort of intellect that the dying is solely due to the fighting.
Ben writes, "The simple fact is that if the insurgents stop, the dying will end."
Yes. But what do simple demonstrable facts and rational comments have to do with this debate?
The point is BushCo I saw this coming and decided the cost was too high. BushCo II is far, far, smarter, more resolute, stronger yadayada than that and went in, kicked ass, and declared victory. Murrica is safer, and we've taken out [fill in BS % here] of al-Q leaders. Oh, I've forgotten the rest of the propaganda.
The numbers are important, poor ol' Ben, because they didn't have to die so our dear leader can look resolute and strong.
D
That's nice, Dano. Glad you're not in charge.
ben: No, the dying is not solely due to the fighting. The dying is also because:
- the hospitals have no electricity or medicines or paid staff;
- there's no clean water;
- there's no sewage treatment;
- there are dead bodies in the streets;
- there are no police working to stop random bandits mugging people in the streets;
- there's no working fuelled transport to get sick babies, grannies, and pregnant women to hospital, and often it's not safe when there is transport;
- there are huge quantities of unexploded munitions lying about, getting picked up by kids, stepped on by grannies, sold as souvenirs;
- and so on.
The dying is, unmistakably, due to the invasion. Some of it (probably most of it) is due to the ongoing fighting. But all of it is due to the invasion.
Dano writes, "The point is BushCo I saw this coming and decided the cost was too high."
When BushCo I made their determination, the World Trade Center hadn't been bombed in 1993 and then brought down in 2001.
Also, when BushCo I made their determination, they thought Saddam was weakened enough that his own people could finish him off. Instead, his own people "only" managed to take 14 out of 18 governates, and Saddam Hussein was able to claw back and maintain power by brutal repression.
"Now, it is a well-established principle of statistics that random sampling will give more accurate estimates than trying to count the entire population."
This would be a new principle, unique to "La la la I can't hear you" Doltoid ?
As a straight statement of fact, this is clearly wrong, and demonstrably so. Indeed, some of us had the idea that statistics used samples of a population to estimate what the population mean might be; as the sample size tends to the population size, the accuracy can get better. But the notion that an estimate should be better than a direct measurement of the population is perverse.
At the very minimum, there need to be all sorts of caveats on this, e.g. to include the condition that it is impossible or impractical to measure the entire population.
I am just wondering what the doltoid response will be ?
La la la I can't hear you ?
j
Ben: "The simple fact is that if the insurgents stop, the dying will end. Insurgents don't stop, dying continues."
One of the principal rationales for the war was that it was necessary to save the Iraqis from Saddam. If it actually failed in this and resulted in more rather than fewer deaths we need to know this in order to assess whether future wars based on a similar rationale are justified.
As it happens I supported the first gulf war, the bombing of Kosovo and the invasion of Afghanistan because in each case there was high probably that failing to act would have produced much greater loss of life. I would probably, depending on the precise circumstances, support similar military action in congo and Sudan.
I opposed the invasion of Iraq because I foresaw almost everything that has goen wrong since.
As for your argument that everything would get better if the insurgents only stopped fighting, there are three important points you overlook.
1. The damage to Iraq's infrastructure isn't going to magically fix itself overnight. Iraqis will continue to die of water-borne diseases and lack of access to medical services.
2. The Kurds and the arabs have been fighting over northern Iraq for hundreds of years. Add in the oil fields in the area and that is unlikely to stop any time soon. The current violence in Mosul and Kirkuk is as much a reflection of that ethnic rivalry as it is of the anti-coalition insurgency. By relying disproportionately on Kurdish peshmurga to staff the new Iraqi army and failing to act agaisnt ethnic cleansing of arabs by kurds, the Coalition risking being seen as supporting the Kurds in this conflict.
3. A future Iraqi government is much more likely to follow the example of other arab nations (such as Egypt and Syria) than it is to follow western standards of democracy and human rights. If Iyad Allawi, who was a member of the Baathist inner circle for over a decade, is still running that government it is likely that it will be essentially a more moderate version of Saddam's repressive state. The Shia may replace the Sunni as the dominant group but the torture and murder of critics of the regime, the rigging of elections and the resulting violent opposition are all likely to continue.
Do us a favour Ben and post the results of the application of your intellect to the helpful comments from Dano, Nick and Ian, above?
Alright, no nead to be snarky, I simply don't see the point of the body count
Nick, what I mean to say, since you guys won't read into anything, is that once the fighting stops, the immediate dying due to fighting will stop, and the problem of dying due to poor infrastructure will be able to be addressed. So, the dying is essentially due to the fighting (yes, that means it's due to the invasion, ok). But leaving will not help at this point. So, for the sake of looking for a solution, the fighting must stop before the dying will stop.
Ian, at this point I'm not interested in the justification for the war. The war happened. Saddam was defeated, and that's that. The problem at hand is how to deal with the problem we have now. To address your enumerated concerns
1. true, not overnight, but the sooner the better.
2. yes, the clash of cultures presents a problem.
3. I fear that what you say is among the likely outcomes.
It seems to me that this war teaches us that, for future reference, nation building in the middle east is a waste of time. But what is the alternative? Does anyone seriously believe that you can negotiate a peaceful and just settlement with a ruthless dictator? Does anyone truly believe that the insurgents are acting in the best interest of the average Iraqi or that they would be better off if we left the country to the Islamist fanatics?
I don't see anyone here posting a solution to the problem? Any ideas what ought to be done?
Ben, for the record, not only do I think that the US shouldn't leave, I think tht Australia, having supproted the invasion, should send more troops.
I see no contradiction between thinking that leaving would make a bad situation worse and thinking we shouldn't have invaded in the first place.
Our phrase here, found in china and antique shops, is: 'you broke it, you bought it'.
The overarching point is that many people saw this coming. The death toll estimates point out the fact that our Administration didn't plan this war very well (the explosions on Sunday rattled windows for a mile radius. Gosh, I wonder where explosives that powerful came from?!? BTW, I know someone in that unit that was attacked). The numbers contradict the propaganda emanating from Washington.
Two main f*ckups...er...architects of the war were just given medals of freedom. Puh-leez. There is no accountability in this Administration, and the enablers wishing to suppress the death toll estimates further this lack of accountability mentality.
Change the ding-dang course of action. Admit a mistake. Do something.
D
That's fair, Ian.
Getting back to the original theme of this post;
"So here's a question that should be put to all those who object to the Lancet study because of the wide confidence interval: Do you agree that a larger survey should be carried out?"
Yes, absolutely. It's been my contention all along that the Lancet study, standing alone, is incapable of providing an accurate count of the overall number of excess deaths, and also the breakdown of the overall number into the most relevant subsets.
Two questions Tim:
"Now, it is a well-established principle of statistics that random sampling will give more accurate estimates than trying to count the entire population."
In western societies, for homicides, and other causes of death such as motor vehicle accidents, nation wide statistics are actual counts, driven by data collection required from law enforcement agencies. These counts are fairly precise, certainly in the high 90 % accuracy range in all likelihood. Is it your belief that a random sampling study would provide a more accurate figure for homicides or motor vehicle fatalities in western societies than the current actual statistical compilations? I realize that an actual count of violent deaths in Iraq is not feasible, but assuming the environment allowed an attempted count, would it not be more accurate than random cluster sampling?
My second question refers to the earlier thread concerning the breakdown of the non-Falluja death figures, provided by Les Roberts.
I noted in my first post there that the numbers don't seem to work, when comparing the extrapolated estimates for excess deaths with the actual study data. The example I used was the study's contention that at least 30,000 of the excess 100,000 death toll outside of Falluja were from coalition bombing, yet on a purely percentage calculation from the excess death figures provided in the study data, I get 17,000 bombing deaths.
I also mentioned that death from bombing did not appear to be the primary cause of death after regime change as claimed, since bombing deaths amounted to only half the total of violent deaths not directly attributable to the coalition, and were also less than the increase in deaths from accidents and heart attack.
No one bothered to address this (which surprises me). The authors must have some rationale for these inconsistencies. The paper itself mentions this on page 4:
" The numbers of excess deaths (attributable rates) were estimated by the same method, using linear rather than log-linear regression. Because the numbers of deaths from specific causes were generally very small, EpiInfo (version 3.2.2, April 14, 2004) was used to estimate the increased risk of cause-specific mortality without regard to the design effect associated with the cluster data."
Tim, is this what the authors are relying on when extrapolating an estimate for bombing deaths that does not correspond to the raw data? And if so, what are their grounds for doing so? With only 1 violent death reported before regime change, (and we aren't told what category it fits into) how can the authors decide that bombing deaths should be extrapolated at a greater rate than other causes of violent death?
Finally, it's also interesting to see the authors (in the letter posted by Tribbs) go on the record with an overall death estimate for Fallujah. It seems that they've adopted a minimum of 20,000, and a high of 50,000 to 70,000, for the city of Falluja itself, while claiming there is only a very remote chance the toll could be under the 20,000 low estimate. I'm wondering why they didn't have the confidence to place this within the paper.
If an actual count isn't possible for all of Iraq, I believe a determined effort to reach a reasonably accurate excess death figure for Iraq requires an actual count for Falluja, with an estimate by larger sampling carried out for the rest of the country.
In western societies, for homicides, and other causes of death such as motor vehicle accidents, nation wide statistics are actual counts, driven by data collection required from law enforcement agencies. These counts are fairly precise, certainly in the high 90 % accuracy range in all likelihood. Is it your belief that a random sampling study would provide a more accurate figure for homicides or motor vehicle fatalities in western societies than the current actual statistical compilations?
Mike, it's a commonplace among criminologists that it is better to use victim survey data than police counts, for a variety of purposes. It doesn't take much imagination or a particularly high stack of PD James novels to come up with a dozen categories of homicide which wouldn't make it into police statistics.
I think you have a point about the extrapolated numbers and the causes; I would say that this is an intrinsic problem of the technique used rather than a mistake made by the survey authors, but I'd agree that the numbers are not really worth much. This is why I suggested that the correct presentation of the figures would be to say that the evidence suggests the invasion lowered the death rate in the Kurdish North, increased it by a factor of 1.5x (CI 1.1-2.9) in the bulk of Iraq, and caused a very significant number of deaths in a small number of high-violence areas. This avoids the headline number, brings the Fallujah information back in and avoids the attribution problems.
D Squared:
Speaking from experience in the field, (in Canada anyway), the victim data normally used by criminologists comes from the police stats (We call it UCR, Uniform Crime Reporting), which is a Federally administered reporting system that requires all Canadian police services to submit crime stats in the same format. Homicide is narrowly defined, before it can be counted and reported.
I agree with what you're saying concerning the risk factor being far less problematic to deal with, but don't you also believe that further sampling would provide enhanced data?
The simple fact is that if the insurgents stop, the dying will end. Insurgents don't stop, dying continues. It's black and white...
Unfortunately, there are Iraqis thinking along these lines:
The simple fact is that if the occupation stops, the dying will end. The occupiers don't stop, dying continues. It's black and white... I know, why should they trust us?...
Commenting on something in a previous thread, Mike, in the first Gulf war the US did deliberately destroy Iraq's civilian infrastructure and intended to use the sanctions to prevent its repair. The idea was that the resulting civilian suffering would place pressure on Saddam, either causing his government to fall or at least putting enough pressure on him to force him to comply to disarmament demands. (It appears he did.)
You can find all that information in a June 23, 1991 article by Barton Gellman in the Washington Post. He interviewed Pentagon targeting planners for the first gulf war and they told him what I just wrote. I think the insurgents are using the same strategy against the current Iraqi government (us). It's a war crime and if I understand the terms correctly, arguably a crime against humanity whoever does it. This is what bothers me about US foreign policy--our leaders can get away with darn near anything and never expect any worse consequence than some name-calling by lefties in a blog comment section.
On the Lancet numbers, apparently I understood how the authors were thinking, for the most part. I think the way the study should have been treated was as a very suggestive and preliminary study containing evidence that the civilian death toll from all causes and in particular from American bombing is much higher than people realized. Therefore a much larger survey should be conducted and there should be no further massive bombing raids as occurred in Fallujah and some other places. Whether the Fallujah sample is representative of tens of thousands of deaths or much less, Americans shouldn't be using those tactics.
but don't you also believe that further sampling would provide enhanced data?
Everyone commenting upon the paper, here, has said much the same thing. Usu. in the context of 'further sampling likely will show the original technique to be fairly accurate'.
Which is likely why it is being repeated again. More sampling, more accuracy. More dots for the scatterplot.
D
dsquared wrote:"Mike, it's a commonplace among criminologists that it is better to use victim survey data than police counts, for a variety of purposes. It doesn't take much imagination or a particularly high stack of PD James novels to come up with a dozen categories of homicide which wouldn't make it into police statistics."
your own example shows clearly that your argument is useless ! Why would a "victim survey" pick up the homicides that don't make it into police statistics ? A victim survey could produce a large number of false positives, which could give headlines and publicity, though...
"I think you have a point about the extrapolated numbers and the causes; I would say that this is an intrinsic problem of the technique used rather than a mistake made by the survey authors, but I'd agree that the numbers are not really worth much. "
Quite. This used to be known as "garbage in, garbage out". It is the data that isn't worth much, not the extrapolation method.
J
Dano:
That's where we disagree. I'm not convinced more sampling will replicate the data from this study. Using your argument, there is no need for further sampling, because you believe the authors got it right (that being an extrapolated estimate of excess deaths, and their composition) the first time.
Donald:,p>
Iraq had managed to rebuild its infrastructure to some extent following Gulf War 1. The heavy bombing of infrastructure in 1991 was not continued throughout the 1990's and 2000-2003. Bombings after hostilities ended in 1991 were targeted on military sites, through enforcement of the no-fly zones.
The impression that is often left when discussing sanctions, bombings and the state of Iraq's infrastructure is that it was uniformly in ruins, up to the time of regime change. This simply wasn't the case. The authors of the study corroborate this with their statement that infant mortality rates in Iraq prior to regime change were consistent with those in surrounding countries. If Iraq's water treatment plants, hospitals and hydro-electric grid were still in ruins prior to invasion, this would not have been the case.
We consistently hear critics of the war castigating the coalition for failing to keep the lights on at a pre-war level, failing to provide clean drinking water, etc. If these complaints are credible, then the implication is that the pre-war infrastructure was functioning at a reasonable level.
As D Squared and Heiko have pointed out, the Lancet should have stuck with a straight risk factor argument concerning excess deaths. The authors decided to provide extrapolated estimates for specific causes of death, and these attributions don't appear to be consistent with the data, and also expose limitations resulting from the sample size. I'm not prepared to agree with you that the death toll from American bombing is " much higher than people realized." As I've argued throughout, determining the casualties from bombing looks to be a complex process that a single cluster sampling is ill-equipped to achieve.
On the question of the use of air strikes by the coalition, I agree with you entirely. Even when dealing with a full blown insurrection in a place like Falluja, the Americans have to find other ways of suppressing it than bombing. The April bombing of Falluja was inexcusable, and I don't know what the Americans hoped to accomplish with it. The ground force in place at the time was far too small to pacify and occupy the city, so the entire exercise was worse than pointless. It killed many innocent civilians.
The insurgents don't even care about the segment of the civilian populace that may be sympathetic to their cause, and use these innocents as cover and protection from coalition attacks. The Americans have an obligation to care, and that translates obviously into the imperative that they refrain from air strikes where civilians are known to be or likely to be. There's no question that this will cause difficulty for the coalition militarily. There is however, no other acceptable approach.
I agree that Iraq's infrastructure was partially repaired by the time of the invasion, but this was in spite of the sanctions and I think the key phrase there is "partially repaired". Damage during the invasion, the insurgent sabotage, and either incompetence and/or corruption of the Bush Administration caused things to go backwards.
From my perspective, some people on both sides of the Lancet paper debate over-reacted--the study wasn't conclusive by any means and people shouldn't think it is, but on the other hand it should make people suspect that the death toll from all causes (particularly bombing) might be considerably higher than one might have thought if you relied on what Western reporters were able to find out.
I think the extrapolated numbers the Lancet people provide are consistent with the data--that's not to say that one should necessarily accept them. (A bigger survey is needed.) The numbers you get from the non-Fallujah data seem reasonable--heck, if you throw caution to the winds and accept the breakdown given some of the numbers seem perfectly consistent with what we know from other sources. For instance, 7 out of 21 deaths were murders and that translates roughly into 20,000 murders. I think Baghdad has been suffering 700 a month, so 20,000 for the whole country might even be small. The Fallujah outlier is what leads the authors to say the death toll is likely to lie on the high side of 98,000, due to what they suspect are many tens of thousands of bombing deaths. That might be wrong, but it's definitely a reasonable interpretation of their data, as I was pointing out in previous posts.
But anyway, further study is needed, and we both agree the US shouldn't be dropping bombs in urban areas.
Donald:
"I think the extrapolated numbers the Lancet people provide are consistent with the data--that's not to say that one should necessarily accept them. (A bigger survey is needed.) "
How can you come to this conclusion? The bombing deaths alone seem to extrapolate radically different from the data, as I've pointed out earlier.
Enquiring Mind:
"The simple fact is that if the occupation stops, the dying will end. The occupiers don't stop, dying continues. It's black and white... I know, why should they trust us?..."
Do you really believe that an immediate withdrawal of coalition forces means peace? Forgive me for being blunt, but this is naive in the extreme. Removing the coalition forces means a partitioning of Iraq at best, and a horrific civil war at worst.
That's where we disagree. I'm not convinced more sampling will replicate the data from this study. Using your argument, there is no need for further sampling, because you believe the authors got it right (that being an extrapolated estimate of excess deaths, and their composition) the first time.
No one said, Mike, that the data could be replicated.
You want the technique to be replicable.
I said replication likely would show the initial technique (shoulda said paper) to be fairly accurate. You can't state my argument is that the study got it right the first time from what I wrote, sorry.
There's nothing radical here. If you're convinced the paper is incorrect, start agitatin' for more people to get out there to prove you right.
Best,
D
"It must be clear to anyone with any sort of intellect that the dying is solely due to the fighting."
At last, another pacifist! Nice to meet you!
Mike;
The text you quoted was an illustration of the opinions I feel are held by some Iraqis. The idea being to illustrate the problem that both sides feel the bloodshead is caused by the other sides intransience. Which in turns justifies their cause. You are doing me a disservice by making out that it is an opinion held by me. For you information I have not, in fact, expressed an opinion here on the justness of the war or occupation.
Mike, on the subject of the bombing deaths, I'd be repeating myself. But I do that a lot, so here's a short version. If we're very naive and just extrapolate linearly and ignore error bars, the 32 non-Fallujah neighborhoods suggests about 17,000 deaths from American air attacks (or what is believed to be American air attacks), but that's only counting neighborhoods that have been lightly bombed or rocketed, so to speak. 1 neighborhood out of 33 was heavily bombed. Take that statistic literally and you'd add 200,000 deaths to the total. Be cautious about it, and you say to yourself "Well, 200,000 seems way high, but if one such neighborhood shows up in a random sample of 33, then it's unlikely though of course possible that they are as rare as, say, 1 in 330. So that's at least 20,000 deaths." So anyway, maybe tens of thousands dead in heavily bombed neighborhoods. The Lancet people are being conservative in interpreting the Fallujah point, presumably because they also suspect 200,000 is too high. But the data point exists, and it suggests the possibility of a great many bombing deaths. That's a reasonable way to look at the data, though not necessarily correct. Of course, the question could and should be settled with a larger survey.
Donald Johnson wrote: "The Lancet people are being conservative in interpreting the Fallujah point, presumably because they also suspect 200,000 is too high. But the data point exists..."
There is one straightforward interpretation. That the data point exists, and that the 200k figure is indeed too high. The conclusion from that could be that the survey is not adequate.
haven't heard that idea, yet. Instead, "that's at least 20,000 deaths"; there is the "possibility of a great many bombing deaths". None of these have any connection with the published conclusions, because any scientist would kick them out of peer review.
but shucks, why bother letting data interfere with preconceptions !
j
Dano said: "I said replication likely would show the initial technique (shoulda said paper) to be fairly accurate. "
so I guess you are suggesting that a repeat would come up with a number within the boundaries 98k +/- 90k. Strangely enough, I can hold the view that this study has such severe limitations it is not worth publishing; and simulataneously that your comment is accurate. But the fact is a confidence interval from 8 to 190 thousand is so wide as to be practically meaningless.
j
Enquiring Mind:
" My Bad," as the say. I completely mis-read what you had posted. I tend to read things once, and quickly, unless I have a need to re-read them.
Sorry!!!
Dano:
I've made several references in previous posts that any future studies would have to use the same methodology as the Lancet study, otherwise there would be no basis for testing the accuracy of the original study's data.
If you want to split hairs over whether your use of "fairly accurate," is somehow inconsistent with my characterization of your words as " got it right," then be my guest.
By the way, why do you believe the data is " fairly accurate," when learned defenders of the study such as D Squared are conceding that the breakdown of the 100,000 excess death estimate, the parts that make up the sum, do not coincide with the authors' written conclusions or appear to accurately reflect extrapolations from the data itself?
Donald:
The problem with your argument is the fact that the authors have made it clear, both in the study and in subsequent interviews, that their 100,000 estimate excludes the Falluja statistics. The " or more," that comes after the 100,000 estimate in the paper is the authors' allowance for the Falluja data. They're clearly stuck with the 100,000 figure without resorting to the Falluja data, by their own choice. That means you are as well.
There is no way around this. The authors say that their study implies at least 30,000 coalition bombing deaths occurred outside Falluja. The data suggests a number far lower, 17,000.
That was why I asked you how you could come to the conclusion that the extrapolated numbers are consistent with the data. The huge discrepancy in the bombing deaths numbers has a ripple effect on the other death subsets as well. As I've said earlier, before we get to a single violent death, before we consider the number of those who might have been killed directly by coalition bombs or bullets, we've got 37% of the excess deaths outside Falluja coming from heart attacks and accidents. The authors' contention that most of the excess deaths were caused by violence, and most of these by the coalition, simply doesn't follow from their own data.
Mike,
As far as splitting hairs goes, 17,000 is not "far lower" than 30,000. They're both well of the order of 23,000, say. You know how wide was the authors' confidence interval on their figures. Are you sure you're not splitting hairs yourself?
[Disclaimer: I haven't even glanced at the study]
Mike,Theoretically, any single study duplicating the Roberts et al. survey should produce a central estimate which falls somewhere between 8,000 - 194,000. (Sound familiar? ;-) Out of 20 such studies, at least one (5%) should fall outside that range. Of the other 19 (95%), there should be a preponderance of studies nearer the central number of 98,000 and fewer toward the extremes. That's all in a theoretical framework which excludes the Fallujah data, of course.But this is all entirely moot, because I doubt anyone is going to duplicate this survey design, let alone multiple times. It's clearly one of limited precision (possibly worse than was anticipated), and a pretty safe conclusion to be drawn from it is that a more useful national estimate will require a bigger survey using many more clusters, more widely distributed across Iraq.
I should have added that the theoretical framework also excluded systematic errors (such as in coverage or of measurement).
Mike;
Do you mind running through the inconsistancies in the study again? I haven't been following your arguement from the start.
You quote this section from the papers summary:
Interpretation Making conservative assumptions, we think that about 100,000 excess deaths, or more have happened since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths.
I think the confusion may come from the grammar, phrasing and proximity of the two sentences. The first sentence is fine - we know that they estimated 100,000 excess deaths. But I don't think the second sentence is intended to have any connection to the first. Violence did account for most of the excess deaths <u>in the sampled data</u> and air strikes from coalition forces accounted for most violent deaths <u>in the sampled data</u>. It's a statement about their data. It's not a statement of the breakdown of the estimate, which is something they don't touch on in the rest of the paper.
I admit this could be worded much less ambiguously.
Two more points:
"...the Lancet should have stuck with a straight risk factor argument concerning excess deaths".
I think this is a little harsh. I think the excess death estimates have been talked about a lot here, and in the news. But looking at the paper, almost all their work is done in terms of risk factors. There's only a paragraph and a couple of sentences dealing with the estimate, and that's just to put the risk factor into an understandable context.
"The authors say that their study implies at least 30,000 coalition bombing deaths occurred outside Falluja."
Where?
They don't say it in the paper. As far as I can see, in the paper they avoid any attempt to break their estimate down beyond excess mortality. I think the only part that can be read as touching on this is the one sentence in the abstract, and that's only due to unclear phrasing.
By the way, why do you believe the data is " fairly accurate," when learned defenders of the study such as D Squared are conceding that the breakdown of the 100,000 excess death estimate, the parts that make up the sum, do not coincide with the authors' written conclusions or appear to accurately reflect extrapolations from the data itself?
This is not a "concession" on my part; I have maintained from the outset that I do not like the (practically universal in medical statistics) practice of extrapolating risk ratios into numbers of deaths. As far as I can see, the authors' descriptions of the data are consistent with that data.
Mike, the Lancet authors are using the non-Fallujah data to get a conservative estimate for the increase in the death toll from all causes. They get the 98,000 figure with the huge error bars. They then look at Fallujah, and for the reasons I won't repeat, suspect that this sample represents a very large number of deaths in heavily bombed neighborhoods, so that the actual increase in deaths is probably above 100,000, due to all those tens of thousands of (perhaps hypothetical) bombing deaths in neighborhoods like the one they sampled in Fallujah.
This might well be wrong if the Fallujah sample was just some wild fluke, but it is a reasonable interpretation of their data.
Frankis:
"Are you sure you're not splitting hairs yourself?"
Yes, I'm sure. 17,000 is " far lower" than 30,000. It's such an obvious concept, I'm at a loss concerning what else to say about the matter. I think I'll just leave it at that.
Tribbs:
My apologies, but I'm not sure I follow what you're trying to establish with your last post. In any event, you don't seem to be agreeing or disagreeing with me, unless I've misinterpreted your words.
I do, however, have a different take on whether the study will be duplicated. I fully expect this will occur, and that we will see more than one subsequent study, with varying sample sizes.
Enquiring Mind:
I'm afraid I couldn't disagree more strongly. It's clear to me that the 100,000 excess death estimate in the first sentence and the subsequent sentence attributing most of the deaths to violence and coalition air strikes are connected. The second statement isn't a reference only to the study data.
If there were any doubt about this, study co-author Richard Garfield erases it in the interview Tim has linked in his initial post for this thread. In this interview, Garfield provides a breakdown of the 100,000 estimate, and in doing so attributes 57,600 deaths to violence, and claims that the majority of these were " attributed to air assaults.":
"In areas of Iraq, with the exception of the North, all had a rise in the mortality rate and most were due to violence. Real change was in deaths due to violence.[The majority of the 57,600 deaths due to violence was attributed to air assaults.]"
This 57,600 figure is consistent with the 60,000 violent death figure that has been commonly used at this blog throughout the Lancet debate, by Tim, Heiko, myself, and others.
"...the Lancet should have stuck with a straight risk factor argument concerning excess deaths".
"I think this is a little harsh. I think the excess death estimates have been talked about a lot here, and in the news. But looking at the paper, almost all their work is done in terms of risk factors."
Enquiring Mind, the excess death number IS the story, and that is what the authors intended. Expressing their results in the form of risk ratios would be meaningless to most of the public. The authors wanted their results to have an impact on public opinion, and using a numerical value for their conclusion was necessary to accomplish this.
Come on now, do you seriously believe the 100,000 estimate was "......just to put the risk factor into an understandable context."?
D Squared:
"This is not a "concession" on my part; I have maintained from the outset that I do not like the (practically universal in medical statistics) practice of extrapolating risk ratios into numbers of deaths. As far as I can see, the authors' descriptions of the data are consistent with that data."
Here's what you said earlier in this thread, when I mentioned again the bombing number discrepancy, and the fact that the bombing deaths were eclipsed by several other causes of death, both violent and non-violent:
"I think you have a point about the extrapolated numbers and the causes; I would say that this is an intrinsic problem of the technique used rather than a mistake made by the survey authors, but I'd agree that the numbers are not really worth much."
If " I...... have a point about the extrapolated numbers and the causes;" and you " agree that the numbers are not really worth much, " how does it follow then that you can now state " As far as I can see, the authors' descriptions of the data are consistent with that data,"?
I thought you had agreed long ago that the authors' conclusion that bombing deaths constituted the single largest cause of death for the 100,000 estimate was clearly out of whack with the data. It's half the number (6) in comparison to the number of violent deaths not directly attributable to the coalition (12). How is this consistent with the authors' explanation of the data? Have you changed your mind?
Donald:
You're wearing me out on this. The authors are on record that their 100,000 excess death estimate (or 98,000 if you prefer) is independent of the Falluja data. Here is the reference from the study itself:
"We estimate that there were 98,000 excess deaths (95% CI 8000-194,000) during the post-war period in the 97% of Iraq represented by all the clusters except Falluja."
Here's Garfield from the interview linked by Tim:
"One of the first clusters we had was in the town of Fallujah. [The deaths] were so much higher than in other parts of the country so we left it out of our analysis. But if you take just deaths that occurred, excluding areas characteristic of high conflict, we saw an excess mortality rate of about 50%, which is to say that all mortality rose 50% in the year after the war compared to the year before it."
Les Roberts has said the same thing in interviews. You keep insisting that the authors are utilizing the Falluja data as a factor for their 98,000 death estimate, when all the while the authors are unambiguous in telling us they're not. That's what " .... left it out of our analysis," means.
Please have a look at what I've posted above, concerning Garfield's estimate of violent deaths. The Falluja data has nothing to do with the 30,000 bombing death estimate that makes up part of the 98,000 estimate.
I don't understand your repeated reference to " huge error bars," when discussing this number. The authors are not implying in any way that their 98,000 estimate is somehow wonky because of " error bars " or their confidence interval. They believe 98,000 excess deaths have likely occurred in Iraq without Falluja.
Mike,
I think it may be that unless you can explain in what way one number is "far lower" than another you'll have less luck than you might otherwise deserve in your arguments here. But if you really believe what you've said (that it's simply an "obvious concept" when one number is "far lower" than another) then I'll stick my neck out and predict that you'll have no luck arguing with numerate people, sorry! Here, in this thread, they're not "concepts" they're "numbers", and they appear only in the context of a statistical study to which you've been objecting.
Some may think it laudable that you're willing to insist on the application of common sense and logic in a debate, without unnecessary resort to statistical methods and concepts. Things become though, at some point, simple enough already - it would be a mistake to attempt to further "simplify" them. A bit too simple would be close to becoming far too wrong.
If 17,000 is "far lower" than 30,000 in your remarks above, I wonder what would be an example of a number that you or I might call "nearasdammitzero"?
Frankis:
"I think it may be that unless you can explain in what way one number is "far lower" than another you'll have less luck than you might otherwise deserve in your arguments here."
Okay Frankis, I'll give it a go.
Let's say your boss comes to you one day and says " Frankis, we need to trim costs a bit, so I've decided to cut your pay. Instead of $30,000.00, you're now going to make $17,000.00. Since we're both numerates, you and I realize that $17,000.00 is not " far lower " than $30,000.00, so I'm confident you'll have no great difficulty with this."
Frankis: " But that's a 43% pay cut!"
Boss: " Frankis, I'm disappointed in you. Please dispense with the common sense and logic, and stick to the numbers. After all, it's not like I'm rolling you back to nearasdammitzero, is it?"
Frankis: "When I get home and tell the wife about this, my odds of surviving past dinner are about nearasdammitzero."
Boss: "Well Frankis, I guess you should have married a numerate, no?"
Frankis, hopefully you won't take offence to this bit of levity. I'm not mocking you or trying to be sarcastic.
I understand that you haven't read the study. The authors claim that the single greatest contributor to their estimate of 100,000 excess deaths is coalition bombing. Their estimate of deaths by bombing is 30,000. The data suggests 17,000. Within the context of a 100,000 overall total, a discrepancy of 13,000 for your #1 cause of death is a big deal. I don't think you would have difficulty making a statistical argument to support this conclusion, or an argument that is based on common sense and logic.
Mike,
Do the statisticians (numerate people would do) here who've read the study agree with you that the number should be 17,000, not 30,000? If so then they presumably agree that you've made that much at least of a useful contribution to a debate - kudos.
Are you also arguing that the difference between 100,000 people dead and 87,000 dead would be a big deal? Well ... I'm sure it'd be a huge difference to all of the 13,000 people who found themselves no longer dead. The 95% confidence interval from the study was something like 8,000-194,000, correct? And you're upset that a number like "100,000 excess civilian deaths" was being headlined, also correct?
Would you tell me then, on another tack, which would be the large number - 1,000 dead US troops or (say) 10,000 maimed ones? If 1,000 dead US troops is an unfortunate outcome then, no doubt, between 17,000 and 30,000 Iraqi civilians killed by US bombing would also be unfortunate (this would be leaving aside the "excess civilian deaths" of Iraqis from all causes consequent to the invasion, not resulting directly from US bombing, but the subject of this study). How much lower, in your estimation, are the US military casualties than the Iraqi civilian ones? "Far lower"? Apparently this would not be strong enough for you, as you believe that 17,000 is already "far lower" than 30,000. How low then are the US military casualties in Iraq relative to the Iraqi civilian casualties of bombing by the US, in your terminology?
The discrepancy that you claim, 13,000 in 100,000, would be comparable in proportion to the difference between 1,000 US military deaths and 870. Would 870 dead be "far lower" than 1,000 dead?
Mike;
I don't really disagree with you much.
The abstract is badly worded. This may not be the authors fault, all sorts of things happen in proofing. But, I think it clearly doesn't refer to a breakdown of the estimate - if it does why isn't this breakdown presented in the body the the paper? If it does refer to a breakdown, it's citing work that doesn't exist. That doesn't make sense. I'm sure the sentence must be intended to refer to the raw data, but something happen to botch the meaning. There is a difference between "estimated excess deaths" (in the population) and "excess deaths" (in the sample)
What the author said in an interview on a website doesn't bother me. It's not peer reviewed. I'm defending the paper, not anything the author shoots his mouth off about. My advice would be stick to the methodology of the paper and not try and breakdown the total excess death estimate.
About the excess death number. I don't follow d2's logic in thinking it shouldn't be estimated. There clearly is a number - the people who died over and above those expected from the rate that existed before the war. Why shouldn't an estimate be made of it? The estimate's a legitimate part of the paper. It's not all the paper. And I'm not defending any number of news articles that should have been written better.
Frankis:
"Do the statisticians (numerate people would do) here who've read the study agree with you that the number should be 17,000, not 30,000? If so then they presumably agree that you've made that much at least of a useful contribution to a debate - kudos."
I'm not seeing any serious argument to the contrary now. Some had been arguing that the Falluja deaths could have been used by the authors as grounds for extrapolating a higher bombing death rate for their excess death estimate that encompasses 97 % of Iraq other than Falluja. However, the authors have explicitly stated that the Falluja data was excluded from their analysis for purposes of extrapolating their 98,000 excess death estimate.
"Are you also arguing that the difference between 100,000 people dead and 87,000 dead would be a big deal?"
My primary argument is that this study, standing alone, cannot be represented as a definitive, accurate overall count of the excess deaths, or the subset composition of these deaths. Further study may prove that it is accurate, but may also prove that the original data is extremely inaccurate, and doesn't reflect the excess death toll accurately.
While some argued early on at this site that the study data and the authors' extrapolations should be considered accurate so long as the study methodology isn't discredited, it seems most here have come to the consensus that further sampling(s) is necessary.
As a result, and getting back to your quote to which I refer above, I think it's premature for an argument such as the one you present. That being the argument that, okay, if 100,000 doesn't work, why do you feel 87,000 isn't equally tragic? If we don't have conclusive numbers to begin with, hypothetical comparisons of the kind you raise aren't particularly meaningful.
"How much lower, in your estimation, are the US military casualties than the Iraqi civilian ones? "Far lower"? Apparently this would not be strong enough for you, as you believe that 17,000 is already "far lower" than 30,000."
Of course, American military deaths are " far lower " than civilian deaths. That is how I would characterize such a comparison. I don't see you point here.
Enquiring Mind:
I don't know whether to scream in frustration over your antics, or laugh myself silly.
You start out by claiming "I don't really disagree with you much.", then proceed to vehemently disagree with everything I said!
Your arguments are simply stubborn, contrarian rear guard actions.
The abstract isn't "badly worded," as you suggest. What will it take to get you concede that this is what it is?
Garfield and Roberts state unequivocally in their interviews that the Falluja data was excluded from their analysis for purposes of calculating their excess death estimate. How in the hell does the question of " peer review " provide an excuse to dismiss what Garfield said in the interview? It's an interview! If you can provide some evidence that Garfield was misquoted, or that the entire interview was a fabrication, fine. Otherwise, he's confirmed that the authors are claiming that approximately 60,000 of the 100,000 excess deaths were caused by violence, and that the majority of these were caused by coalition bombing. That's why this statement from you is beyond ridiculous:
"I'm defending the paper, not anything the author shoots his mouth off about."
Do you realize what you've just said here? You're defending the author's paper, against the author himself, because he's saying something you don't like! I've got some news for you Enquiring Mind, it's Garfield's study, not yours. For God's sake, the author didn't " shoot his mouth off," he repeated and elaborated on a conclusion of the study that is in the study text itself, the one you keep trying to claim doesn't say what it clearly does.
"My advice would be stick to the methodology of the paper and not try and breakdown the total excess death estimate."
The total excess death estimate is comprised of the sum of the breakdowns of death subsets from the data. The overall estimate is the calculated from these death subsets, and this practice is part of the methodology of the study. If you believe the overall estimate is accurate, then you should have no objection to any scrutiny of the subsets. The two are not independent from each other, they are unavoidably linked.
By the way, were you following your own advice about sticking to the " methodology of the paper," when you engaged in this bit of speculation concerning heart attack mortality as presented in the study?:
"You then engage in speculation about the causes of death. You have not cited any evidence that your ideas are based upon. It's entirely possible that deaths rose among men and children because these two groups had stress elevated most by the war, and it's entirely possible that children may have congenital problems but these only kill them in certain circumstances."
Come on enquiring mind. Let's cut this out.
Mike,
The actual argument I was making is not as you have it above, but that in a difficult yet worthwhile study of this nature the uncertainties must be, unavoidably, great. Of course the authors have provided a statistical analysis that tells us just that (I'm sure). If you've fairly identified an error in an estimate they've made that's good, but there's no future in your wish for "conclusive numbers" -
they're simply not a possibility and I'm sure you know that.
You called 17,000 "far lower" than 30,000, yet I didn't think you'd call a number like 1,000 American deaths "vanishingly small", or anything like it. This seemed to leave you running a bit short of adjectival resources.
You say "Of course, American military deaths are " far lower " than civilian deaths. That is how I would characterize such a comparison. I don't see your point here". The point was only this: if 1,000 is "far lower" than 17,000, and if 17,000 was and perhaps still is "far lower" than 30,000, then meaning seems to be being lost in there, somewhere. That kind of meaning is what the (difficult, in some cases) statistical analysis attempts to provide.
The par you've writtten above under "My primary argument ..." seems fair and reasonable to me.
Hi Frankis,
I don't think the study provides any substantial information we didn't already have before. In that respect, I don't think it was a worthwhile exercise at all.
Ex violent death, the evidence is far too weak. For violent death, it's as imprecise (several ten thousand in all likelihood) as our previous knowledge, and where attribution of those violent deaths is concerned, the study's results are just inadequate.
We want to help Iraq and in doing so we need good information, like where are we going wrong, and what are the most urgent problems that need addressing.
And I think security wise, it's general crime, kidnappings, IED's and small arms fire that are the biggies, not aerial bombardment by coalition forces.
The study is quite unhelpful, except, if you wish to make the point that "things are worse than before and so I was right in opposing Bush's reckless policy of regime change".
When the question is one of how do we do better now, I think it is downright misleading in identifying coalition bombing strategy as the major area that needs improvement.
Hi Heiko,
As I haven't read the study I think I'll just thank you for your interesting response to my comments, and bear your opinions (and Tim's, Mike's, D2's, and respected others' of course) in mind.
Regards