Lott on the Lancet study

This is really weird, Tim. Lott quotes the entire section from the New York Times that you quote, plus he quotes some criticisms by the Times. Your discussion is completely misleading because it makes it appear as if Lott wasn't accurately describing the discussion in the Times. Lott quotes the following:

"Three weeks ago, The Lancet, the British medical journal, released a research team's findings that 100,000 or more civilians had probably died as a result of the war in Iraq. The study, formulated and conducted by researchers at the Bloomberg School of Public Health at the Johns Hopkins University and the College of Medicine at Al Mustansiriya University in Baghdad, involved a complex process of sampling households across Iraq to compare the numbers and causes of deaths before and after the invasion in March 2003.

"The 100,000 estimate immediately came under attack. Foreign Secretary Jack Straw of Britain questioned the methodology of the study and compared it with an Iraq Health Ministry figure that put civilian fatalities at less than 4,000. Other critics referred to the findings of the Iraq Body Count project, which has constructed a database of war-related civilian deaths from verified news media reports or official sources like hospitals and morgues.

"That database recently placed civilian deaths somewhere between 14,429 and 16,579, the range arising largely from uncertainty about whether some victims were civilians or insurgents. But because of its stringent conditions for including deaths in the database, the project has quite explicitly said, ''Our own total is certain to be an underestimate.''

"It has refrained from commenting on the 100,000 figure, except for noting that such a number ''is on the scale of the death toll from Hiroshima'' and, if accurate, has ''serious implications.'' Certainly, the Johns Hopkins study is rife with assumptions necessitated by the lack of basic census and mortality data in Iraq. The sampling also required numerous adjustments because of wartime dangers -- and courage in carrying out the interviews. Accordingly, the results are presented with a good many qualifications."

uh bob, you want to continue on from where you stopped, or must we do it for you?? (BTW, Tim did provide the link)

But let us note that the bottom line in that the Hopkins team did the best job possible (according to the Times section which you truncated) with the available resources and information. They also note that the US could do a better job at any time it wishes, but chooses not too. What does that tell you.

***************************************************

Ultimately, the researchers are saying that these are the best estimates available and that better ones could be obtained if the occupying forces and the Iraqi authorities wanted them.

"This survey shows that with modest funds, four weeks and seven Iraqi team members willing to risk their lives, a useful measure of civilian deaths could be obtained," the researchers wrote in The Lancet. "There seems to be little excuse for occupying forces to not be able to provide more precise tallies" that could be confirmed by independent bodies like the International Red Cross or the World Health Organization.

What is Washington's response to this argument? The dismissive statement by the head of the United States Central Command, Gen. Tommy R. Franks of the Army, that "we don't do body counts" has been repeatedly quoted as more or less the final word on American policy.

*********************************************

Bob you are beneath contempt as is General Franks

Tim:

I'm the guy you were debating at Daimnation, and you never got back to me in relation to my questions concerning your defence of the Lancet study. I'm basically re-posting most of what I stated in my last comment, and am wondering if you have any response to the flaws I see in this study.

What do you say about the obvious effect on accuracy that is created by the very low violent death toll from the non-Falluja clusters? (only 21 deaths) If a separate sampling, still using the same methodology, came up with 6 violent pre-war deaths and only 10 violent post regime change deaths, what would that do to the findings and conclusions we currently have from the Lancet piece? Is such an outcome of a separate sampling that unlikely? The 21 figure is simply too low to resist dramatic manipulation, and not only from additional surveys, but also as I illustrate with the authors' subterfuge concerning insurgent fatalities, insurgent-caused fatalities, and the lack of elaboration in the 12 deaths not caused by the coalition.

Do you have any defense for the glaring hole in the study's credibility that arises from no death in the study being categorized as an insurgent fatality? What would this do to the impact intended by the authors on the non-Falluja death toll of 21, if several, even many of the 15 men involved, were insurgents?

How do you reconcile the study's conclusion that only 3% of violent deaths in Iraq are at the hands of insurgents?

Why are we not allowed to know how many of the non-Falluja deaths fall into the category of the 12 violent deaths that had nothing to do with the coalition? I'm going to go out on a limb here, but I think it's a thick one, and I'm going to suggest that the reason we aren't told this is because most of these deaths were from the non-Falluja sampling.

How do you justify the authors' attempt to convince readers that "Most individuals reportedly killed by coalition forces were women and children," on the basis of only 6 such deaths involving women and children among the entire non-Falluja sampling? What do you say about the obvious dishonesty prepetrated by the authors in making such a statement, while at the same time inviting the reader to selectively ignore the Falluja cluster numbers? 6

I didn't mention in my first post, because of my haste in preparing it, but another factor that worked very much in favour of the authors' agenda was their sampling identifying only 1 violent death occurring during the pre-war time period. In the same way that the low number of violent deaths AFTER regime change is prone to dramatic shifts if new, identical cluster samples were undertaken, the number of 1 is obviously so vulnerable that it can be of no use in a study that seeks to extrapolate to nation wide totals.

To illustrate this better, let's consider your home country, Australia, since you've also incorporated Australian death figures into your attempted rebuttal of Bolt. Let's say we did a random cluster sampling of the same number of homes and individuals in 33 separate neigbourhoods across Australia, for Australian deaths of all types and also violent deaths over the same time frame as the Iraq sampling.

Is it possible that we could come up with 2 violent Australian deaths among these 7,800 people, for the same time frame as the pre-invasion period covered by the Iraq sampling?

Don't assume this is so far fetched. After all, by your own admission, the crude mortality rate in Australia is 50% higher than that of pre-regime change Iraq, if we believe the 5 in 1000 crude mortality rate the Johns Hopkins people are trying to sell. So it isn't out of the realm of possibility that a cluster sample of this many people would turn up 2 violent Australian deaths. (By the way, please explain to me how the crude mortality rate in Australia can be that much higher than Iraq's, while the life expectancy figures indicate Australians live approximately 15 years longer than the average Iraqi. Frankly, I see no possible way the 5 in 1000 crude mortality figure has any prospect of being accurate).

Sorry for straying off there. So if we have 2 violent Australian deaths during the same 14 month period when Iraq had 1, does that indicate Australia was twice as violent as pre-war Iraq? Of course not.

Moreover, we know this isn't true, and here's one example of how silly these figures are as any indication of nation wide numbers.

Iraq Body Count, a group that obviously doesn't approve of regime change, released a press statement in September 2003, claiming that an extra 1,500 violent deaths had occurred in Baghdad in the 4 1/2 months following Saddam's overthrow. In making their case, the group estimated that the normal, pre-invasion violent death toll in Baghdad for that time period would be approximately 1,300. (And that's just Baghdad).

I don't know where IRBC got their pre-war figures, these seem to be virtually impossible to find, and one can see why, given the terror state run by Saddam didn't lend itself to statistical gathering of any kind, much less accurate statistical gathering.

Still, if we accept that pre-war figure, that means approximately 3,500 persons a year in pre-war Baghdad (not Iraq as a whole) could expect to meet a violent end. Tim, the murder rate for all of Australia has never topped 400 in a year! I realize " violent deaths " encompasses more than just homicides, but I think you get my point. Coming up with only 1 pre-war violent death in a place as violent as Iraq was clearly an early Christmas present for the Johns Hopkins authors, and they were only too happy to exploit it.

Tim, this is a death study, correct? What is the intent of a death study other than to gather and examine statistics involving actual deaths? I've been harping on the violent death figures contained in the study, but there is a bigger number in play here, and that is the number of DEATHS IN TOTAL AMONG THE SAMPLING, FROM ALL CAUSES. The Falluja cluster accounts for ALMOST 40% OF ALL DEATHS FROM ALL CAUSES. Tell me Tim, what type of death study asks the reader to ignore 40 % of the deaths included in its own data? On top of that, the authors ask us to ignore 75% of the violent deaths (when it suits them) from their own data. How can it follow that their conclusions are reasonable?

An increase in violent deaths from 1 to 2 would not be statistically significant, but there was an increase from 1 to 21, (not counting Falluja) which is statistically significant.
If you read the study, you'll find that they usually give confidence intervals for their numbers -- that's where they account for the possible errors caused by small samples.

The 3,500 violent deaths per year in Baghdad pre-war included accidents, so the corresponding number in the study is 1 violent death + 4 accidents = 5 deaths. What would we predict from the Baghdad numbers? 3,500/5M (Baghdad pop) * 7500 (people in pre-war sample) = 5.25. That's pretty close , isn't it?

Your last paragraph is just silly. Now you are complaining because they excluded Falluja?

Oh and the explanation of why Australia has a higher crude death rate than Iraq is here

Tim:

I've now asked you 3 times (twice at Daimnation, once here) for some sort of explanation/defence for the Johns Hopkins authors' deception and dishonesty in relation to the selective use of the Falluja cluster data, the failure to reveal how many of the 12 non-coalition-caused deaths are from the non-Falluja clusters, and the ridiculous assertion that the deaths of 6 women and children can be extrapolated to conclude that women and children accounted for the majority of casualties among the 100,000 or more non-Falluja total. Again I ask you, how can we take an Iraq death study seriously that refuses to categorize a single death as that of an insurgent, and creates the belief that insurgents are responsible for a miniscule percentage of the deaths in Iraq?

You also offer no rebuttal to my point concerning the volatility in the low number of non-Falluja violent deaths. Yourself and the others who defend this study would simply have no choice but to equally defend a second survey using identical methodology, if it resulted in the figures I offered, say 10 post war violent deaths and 6 pre-war such deaths. Again I ask, can you come up with any reason why such numbers couldn't be derived from a second study? As I said at Daimnation, a second survey could halve, quarter or double the post war violent deaths of the first survey. The low death figure is ridiculously vulnerable to fluctuation in the context of Iraq.

My point concerning an Australian survey coming up with a pre-war time frame violent death figure of 2, versus the 1 figure from the Iraq survey, was used to illustrate that the Iraq figure is utterly useless as a benchmark for comparison to a post-regime change violent death count. It is further evidence that a second survey could result in a much higher pre-war violent death figure. I don't believe for a moment that Australia was twice as violent as Iraq during the 14 month pre-war period cited by the study, but if you're going to defend the pre-war figure from the actual study, how can you dismiss a conclusion that a separate study in Australia could result in a doubling, even tripling or quadrupling, of the Iraq total of 1?

My last paragraph is anything but " silly," as you suggest. If you'll recall my first post at Daimnation, I complained that the authors weren't being compelled to defend the number of deaths their study actually extrapolated nationwide, which is in the area of 300,000 with the Falluja data included.

What is " silly " is the ludicrous defence of a death study that disavows at times, and selectively includes at others, almost 40% of the overall deaths from its own data, and 75% of the violent deaths from its data. This is Monty Pythonesque in its absurdity.

Two final points, concerning those you make on the subject of the IBC press release and the crude mortality rate.

First off, where in the IBC press release or site did you find a reference that four in five violent deaths in Baghdad pre-war were " accidents?" I can find nothing of this nature. I'm not saying your incorrect, I'd just like to see it for myself.

As for the crude mortality rate, I've read what you've posted earlier from your link, yet I profess to being unable to reconcile a substantially lower crude mortality rate vis a vis Iraq and say, Australia or the U.S., and the approximate 15 year difference in life expectancy rates. My lay person knowldege base on the subject has always included the belief that such a difference in life expectancy rates is beyond huge, and betrays a much higer death rate in the country with the lower life expectancy figure.

Thanks.

It's fascinating to me how some people are shocked, SHOCKED that the most powerful military in the history of the planet spends US$200B and, as a result of this expenditure, kills people.

Mike, as a taxpayer you should want an accounting of where your family's taxes are going. Fer chrissake, $200B of bombs and bullets down the toilet and there aren't tens of thousands killed? What the heck kind of military is that?!? What are militaries for?

Our military, judging from past orders, isn't there to protect against looting, isn't there to secure the populace, so what are they doing?

Jus' wonderin', is all.

D

The authors may have been successful in receiving confirmation in 81% of cases where they asked, however, they didn't always ask and so only got 63 confirmations for a total of 188 deaths. What they don't say is, which deaths they asked confirmation for, more specifically, how many of the deaths in Fallujah did they verify? I'd guess none (and for understandable reasons, they had reason to fear for their lives, and not from coalition bombs ...).

A couple more points:

1. I noticed the following bit on Medialens "We did not count disappearances as deaths." Nor would they have counted deaths in prisons. The 5000-10000 people per year that are estimated to have been executed by Saddam just before the invasion would not have been picked up by their methodology.

2. You can't blame just one side for every single death. Japan may have been the aggressor in WWII, yet do we try to pin blame for nuking Hiroshima on them?

One way of looking at the situation is by saying that most of what's wrong in Iraq today is the fault of the "insurgency"/Baathists/islamist terrorists, ie Saddam's legacy.

And Dano, the military is paid to kill terrorists. Few people mind that, they do mind them killing innocent children needlessly and they are downright angry with people who behead teachers, children, aid workers or just about anybody who wants to help Iraqis.

Or in other words, the so called "insurgents" who were "defending" Fallujah (and who made it abundantly clear that they wouldn't be satisfied with anything less but complete domination and subjugation of the Shiite majority).

Hi Mike,

I agree with a lot of what you say; there are a number of issues that Tim is clearly unhappy to address.
For example, when he glosses over your point that the authors exclude (for the headline number of 100,000) or include Fallujah (as a justification for - most of those killed were killed coalition bombing and most of those were women and children -) as it suits them, he replies with this non-reply:
"Your last paragraph is just silly. Now you are complaining because they excluded Falluja?"

The trouble with this study isn't the "science" or "methodology". It's the presentation of the results. What they've found good evidence for adds little to what we knew already, several ten thousand Iraqis have likely died from violence after liberation, while in the last few years under Saddam there was no mass murder or massive fighting going on (and because they don't count disappearances and didn't account for the prison population, the methodology didn't account for most of those executed by Saddam, a figure other sources put in the 5000-10000 per annum range).

They do provide evidence that the infant mortality rate may have risen, but this is much weaker, due to a number of factors (the low number of infant deaths involved, the potential for selective recall bias known from other studies, the conflict with Unicef estimates for 2002 that isn't properly addressed, the fact that their own explanation for the connection of the supposed increase with the war - three infants killed by coalition bombing [in Falluja?] and three home births brought about by security concerns - is rather different from Tim's attempt to explain it all by reference to nutrition) and that a large number of children may have been killed by coalition bombing in Fallujah, which also appears weak, as they are unlikely to have asked for death certificates, may have been lied to and may have picked an unrepresentative cluster even within Fallujah.

But how does this get presented to the media (right before the election)? "100,000 civilians killed", "by coalition bombing", "most of those killed women and children". "Much worse than Iraqi body count" (which itself isn't reliable by the way with no on-the ground verification of deaths, or whether these deaths were truely innocent civilians killed by coalition forces).

The crude mortality rate for Iraq was estimated as 8 by WHO for the year 2002. However, to address your last point, Iraq's population is very young (due to high birth rates). Therefore, there are few old people, and even with a lower life expectancy,you'd ordinarily expect a lower death rate than in a country like the US that has many old people. To simplify, if nearly everybody in the US was over 70, even with an average life expectancy of 75 you'd expect an awfully large number of deaths per year. Even if life expectancy was 30 in Iraq, you wouldn't necessarily be surprised to find very few deaths, if nearly everybody in Iraq was aged between 1 and 20.

And Dano, the military is paid to kill terrorists. Few people mind that, they do mind them killing innocent children needlessly [...] and they are downright angry with people who behead teachers, children, aid workers or just about anybody who wants to help Iraqis.

Exactly to the first part before the ellipsis. That's why the numbers must be debunked by the enablers and wargasmers.

No one can know the military kills lots of innocents too. The amount of money spent to publicize all the smart high-tech weapons will have been wasted if that happens.

See, in the sales job that led up to the war, we were duped into believing that the streets would be lined with rose petals under our brave soldier's boots, and the pretty films with high production values we watched showed precision weapons only killing bad guys.

Anything contradicting that load of hooey might hamper the future sales pitch for the next war.

As for the quoted part after the ellipsis: gosh, really?!?!?!?! Wow. I had no idea.

I wonder what would happen to the administration's highly burnished luster if the military were seen to be killing lots of civilians in addition to killing the bad guys. I wonder what that would do in the public's mind...hmmmm...golly, again I have no idea.

Best,

D

Heiko:

Thanks for the clarification concerning the crude mortality rate. I can't pinpoint why, but I confess to having a difficult time getting my head around the concept. It is clearer now, thanks.

Thanks for the support here, I'm wondering about this statement from you:

"The trouble with this study isn't the "science" or "methodology". It's the presentation of the results."

I don't think there's any question (and you and I are certainly in agreement) that the authors intended this study to be polemical, and that is exactly what they have produced. I've used the term before, but " dishonest " is the most accurate description of the manner in which the authors have framed and presented their results.

However,I still maintain their methodology and science is part of the problem as well, simply because it produces statistics that do not appear to be reliable. I'm not saying the authors utilized flawed methodology and science. I'm sure their work was conducted in a way that meets accepted standards for conducting such surveys. My disagreement on their methodology lies with my belief that a minute, random snapshot type survey is not capable of providing reliable data that can be extrapolated into nation wide totals on an issue as complex as this. My key argument in relation to this is the point I made concerning the vulnerability to dramatic manipulation of a figure as low as 21 (the non-Falluja violent death figure). As I describe in my posts here, this number is prone to serious revision if a second sample was taken using identical methodology. Any such revision arising from a second sampling, in my view, eclipses all other complaints concerning the accuracy of the actual survey, and destroys any defence of the original survey.

What's your take on this?

So Mike, you agree that they have followed the standard methodolgy for surveys, but argue that surveys cannot provide reliable data. Basically you are rejecting the surveying as a technique. Your key argument is wrong. If you repeated the study with the same methodology, you would be unlikely to get exactly 21, but the number would likely be 21 +/- 9. If you don't believe me, you can easily conduct some experiments with a spreadsheet. Enter the formula =IF(RAND()<0.21;1;0) into a cell. That gives the value 1 with a 21% probability. Copy it into the 99 cells below and add them all up. The result probably won't be 21, but it is very unlikely to be a half or a quarter of 21. I copied the column ten times and got these totals: 23 21 23 26 19 27 25 22 22 21.

Heiko, could you explain the methodology used to produce the estimate of 5000-10000 per year executed by Saddam?

Hi Dano,

we are in perfect agreement that the amount of suffering among innocents is enormously important.

As in medicine, a cancer is only worth removing, if we are likely to help the patient in the process.

That's why we aren't doing regime change in Pakistan, Iran, China, Cuba or North Korea. The cancer isn't as serious as in Saddam's Iraq, and the collateral damage could be much worse (eg in the case of China, hundreds of millions dead).

A quick comment also on the "we don't do body counts" statement. Madeline Albright made a big mistake, when she veered off message just once by not saying "Sanctions save lives by preventing Saddam from causing even more misery, both to his own population and neighbouring countries. The blame for the deplorable state of child health in Iraq lies entirely with him." And instead allowing through a non-denial "Yes, it's worth it.", when asked whether sanctions are worth 500,000 dead children.

The military's aim is to take out the bad guys while doing their utmost to protect innocent life. They also need to communicate that across.

Hi Mike,

I agree the number of deaths ex Fallujah is very low and that that makes them prone to "manipulation" as you call it. After all, just one family lying about a pre-war death, while making up two post war deaths has a massive impact, when there is only 1 pre-war violent death recorded, but 21 post-war.
Not to mention, that for attribution to causes, division into innocents/terrorists killed, the numbers become really small, eg violent deaths of children (4) and women (2) ex Fallujah.
Furthermore, you have to assume that the likelihood of dying is fairly evenly spread across the population. This is important in for example the case of Basra. They didn't survey it. But Basra could conceivably have been the site of considerable killing of Shiites by Baathists (before, during and/or after the invasion).
So, they could have, in theory, overlooked say 5 pre-war deaths in the Basra area.

Yes, the methodology is not very reliable for violent deaths, but the authors have a good defense in that they did as well as they could on that front (though one does wonder why they thought Fallujah was safe enough to survey, but Basra wasn't).

However, in spite of all this, the numbers for violent deaths ex Falluja (1 to 21) are so overwhelming that they do make an increase in the violent death rate in the sampled population very likely (ie not all of Iraq and excluding army barracks and prisons, these are important exclusions, as it is rather conceivable that the violent death rate in prisons, army barracks or Basra could be rather different from other subgroups).

How likely we can't say just from the raw data in the way Tim tries (it'll be 21 plus minus 9 in all likelihood). It's feasible that virtually all of Saddam's killings outside of prisons, army barracks etc., occurred in just a few locations, none of which were sampled. In principle, a million deaths could be overlooked in that fashion (if he had nuked Basra in 2002, the survy wouldn't have picked it up).

I do think that a different choice of clusters might produce 5 pre-war and 10 post-war deaths. For example, a single family in Basra could have reported five of its sons killed by Saddam's Fedayeen.

There is clustering in two senses, the general location and the family unit. These deaths aren't independent from each other. If a house gets bombed, several family members can die. If one family member is killed by Fedayeen, the likelihood of another family member (or a neighbour) being killed rises compared to the population in general.

Having said all that, the deaths appear evenly enough distributed in time and space to me that it looks to me (taking other evidence into consideration) like they are onto a real increase that chance won't easily make go away.

To put this a different way, their findings for violent death are reasonably consistent with other evidence and to me appear to be reasonably solid, even taken all the quibbles into account.

Hi Tim,

the 5000-10000 kind of range of other sources was from memory and rough, and I am sorry, should I have inadvertently given the impression that I was reporting the findings of a specific study, rather than my impression of what the general consensus was, roughly.
http://www.jewishvirtuallibrary.org/jsource/anti-semitism/hriraq02.html

Here a claim is made that: 'In April the U.K.-based Guardian newspaper reported that Lieutenant Colonel Mohamad Daham al-Tikriti, a recent defector from the General Security Service, admitted that in February 150-200 civilians were killed "at random" on suspicion of conspiracy and buried in a mass grave near Baghdad as part of a larger effort in which 1,500 civilians were summarily executed in the first 2 months of the year.'

I was also aware of the following study:

http://www.phrusa.org/research/iraq/release_032304c.pdf

They report that nearly half of the people surveyed reported serious human rights violations in the period 91-03, 30% of which are reported as having occurred between 2000 and the first six months of 2003. Nearly all of those violations are blamed on Saddam's regime (in 95% of cases, blame was put on groups affiliated to the Baath party).
They include (in a sample of 16520 individuals):
out of a total of 1018
212 cases of torture
210 killings
166 cases of separation and disappearance

The locations of these abuses were:
408 (in the home)
93 (prison)
92 (torture centre)
58 (military)

For the governorates considered (a population of just over 3 million), a yearly total of nearly 17,000 serious human rights violations results (my own calculation from the author's data, as are the following numbers). If that were extrapolated to the Shiite fraction of Iraq's population, a total of 80,000 serious human rights violations per year in the period 2000 to first six months of 2003 would result.

The study doesn't say, how the abuses split down for particular periods of time. It's possible that between 2000 and 2003 the figures are more reflective of torture rather than killings. If we just take the average, we'd get nearly 20,000 killings per year among Shiites in the latter period, which seems too high to me. Therefore, I assume that the other serious human rights violations must have figured more strongly in the mix in that period.

It's also noteworthy that the period from 2000 to 2002 was significantly worse than the period between 1994 and 1999, while the worst abuses (50% of the total) occurred between 91 and 93 (this may, however, be due to selective recall: killings of adults may have pre-dominated between 91 and 93, explaining why people did remember them, while afterwards other human rights abuses, more prone to selective recall, may have dominated, and the post 2000 increase compared to 94-99 may therefore be spurious).

Why aren't you as sceptical about this other study as the Lancet one? They both used clustered sampling, and while the Lancet confirmed 81% of the deaths they asked about, the other study confirmed none of them.

Heiko, Its time that you learned a little about how the world works. You clearly divide the world's governments into the lexicon of "officially designated bad guys" (ODBG; mostly non-aligned country's who don't play ball according to U.S./U.K. corporate rules) and officially designated good guys (ODGG, who are partners in the transparent 'war on terror'). Many of the latter countries are using the WOT to wage wars on terror against their own populations, with full support from Washington and London. These countries include Turkey, Algeria, Russia, China, Uzbekistan, Nigeria and Colombia. The leader of Uzbekistan boils his opponents alive in water; for this brutality his regime has seen increased support from the Bush and Blair governments over the past few years. The U.S. and U.K. supported Suharto almost until the very end of his regime. He was one of the world's biggest mass murderers and torturers of the second half of the twentieth century, certainly as bad as Saddam Hussein, but as long as he was "our kind of guy" (to quote Bill Clinton) the brutal nature of his regime could be overlooked. Similarly, our support for Obesanjo's regime in Nigeria and Uribe's in Colombia reeks of hypocrisy with respect to human rights.

Most importantly, the U.S. and U.K. continue to support vile regimes that support western business and military interests. This has always been the priority of western foreign policy, with civilian lives totally unimportant. As Dano said, the U.S. has spent close to 200 billion dollars on the latest Iraq fisaco in order to ensure that a pro-western government (read: iron-fisted regime) comes to power. The model for Iraq 'democracy' will be Honduras, El Salvador or Guatemala - banana republics and oppressive quasi-democracies that support western conglomerates.

Let me start off by making sure I don't commit the same error as Madeline Albright, a non-denial to a poorly phrased question.

Both studies have a number of findings, eg the Lancet study reports a certain number for infant mortality for the pre-war period. In both studies, each specific finding deserves a particular level of, if you like to use the term, "scepticism".

I am very sceptical about the infant mortality rate findings, for example, where, in fact, they did not ask for any death certificates.

I am also extremely sceptical about the findings for Fallujah. What I think they may have done (for understandable reasons) is to ask for certificates, when there was no need and one could take people on trust, and chose not to ask, where there were much better reasons (particularly in the case of Fallujah's reported 24 child deaths) to question the reliability of people's accounts.

Hi Jeff,

yes my view of "Western" motives is rather different from yours. I see them as largely altruistic, and where this isn't strictly the case, I think "Western" interests largely co-incide with those of other nations.

I do think that we are genuinely interested in a democratic and prosperous Iraq.

I also think that in the cases, where you claim we are supporting dictatorship, we are really helping the people, for example in the cases of China or Pakistan.

We cannot invade China without causing unbelievable suffering, I mean the kind of damage that WWII would start looking minor compared to. So the best thing to do is to invest and trade with China, which makes the Chinese better off, and helps us too. Not to mention, a wealthier and more educated Chinese population is busily making the place freeer in other respects as well (freedom of expression for example).

Finally, I think the elections in Iraq will bring moderate Shiite groups into power who will ask American troops to leave, not instantly, but according to a reasonable time table. I also think that Iraq's oil industry will remain nationalised.

Mike,
"I don't think there's any question ... that the authors intended this study to be polemical, and that is exactly what they have produced."
Of course, the rest of us already knew that you (and Heiko) were reasoning backwards from conclusion (the study must be flawed) to arguments. I just had no idea that you were aware of it as well. How curious.

Wu

Hold on a minute! If the Lancet study obtained death certificate in about 85%(?) of cases, then that means these bodies passed through a morgue or a hospital, which are covered in the IBC statistics. Therefore, all we have to do to find out the true civilian death toll is increase the IBC figure by about 15%. That puts the figure at around 19,000. But we also have to subtract 50% of the increase, because that's the amount of combatants said to make up the Lancet figure. That puts the total number of civilian casualties of the war at about 17,000, only slightly above the IBC figure. There we have it - solved.

Tim:

"Basically you are rejecting the surveying as a technique. Your key argument is wrong. If you repeated the study with the same methodology, you would be unlikely to get exactly 21, but the number would likely be 21 +/- 9."

I disagree with your logic. My argument is that a survey of this type is not accurate for purposes of extrapolating nation wide death tolls in an environment such as Iraq. Even ignoring the many problems raised by other critics of the study (Inaccurate and unreliable pre-war census data, lack of death certificates for some deaths, recall bias, etc), this methodology is most flawed because of the low number of violent deaths recorded. Heiko has made an eloquent case in illustrating the likelihood for dramatic shifts in the pre-war and post-war violent death numbers, depending on where the sampling occurs. The post-war deaths, as he notes, are not evenly spread across Iraq, nor were the pre-war fatalities, however getting accurate data on the pre-war carnage of Saddam is virtually impossible.

In my view, 21 +/- 9 is an unacceptable margin even if there could be reasonable confidence in it, and I don't see that there can be. You're talking about a near 50% margin of error. If a second sampling revealed a figure at the low end of the margin for error (12), and was paired with an increased pre-war figure during the same sampling, this would completely destroy the validity of the results of the actual Lancet study.

I don't see any way around the problems created by such a low death figure as a basis for the conclusions the authors make. I mean, more clusters than not in this survey recorded no violent deaths at all. None outside Falluja, it appears, recorded more than 2.

By the way Tim, any response to all the other questions I pose, concerning the dishonesty of the authors in the presentation of their conclusions? I'm also wondering about your IBC reference concerning the 80% accidental death rate among pre-war baghdad violent deaths.

Umm, no. The Lancet study measures all deaths. The IBC does not.

I think the Lancet authors should give the breakdown in the 21 non-Fallujah deaths, but one statistic they didn't give in their paper which was mentioned in a Medialens comment was that 5 of their 33 samples showed deaths from air strikes. I think that should have gone in the paper. even with big error bars attached.

Heiko, your willful naivete about Western motives is depressing. Go look up Barton Gellman's June 23, 1991 article in the Washington Post about American bombing directed at Iraqi infrastructure. You'll Pentagon spokesmen explicitly saying that they wanted Iraqi civilians to suffer and that sanctions would help prevent repair. in hopes that it would put pressure on Saddam. It's the same tactic the insurgents are using now. Or go look at some other examples in recent American history (some of them involving the same people currently working for Bush), where the US supported mass murderers and lied about their crimes. I don't think it is correct to always assume the worst about American government motives, but in some cases the evidence is quite damning.

There seems to be a general assumption that you believe in this study if you dislike the war, and vice versa.
This is a serious error. The science is that the data supporting the article must stand on its own two feet, and make a coherent and convincing story on its own. This is the crux of the matter.
As evidence goes, this Lancet study is weak. Large numbers of their data (deaths) were not checked at all, and of those that were checked, 19% could not be verified. The sample is subject to recall bias- as the authors note, cluster sampling in a non-homogenous population, and the authors have also committed the cardinal sin of post-hoc data manipulation.
You cannot rely on this dataset, as anything other than an indication.
It is deeply depressing to see people arguing that this paper shows that the war is immoral, or that it wasn't. The public policy that underlies a war just can't be attacked on the basis of a Lancet article that isn't particularly convincing.
J

Mike, your charges of dishonesty are ridiculous. If you want to accuse them of dishonesty, why not just say that they made up all the results and be done with it? If you look at random numbers I got, you'll see that not once does the number double. Random variation means that you might see a 50% increase, which would not be statistically significant. But violent deaths increased by a factor of 21. That is not going to happen by chance.
The Lancet study reported 4 accidental and 1 violent death before the invasion. That total of 5 deaths is what corresponds to the morgue numbers for pre-war violent deaths.

Heiko, both studies use similar methodology, but the Lancet one is clearly more reliable, since they verified deaths and had a shorter recall period. Why is it that the Lancet study is the only that get criticized?

I'm perplexed by your comments on infant mortality. I thought your argument was that the rate they found was too low. How could not asking for death certificates for infant deaths by realted to this?

"But violent deaths increased by a factor of 21. That is not going to happen by chance."
this is of course a wild exaggeration, and a claim which the study authors did not make. With the small numbers involved, and the fact of cluster sampling, this is entirely possible by chance, even if the "chance" isn't big.
as always, "they verified deaths" is shorthand for "they attempted to verify a small number of deaths, and failed to verify death in 19% of these cases".
And the general point isn't that impressive anyway. Small and badly controlled epidemiology studies aren't worth the paper they are written on, no matter which politically convenient viewpoint they support.
J

Heiko,

If "we" are truly interested in a democratic and prosperous Iraq, why aren't "we" so interested in establishing prosperous democracies in Saudi Arabia, Oman, Bahrain, UAE and other oil-rich sheikdoms which are by-and-large brutal dictatorships? As another example, why did the U.S. wage a terrorist war against Nicaragua between 1984 and 1989 for overthrowing a dictatorial regime under Somosza - a change which led the inter-American development bank to announce by 1984 that, under the Sandanista's, Nicaragua had the fastest growing economy in Latin America and a model for the region? The Reagan administration responded to this news by initiating a terrorist war of agression against the country as well as a blockade that eventually left tens of thousands dead and the economy in ruins. The Nicaraguans won a decision at the International Criminal Court in which the U.S. was found guilty of "unlawful agression" and ordered to pay approximately 5 billion dollars in reparations. Refusing to pay a cent, the U.S. stepped up its terrorist war through its proxy army (the Contras, who slaughtered countless civilians) and ignored a U.N. general assembly resolution condemning it (the vote was 153-2 with the U.S. and Israel voting against). What is really hysterical is where you think that western foreign policy is basically altrustic. This is complete and utter nonsense and is demolished by historical examples, of which there are many. As Edward Herman and Noam Chomsky showed several years ago, the U.S. and U.K. invest more heavily in countries that torture their own citizens. This is not because the west condones torture, but because these countries tend to be the best for western conglomerates to maximize profits. Thus, countries that torture priests, crush trade unions, pay low wages and have weak or non-existent environmental regulations tend to be the best for business investment and profit maximization. Human rights does not even appear as a blip on western foreign policy radar screens; in fact, it tends to conflict with the primary agenda (exploitation of the capital and resources of other nations for profit). Heiko ought to read some of the declassified U.K. documents of foreign policy advisers in the 1950's and 1960's (the most recent that are available). They expose Heiko's myth as a lie that western governments are "basically benevolent" that wish to spread democracy. In fact, what they show is that the U.K. more often wishes to bolster repressive regimes that deter democracy so long as this benefits U.K. corporations (it is unquestionably the same strategy pursued by U.S. planners for over 50 years). One document actually expresses concern that "Foreign countries will embrace populist forces that wish to redistribute the profits of resources to local communities". In other words, the planners were concerned that countries may wish to use their own wealth to benefit their own people, rather than western interests! In 1948, U.S. planner George Kennan uttered these famous words, "The U.S. has 50% of the world's wealth but only 6.3% of its population. In this situation we cannot fail to be the object of envy and resentment. Our real task in the coming period is to devise a pattern of relationships which will permit us to maintain this position of disparity without positive detriment to our national security. To do this we will have to dispense with all sentimentality and daydreaming, and our attention will have to be concentrated everywhere on our immediate national objective. We need not deceive ourselves that we can afford the luxury of altruism and world benefecation. We should cease to talk about such vague and unreal objectives as human rights, the raising of living standards and democratization. The day is not far off when we are going to have to deal in straight power concepts. The less we are then hampered by idealistic slogans the better." Kennan was not an extremist, but a moderate. But these chilling words undermine Heiko's utterly superficial understanding of western polciies around the world.

Tim:

I've finally realized I'm completely wasting my time here. You have no intention of providing any response to the issues I've raised, because you can't.

You state that " Mike, your charges of dishonesty (on the part of the study authors) is ridiculous." Why then, do you provide no defence of the authors' as rebuttal to my accusations?

All you've been doing is lying in the weeds, avoiding my questions, while I respond to anything you post. You only pop up when you see something in my posts you're willing to respond to, such as certain aspects of the methodology of the study.

It is blatantly dishonest of the authors to suggest that the 100,000 death figure they try to sell without the Falluja data, consists mainly of women and children, and mainly of fatalities caused by the coalition, when their own data shows that the violent deaths of men from the non-Falluja clusters outnumber women and children combined by 2 1/2 to 1.

It is blatantly dishonest of the authors (not to mention statistically absurd) to suggest they can extrapolate a figure that constitutes the majority of the 100,000 non-Falluja figure, from the violent deaths of 6 women and children.

It is blatantly dishonest of the authors to fail to reveal how many of the 12 violent deaths not caused by the coalition are from the 21 violent deaths of the non-Falluja clusters.

It is blatantly dishonest of the authors to suggest there is any accuracy in data that refuses to attribute a single violent death as being that of an insurgent.

It is blatantly dishonest of the authors to suggest there is any accuracy in data that suggests only 3% of the violent deaths in Iraq are being caused by insurgents.

"Random variation means that you might see a 50% increase, which would not be statistically significant. But violent deaths increased by a factor of 21."

Excuse me, a 50% increase from one survey to the next, assuming identical methodologies, is incredibly significant. Why do you keeping beating this dead horse concerning 21 versus 1? We've been through that, this has no chance of staying even remotely consistent from survey to survey, when attempting to measure deaths in an environment like Iraq. If we made a decision that 70% of the violent deaths from the Falluja cluster were valid and reliable to be included in a nationwide extrapolation, the 30% WE LEAVE OUT (16) WOULD ALMOST MATCH THE VIOLENT DEATH TOLL FROM THE NON-FALLUJA CLUSTERS, the ones you and the authors are completely and determinedly relying on for a 100,000 figure.

"The Lancet study reported 4 accidental and 1 violent death before the invasion. That total of 5 deaths is what corresponds to the morgue numbers for pre-war violent deaths."

Tim, this is garbage. You're moving the goalposts again. My figures come from IBC, and you're using Lancet numbers. You're trying to compare apples to oranges, and you know it. There is no mention in the IBC figures of accidents counting as violent deaths, (which was why you responded in the manner you did)nor is there any likelihood that IBC did so.

Tim, it is disingenuous of you to continue to dodge me on these issues. I don't expect any response from this, my final post here. Even if I were to receive one, what stock could I place in its integrity, given all the earlier opportunities you've had to provide one. You've displayed a stubborn, fanatical determination to accept statistics that pander to your bias. In the process, you've chucked your objectivity. That requires an even greater level of stubbornness and fanaticism. In ignoring the issues I raise, you've shown a willingness to mislead and lie to yourself.

Please, don't bother us at Daimnation again. You're bringing down the quality of debate there, and as I said earlier, wasting everyone's time.

Mike,
"Random variation means that you might see a 50% increase, which would not be statistically significant. But violent deaths increased by a factor of 21." Excuse me, a 50% increase from one survey to the next, assuming identical methodologies, is incredibly significant.
With that single sentence, you display complete lack of knowledge of statistics- significance has as specialized meaning here. The fact that you are unfamilar with it reveals you as a parrot who has merely copied some blowhard's critique of the Lancet study without even really understanding them. Or made up your own, they're certainly incoherent enough to be homebrewed.
Which also explains why you keep repeating the same questions long after Tim has answered them- you barely understand the questions themselves, let alone the answers. It also explains why so many of your questions/observations are so laughable. Stuff like What do you say about the obvious effect on accuracy that is created by the very low violent death toll from the non-Falluja clusters?- do you understand what confidence intervals are? If you do, do you also understand that this answers your question?
And then you threaten to go away, as if explaining the same thing over and over to an ignorant jackass is such a damn treat. Pitiful.
Wu

Hi Tim,

both studies have methodological strengths and weaknesses. You picked up two points (shorter recall period, verification of some of the deaths), where the Lancet study can be argued to be stronger. On the other hand, the other study has got a larger sample for a smaller population and they consider lots of things apart from deaths, to name some of their strong points.

However, as I've said in this thread before, it's not the methodology employed by the Lancet study I am criticising. I accept that the authors could not do much better under the difficult circumstances they had to work under (largely at any rate, I do have some minor quibbles).

What I am criticising is what is being made of those results, both in terms of the conclusions being drawn, and much worse, in terms of how those have been presented to and picked up by the media.

The authors of the other study are much more careful in their interpretation, and in the way the findings are presented to the general public (in fact, in the related briefing paper, heavy emphasis is placed on the responsibilities of the "occupying" powers).

On infant mortality, I was only arguing that your own point (basically the study is great, because at least they checked some deaths, while the other one didn't) didn't apply for this issue.

Or in other words the relationship runs as follows:

1. You argue (or at least appear to argue) that the study (and therefore also, by implication, what they've got to say about infant mortality) is 'good', because they've checked some death certificates.

2. I am arguing that you cannot just declare the whole study to be 'good' based on this point AND (note the word) then extend that to all of its findings. In the case of infant deaths the point just doesn't apply.

Hi Carleton,

I don't think Mike is an experienced statistician, or that you are either for that matter (correct me, if you are and it's gotten lost in your heated rhetoric).

However, you don't need to be a statistician to see that an extrapolation of 21 deaths to 60,000 is rather shaky (the 100,000 figure is overall excess mortality), when one cluster that is excluded throws up something like 50 deaths, and half of the roughly 30 clusters have no violent deaths at all.

As I said in a previous message, Saddam could have nuked Basra in 2002 killing a million, and the study wouldn't have picked it up. It's pre-war total would still stand at 1 (extrapolated to around 3500 or so for the whole country).

You ask about Mike's understanding of confidence intervals. If you do understand them, in fact, you'd realise that this doesn't answer his concerns at all.

The calculation of the confidence interval in the study is completely independent of the fact that Saddam didn't nuke Basra in 2002. It's a calculation, if that piece of information isn't fed to it, it's completely agnostic, as to whether such an event ever happened or didn't, or how likely it is.

So, based on that calculation, you could still be claiming 6000 - 200000 excess deaths with a 95% confidence interval, even though you'd know with 100% certainty that this was complete bullshit, if Basra had, in fact, been nuked by Saddam in 2002.

These statistical confidence intervals are based on heaps of assumptions.

Let me get back to the point that Tim likes to dodge, the exclusion of Fallujah for the total (100,000 excess deaths), but the inclusion for the statement: (most killed by coalition bombing were women and children).

Mike is completely correct, Tim just hasn't addressed this at all, and no, even if you do understand the meaning of confidence intervals, that issue doesn't go away. It's at the heart of what's wrong with the study (ie the way the data produced have been presented and picked up by the media).

Carleton Wu: "With that single sentence, you display complete lack of knowledge of statistics"
just checking- is that the same Wu who confused "post-hoc data manipulation" with "discarding outliers" ? The Wu who thinks that discarding 25% of your data as an outlier is okay ? The same Wu who mixed up "prone to systematic errors" and "dismiss out of hand" ?
In that case, that would be the same Wu who thinks that science is about assuming data is probably approximately correct...
For my part, I think Mike is spot on- Tim just don't bother answering the difficult questions. And I am assuming he is pretending not to understand quite a few.
j

Dear Mike, the 3,500 violent deaths per year seen in the Baghdad morgue included accidents. See this NYT story:

Before the war, before the fall of Saddam Hussein's government, seven or eight bodies arrived each day at this nondescript building in northeastern Baghdad for autopsies. Most deaths resulted from car crashes or other accidents. Killings were rare, and gun violence rarer still, a testament to the monopoly that Mr. Hussein held on the use of force.

I don't think the rest of your stuff warants a reply. You don't understand basic statistics and USING CAPITAL LETTERS is a poor substitute.

Hi Jeff,

most of what you write, I'd like to check through before saying much about it. A few supporting references from yourself would be appreciated, as they'd make my job of fact checking less cumbersome.

However, I can readily answer your first question. You can assume that everything positive the US does in its relations to a country (eg trade, aid) will be in support of its "junta", and everything they do to pressurise them (eg sanctions, ie a refusal to trade, or regime change) must be there to terrorise the masses.

But, on the whole, I believe something rather different, namely that the US trades with Saudi Arabia, rather than imposing sanctions or attempting violent regime change, because they genuinely believe that this strategy is also, largely, in the best interests of the people of Saudi Arabia and the world in general.

In other words, I do believe that "we", ie the democratic governments in the world and the near totality of their citizens, want "Saudi Arabia, Oman, Bahrain, UAE and other oil-rich sheikdoms" to become "prosperous democracies". "We" are, however, of the general opinion that the way to make this happen is neither war, nor sanctions, but rather "soft power", i.e. economic incentives, trade, communication, education.

Hi Tim,

in fact, I think you chose to reply to the bit in Mike's posting that was least deserving of a reply (after a little bit of checking back through the thread on what sort of obscurities you are actually debating here, it seems that Mike remembered a claim that the Baghdad pre-war violent death rate was extremely high, supposedly made by Iraqi Body Count, you then claimed that the figure actually included accidents and was in agreement with the Lancet study, and Mike dug in his heels, because you didn't really address the reference to the Iraqi Body Count).

I don't know what the Iraqi Body Count claimed, and whether they got it wrong, or whether they used a different definition of violent death from the one used in the Lancet study etc...

But is this really the only thing you think is worthwhile replying to?

One interesting point that Mike makes (not that this one is the most significant to reply to either in my opinion, the "most of those killed were killed by coalition bombing and most of those were women and children" subject is rather more worthy) is that the study refuses to put a number on insurgents/terrorists killed. Many of the men could have been legitimate targets, so they say, but then again, maybe not.

The way this easily comes across is just perfect for "insurgency" propaganda. The coalition has managed to kill a 100,000 innocent civilians, hardly any "insurgents", and the "insurgents" themselves have caused little loss of innocent life.

There is ample room in the actual data reported for a rather different interpretation that would be more in agreement with what the coalition themselves say, ie several ten thousand dead terrorists/Baathists, much less loss of innocent civilian lives, and much of that due to terrorism, and the rest virtually entirely due to the tactics of the "resistance" whose aim it is to do everything they can to make sure there'll be civilian losses on the battlefield.

Heiko, the study does not say that the coalition killed 100,000 innocent civilians. You know this. If someone claims that is what the study found, then they are wrong. But the appropriate response to that is not to attack the study but to point out what the study actually found.
I'll bet that if they had tried to classify violent deaths as insurgent/non-insurgent your complaints would have been even louder. The only thing they could have done there would have been to have asked whether the dead person had been an insurgent and you would be complaining that the information wasn't verified and people had an incentive to lie about it.

It's 100,000 excess deaths, which likely includes some insurgents who are dead now but would not have been if we hadn't invaded Iraq. We could find out more, but it's pretty obvious that the coalition doesn't want to find out more.

Heiko, I haven't checked all of the numbers yet. But apart from Mike's hysterical, and often illiterate rants, there are two recurring themes to these posts I see that are worth discussing - 1) that the bootstrapping of 33 study clusters to all of Iraq is unacceptable, and 2) that the claim that most of the fatalities were women and children, or at least, non-combatants.

Regarding 1), The methods they used to extrapolate results and risks have been reliably used in studies like this before. Relative risks for each cluster were generated assuming that each was allowed to have a separate baseline rate of mortality that was increased by a cluster-specific relative risk after the war. The
average relative death rate was then estimated using a conditional maximum likelihood method with the total number of events over the pre-war and post-war periods for the baseline rate. Then, their final CI was derived using a well characterized bootstrapping method (McCullagh P, Nelder JA. Generalized linear models. London: Chapman and Hall, 1989; Diggle PJ, Heagerty P, Liang KY, Zeger SL. Analysis of longitudinal data, 2nd edn. New York, NY: Oxford Science
Publications, 2002; Efron B. Bootstrap methods: another look at the jackknife. Ann Stat 1979; 7: 1-26). I have yet to see any critic address these points specifically, with examples and sources.

Regarding 2), The study did not ignore this point. It is discussed at length on page 7. Roberts et al. point out that passive surveillance systems such as those used by IBC are typically of low sensitivity, and it is no surprise that, for instance, IBC underestimates Coalition Force casualties. They also note that IBC's trends closely parallel theirs however and can be used in that capacity, but not for actual counts. They note that of the 61 violent deaths attributable to U.S. forces they tabulated from their clusters, 28 were men, 28 were children under 15, 4 were women and one an elderly man. It is not reasonable to assume that significant percentages of children, women, and the elderly were combatants, particularly in Iraqi culture. Thus, even if assume that all men killed by U.S. forces were combatants, we're still left with over 54 percent of the deaths as non-combatant collateral damage. If at least half of the men killed were also non-combatants (not an unreasonable assumption), that figure goes to 77 percent.

Based on all this, I think it's safe to say that in the very least, tens of thousands of innocent civilians died violent deaths due to the Coalition Forces' invasion of Iraq. It is also clear that anyone who claims that the 98,000 median figure for total violent deaths in Iraq is unsupportable cannot make their point by generalized ranting about "small numbers", death certificates, or generalized "that's rediculous!!!..." statements, as some folks (not you though :) ) have done at this blog recently. If these critics want to be taken seriously, they need to provide something that I have yet to see any critic of the Roberts et al. study provide - namely, a clear, and well cited refutation of the specific clustering, bootstrapping, and per-cluster relative risk methods cited in the paper and described in detail in their cited sources (as provided above), followed by a demonstration that the known problems with these methods are not accounted for in their resultant Confidence Interval.

Finally, it has been my experience at least, that most of those who attack this study are ultimately out to justify the U.S. led invasion of Iraq. Thus, they want to show that Iraq is happier, healthier, and better off today than they were in 2002, and massive nationwide carnage does not fit too well with this agenda. Therefore, given that the issues surrounding cluster bootstrapping methods tend to under-estimate results rather than overestimate them, these folks need to demonstrate that under-estimated results are far more likely than overestimated ones. When I see someone actually demonstrate any of this, with numbers and clearly cited examples, I'll take the criticisms more seriously. Best.

Scott Church wrote:
"that the bootstrapping of 33 study clusters to all of Iraq is unacceptable"
Just for info, Don Quixote, you are the only person on this page who has mentioned "bootstrap".
In simple terms; I haven't seen anyone complain about the methodology of the extrapolation, which may be entirely appropriate. The complaint is that the use of cluster analysis, linked to a relatively small sample size, is prone to systematic error.
Oh, and 19% of the small sub-sample of deaths could not be verified; so there is a real problem with recall bias. Specifically, their final confidence interval does not take any account of the uncertainty in their estimates in numbers of death associated failure of verification.
"most of those who attack this study are ultimately out to justify the U.S. led invasion of Iraq"
this is a straw man argument, ad hominem, and irrelevant. The study should stand up as good science, irrespective of what policy use the results may be put to. The fact is that cluster sampling is prone to error; that small sample size studies are prone to error; that recall bias has made many studies meaningless; that there is a statistical error in this study, called post-hoc data manipulation. These are four good reasons why this study cannot be relied upon to be accurate.
J

Hi Donald, my reply to you seems to have mysteriously disappeared overnight. The 5 out 33 number is interesting, and indicative of 4-8 deaths from coalition bombing outside of Fallujah (as no cluster outside of Falluja recorded more than 2 deaths). 0-6 of these could be women and children. All could be combatants or none.

It's most unfortunate that my other comments have disappeared. In summary, I think that in 100 years time, the whole world will be where the European Union, Japan, Canada and US are today, and there'll be no more war. This isn't because Europeans or Japanese today are better people. The genetic make-up of the population hasn't changed, what has changed is the "system", factors like prosperity, education, a free press etc...

Hi Scott,

on 1), that's not the issue that concerns me. One just shouldn't read too much into things like the 8000-200,000 confidence interval. All of this is based on assumptions, and to interpret the findings we really need sound judgement based on other information we have about the situation in Iraq. Keep the, what would be different about this study's findings, if Saddam had nuked Basra in 2002 example in mind.

On 2), well yes, if you include Fallujah. If you don't, you are suddenly down to 6 women and children killed.

So, no I don't think it's safe to conclude that tens of thousands of civilians have been killed by coalition forces. A few thousand is what the coalition forces and the Iraqi government estimate.

Also, 98,000 is a number claimed by the Lancet study for excess deaths, not total violent deaths.

Heiko, I am perplexed by your comments about Basra. To reduce travel requirements it was paired with Missan. It was paired with Missan because they had similar violence levels. So if Saddam had nuked Basra, it would have been surveyed and a large number of deaths found.

What I am saying is that this requires judgement on the part of the researchers, which is based on incomplete knowledge. Of course, they would have known about such a momentous event and could have accounted for it. But if they hadn't, their calculation would have come out the same, and overlooked those 1 million deaths.

You just cannot get away from the fact that a lot is based on assumptions, and because these are about things we have poor knowledge of (like the geographical distribution of Saddam's killings), those assumptions may be quite faulty.

When properly interpreting the 8000-200,000 95% confidence interval one needs to take account of this.

Hi Scott,

whether you were in favour of removing Saddam or not, we are where we are and also need to look to the future.

The terrorists are trying to destroy Iraq, and propaganda is one of their tools.

It is a good idea, in my opinion, to forcefully reject their propaganda line ("legitimate" resistance defending the Iraqi people from American slaughterers, and while the Americans only seem to be able to kill women and chilren, the resistance is taking few losses, while only out to kill American oppressors. Any kidnappings must be the work of ordinary criminals or staged by the Israeli secret service and are part of the big Zionist/crusader plot to undermine Islam).

Even if you don't happen to like the decision taken by the coalition to remove Saddam from power.

A correction--the source for the 5 sites out of 33 that experienced deaths from bombing was an email to the medialens website from one of the Lancet authors. I don't think it was in the Baltimore paper article in case any one goes looking.

Two points about the other 4 sites (excluding Fallujah). First, all of the children in the study were killed by air strikes--24 in Fallujah and 4 outside, if my memory is right. I conclude this based on the following-- The 12 people out of the 73 not killed by the Americans were adults. 58 of the 61 killed by Americans were killed by air strikes and the remaining 3 were adults. Therefore all the children in and out of Fallujah in the study were killed by American air strikes. 52 people were killed in Fallujah and 58 were killed in air strikes, so you also know that at least 6 people were killed by air strikes outside of Fallujah, 4 of them children.

Now if we really want to extrapolate from 21 deaths, then 60,000 people died violently outside of Fallujah, and at least 6/21 from air strikes (about 17,000) and 4/21 were children (about 11,000). Of course it's silly to take those numbers seriously. What the paper authors do is take the 21 deaths as a guide to roughly how many people are dying from violence and then they point to the Fallujah cluster as evidence that the number is probably higher, due to American air strikes. They point out that one of their neighborhoods was in Sadr City and showed no violent deaths in an area where there had been much violence, and that many
neighborhoods in Fallujah and other towns seemed to be just as devastated as the one they sampled. So they reasonably suspect that it is likely that most random samples of 33 neighborhoods would turn up a Fallujah-outlier or two, just as it also turned up a neighborhood in Sadr City where no one had died. (Nobody seems to complain about that.) Is this conclusive proof that massive numbers of people are dying from American bombing? No, of course not. But it is a reasonable thing to suspect and given the data it's the most reasonable thing to conclude.

On the more general question of violence, if you sample 33 neighborhoods and find that 15 of them show violent deaths and 5 of them show deaths from air strikes, then one can safely say that this country is suffering from a lot of violence. And it's clear many are civilians killed by the US. You can go on and do crude back-of-the envelope calculations assuming only 1 or 2 deaths per site. knowing the number of people at each site and the population of Iraq and you'll get the sorts of numbers I was citing above.
I don't have a background in statistics, but I think that the study makes a good case for many tens of thousands of deaths from violence, with a large fraction being civilians killed by America.

On the broader question about America, the West, and so forth, I too prefer Western democracy to the known alternatives and as a Christian I prefer Christianity (the more liberal kind) to the alternatives. But I know the history of my country reasonably well and while I am proud of much of it, that history also includes a very long series of crimes against humanity on both small and large scales. And plenty of lying. John Negroponte, currently the American in charge in Iraq, held a similar position in Honduras in the 80's and he lied about Honduran death squads then. If you expect the American government to be honest about human rights violations committed by itself or its allies than you are either very naive or willfully blind.

Donald Johnson : "Is this conclusive proof that massive numbers of people are dying from American bombing? No, of course not. But it is a reasonable thing to suspect and given the data it's the most reasonable thing to conclude."
Don, this is a matter of balance. You have highlighted the fact that the combination of low sample numbers, and cluster sampling, is a potent cocktail for misleading results- and you exemplified this with the Sadr data. So it is clear that this exercise uses a sampling methodology that is known to have weaknesses, and it is known that such verification of their deaths as they did failed on 19% of cases.
against that, you have to bear in mind that there are other data, such as those compiled by Iraqi hospitals. And these much larger numbers show a major discrepancy against the Lancet data, with its tiny numbers.
Bottom line: it is brave to conclude that the Lancet study, with its known methodological weaknesses, is "more reasonable" than the other data.
J

[Deleted, see my comment policy, TDL]

Mike, this is my blog and I don't have to tolerate abusive posters. You are welcome to post your argument again if you remove the personal attacks.

Tim:

Your bias is showing. Seems you had no problem with Carleton referring to me as an "ignorant jackass," since his post stands.

By the way, what was abusive? My reference to Scott as a "pompous twit?" If that's what you're referring to, I have no problem with you deleting it. It doesn't change what he is, whether the post lives or not. But why, may I ask, did you delete my lengthy post that contained my most recent argument? Where was the abuse that warranted this post's removal?

So Mike, I'm a "pompous twit" huh? You'll be happy to know that you're not the first to attack me with that label. It's the last and most desperate resort of those who are weak, and can't defend their views properly with facts and professionalism.

And Tim, once again, thank you! Not just for defending me, but for defending the integrity and professionalism of this blog which you are so gracious to maintain for us. It is appreciated.

Heiko, you would do your case a lot of good by addressing the possibility of an underestimate. As it stands, you are talking about the measurement problems as if any uncertainty about the confidence interval was a reason for believing a figure in the lower half. This is what I called "Kaplan's Fallacy" in my original post.
Basra was in fact, not nuked in 2002 by Saddam Hussein (not least because he didn't have a nuke to nuke it with). However (and to take only one example from the top of my head), Najaf was the scene of very intense fighting indeed after the invasion, and was not sampled.
More generally, arguments that the survey sample might not have been representative under various hypothetical scenarios are not particularly good evidence that the sample was, in fact, unrepresentative. The fact that the non-Kurdish clusters were so consistent in the story they told (buig increases in death rate, mainly due to increases in violent deaths) suggests that the onus is firmly on anyone who wants to claim that the sample was flawed.

Hi Heiko, About the 98,000 excess deaths, yes, you are right. That was what I meant, but perhaps I could have worded myself better. You are also right about the 61 violent deaths as including Fallujah. Given that Fallujah was an outlier (as we all know), my argument would have been better formulated if I had started out not including it. Ultimately, the raw numbers from the study are as presented by Roberts et al. in their Table 2 (Page 4). There, if I'm reading them right, outside of Fallujah there were 21 violent deaths of which 8 were women, children under 15, and elderly. The remaining were men. Applying the same logic I used above, roughly 38 percent of the deaths (about 37,000) were non-combatants if we assume (conservatively, I think) that all men were combatants. That number would then go to about 69 percent if 1/2 of the men, say, were not combatants - or about 67,000.

Yes, I am winging these numbers, and they can't be taken literally. The larger point is that no matter how we play the numbers on this out, even with pretty conservative assumptions, it's pretty safe to say that a lot of people have died, and continue to die violently since the invasion - well into the tens of thousands by any set of reasonable assumptions we start out with. Also, that a significant percentage of those killed were non-combatants who were caught in the crossfire - like the story you told about Aya's grandfather in the "A new attack on the Lancet study" post. That story broke my heart!

The large death estimates from these estimates, including the tens of thousands of innocents, follow from the 98,000 median excess death figure. The reliability of that, and the wide confidence interval associated with it, ultimately comes down to the reliability of the methodology. Roberts et al. arrived at those numbers in a specific way involving the methods I mentioned (e.g. bootstrapping, cluster-specific relative risks after the war, and conditional maximum likelihood methods). Their clusters were pretty well characterized and statistically significant for a cohort study like this one compared to other similar studies. So if there is a problem with their numbers, it will be because of flaws in their data reduction methodology. This is the problem I have with many criticisms of this study. Most of what I've seen are generalized "you can't trust big numbers that started from little numbers..." arguments, few of which ever actually address the data reduction methods the big numbers came from. These arguments are purely hand waved. Why can't I trust big numbers from little numbers? Where exactly are the statistical methods flawed? To take down a study like this properly, someone needs to demonstrate that Roberts et al. started with poorly characterized clusters (and this needs to be spelled out specifically), and used data reduction methods that were not sound for this type of cohort study. In addition, they need to demonstrate that whatever issues there were with their data characterization methods were not accounted for in their CI.

Also, regarding that confidence interval, it can't be emphasized enough that it is not true that one point in that spread is just like any other. The low and high ends of it are very unlikely (<5 percent probability) extremes. Values closer to the median are far more likely. In that regard, it is striking that the CI does not include zero - On only 1 chance in 20 odds, the very best that the world gets over there... is three 9/11's since we invaded, and until some comes up with a proper mathematical demonstration to the contrary, 20 to 30 9/11's is more realistic. To me, that's pretty grim.

Hi Heiko, One other thing I forgot. On a non-scientific note, per your comments about Saddam, I am most in favor of his having been taken down. Truthfully, I really don't think anyone thinks otherwise. I think that if there is disparity of views on the war, it's not because any love is being lost on Saddam - it's because people wonder whether it was necessary to take him down like this. I think most people would agree that war should always be a last resort (does anyone really think otherwise?...). To my mind, that means that by the time we invade, all other options have failed, we've been over the intelligence 1000 times, we have plans and backup plans, and more backup plans for what will fill the void when the dictator is taken down, and we have carefully (and I do mean carefully) thought out what the impacts will be on the innocents who live there - before, during, and after the war is over. It's pretty disturbing to me that we in the U.S. charged off on a $200 billion bloody, carnage ridden invasion of another country that might well have led to tens of thousands of excess deaths if we're lucky, and hundreds of thousands if we're not... before our president had even been over his own damn intelligence reports carefully enough to find out that there was no evidence of WMD's or a working relationship with Al-Qaeda (the arguments the war was sold on in the first place), before he or anyone else had thought up a contingency plan for dealing with insurgents, and before he had thought out any of the larger political and religious issues that awaited us there when we tried to set up another government to fill the void, and even before he'd thought to train his own troops on dealing with urban insurgency. To me, that's not the sort of careful, thoughtful, and prayerful approach taken by a nation that is regretfully engaging in what they believe to be unavoidable combat, and shedding a tear as they do so. It's the trigger happy recklessness of someone who can't wait to kick some butt - shoot first, ask questions later. If I were in Bush's shoes, I may well have invaded, but I damn well would've thought, wept, and prayed more about it first, I would have done more homework... and I would've invaded the country that actually had the WMD's and the working relationship with Al-Qaeda. I'll bet that most people who are upset about the invasion have similar feelings.

As for the terrorist propaganda, you're absolutely right, and we shouldn't play into it. Although I think we need to remember that not everyone shooting at us is likely to be a terrorist. It's not reasonable to think we can invade another country, bomb it back to the stone age, impose out idea of how things ought to be on them, and not expect some resistance from people who resent being invaded - especially if they've lost loved ones in the combat.

Mike, both of your posts contained personal attacks. Carleton should not have called you an "ignorant jackass", but I'm usually reluctant to intervene unless it looks like things are turning into a flamefest. Your post attacking Scott offered no arguments at all, just name calling, so I deleted it.

You can repost your other comment if you remove the personal attacks.

Hi dsquared,

I think there is the possibility of a significant undercount for infant deaths, in particular.

As for violent deaths from coalition bombing, several hundred thousand is just not credible. We know a lot about what happened, the study's numbers need to be looked at in the context of what other knowledge we have.

Several ten thousand violent deaths post war is well supported by the study's data, and is well within what other evidence indicates.

Hi Scott,

excess mortality includes infant mortality. For violent deaths, we are talking more like 60,000.

So, you'd extrapolate around 12,000 children, 6,000 women and 6,000 elderly killed by violence, but the numbers are too small to allocate this reliably to coalition killing, ordinary crime, insurgent activity etc...

Yes, not every point in the confidence range has the same likelihood. However, neither is it the case that the study shows with 97.5% confidence that more than 8000 excess deaths occurred. There are just too many unknowns, assumptions etc. that flow into their calculations. How significant were prison deaths before the invasion? How significant was nutrition as a factor for infant deaths in Iraq? Does that explain a drop in infant mortality from 108 to 102, or to 29? If it doesn't, why did they undercount infant mortality so badly? Recall bias? Poor interviewing?

97.5% would be accurate, if a number of assumptions about things we have rather imperfect knowledge about are true.

There is no way to account for these factors easily to come up with a hard number.

But, based on my judgement, I'd say that the evidence is that general and infant mortality has not significantly changed overall, and violent death has increased by several ten thousand, with the largest proportion of those killed after the liberation being "insurgents", and a significant number being innocents killed by terrorists.

I do think that there is room for judgement particularly with the infant mortality figures, ie a reasonable person can look at the information and conclude that the most likely estimate is a significant increase from a much lower figure than 102. I accept an increase from 40-50 to 80-90 post-war as something a reasonable person might conclude, even if I am not convinced myself.

On violent death, I don't accept hundreds of thousands as something realistic, ie I think a reasonable person with the knowledge available to me would discount this as baseless. Jeff mentions that Fallujah got 70% destroyed. However, my home town of Dueren was nearly completely eradicated and something like 3500 people died. The most reliable estimates put Dresden in the low ten thousands, a city that went up in a firestorm.

Killing a few hundred thousand through bombing is quite hard, even if the aim, as with Dresden or Dueren was as much loss of civilian life as feasible.

Only three deaths of civilians are from small arms fire by US soldiers in the Lancet study (the kind of incident that killed Aya's grandfather), which is bad enough of course as it translates to around 10,000 deaths.

Hundreds of thousands of people killed in airstrikes don't just get overlooked by hospital accounting.

It's true that in the US every year 40,000 people die on the road and the press doesn't report much about it. But in Fallujah, single strikes on "safe houses" got massive press attention. The insurgents did their level best to publicise deaths. And in April they claimed something like a thousand dead, not 40,000, the kind of figure you get from extrapolating the Lancet data. And extrapolating August, we get 100,000 people killed in Fallujah (i.e. the 3% of Iraq the cluster stands for, 3% of Iraq is a bit more than Fallujah itself, but not that much more, about twice to three times the population of the city itself).

Hundreds of thousands of dead would imply that the press would only pick up 1 in 20 deaths from coalition bombing, even though insurgents, and their sympathisers, have an interest in inflating the true numbers when approaching the press. How are these numbers supposed to be hidden away in places like Mosul or Baghdad?

Hello all, back again. Although I fully intended to walk away from this after my last post, the latest from Tim, Carleton and Scott demand a response from me. It has been altered and expanded, and hopefully will pass the Tim Test.

I made it clear to Tim at Daimnation, the site where our debate started, that I do not have a background in statistics. Contrary to what Tim and Scott suggest, a lack of a degree in statistics does not disqualify me or anyone else from attacking the obvious flaws and outright distortions contained in this study. The position of Tim and Scott is based on a steadfast determination to confine the debate within a statistics bubble, to keep it impervious to challenges that are grounded in common sense and an analysis of the situation in Iraq both pre and post invasion.

What I find completely hypocritical is the canard put forward by Tim and Scott that my lack of statistics training disqualifies my opinion, yet people WITH a statistics background (and not just Heiko) are making the same key criticism of the study that I do, that the very low numbers of deaths from this study are not conducive to accurate nationwide extrapolations within the context of Iraq.

Carleton, I can only assume from this quote that you either haven't been reading this thread, or recall only what you want to recall:

"Which also explains why you keep repeating the same questions long after Tim has answered them- you barely understand the questions themselves, let alone the answers."

Not only has Tim not answered my questions, Carleton, he finally acknowledged in his final post that he wasn't going to, hiding behind the feigned indignance so popular with those engaged in a cop-out, that my questions aren't worthy of replies.

I've raised some very straightforward, obvious concerns with this survey, such as the deliberate, deceitful practice of the authors to not identify where the 12 non-coalition violent deaths belong. The reason we're not provided with this information is apparent. The authors are concerned that it will adversely affect their propaganda message. Fully half the 21 violent deaths in the non-Falluja clusters may well come from this group. Then, we still have the matter of how many of the remaining half were insurgents. The devastating impact this would have on the authors' contention that the coalition directly killed most of the excess 100,000 death figure is the reason we're not provided with this information, and consequently, IT'S ALSO THE REASON TIM WON"T ADDRESS IT EITHER.

But nothing is stopping you Carleton, or you Scott, from trying to rebut what I've been arguing.

This study is full of contradictions that can't survive real world scrutiny, outside the laboratory bubble where it's nurtured. For example, the authors recount their concerns and experiences over the inherent dangers in conducting the studies. Yet all the dangers they describe are created by the insurgents. The authors express no concern over the coalition being a threat to their safety in any way. On the contrary, they talk of fears that Iraqi police and guardsmen at checkpoints will identify them later to insurgents. They express their belief that study interviewers will almost certainly be killed if they leave their vehicles in or near Falluja, or if they're found with a GPS unit, it will mean execution.

How then, does this reconcile with the study's conclusion that insurgent-caused deaths are inconsequential in the study's apportioning of the 100,000 excess deaths? A mere 2 out of 73, or 3% of all violent deaths. How much more of a contradiction do you want? According to the authors'data and resulting observations, the coalition is responsible for the vast majority of all violent deaths, the insurgents virtually none, yet the fear that drives the limitations of the actual sampling is entirely insurgent caused.

Another example of the study's failure to withstand a non-statistician examination lies in the authors' contention that the vast majority of violent deaths occurring post regime change were the result of air strikes. In the interview of Roberts reported in Medialens, as mentioned in this thread by Donald Johnson, Roberts reveals that 5 clusters reported bombing deaths. This is information that should have been readily available in the study report itself. While the authors still withhold several other key pieces of the puzzle, the revelation concerning the dispersal of the bombing deaths allows, through the process of elimination, a clearer picture of the breakdown of the non-Falluja deaths.

It now appears likely that at least 10, and possibly all 12 of the violent deaths not inflicted by the coalition are contained in the 21 non-Falluja violent death toll. It can also be reliably estimated that 6 or 7 is the number of deaths resulting from air strikes, spread among the 4 non-Falluja clusters that reported such deaths.

Do these breakdowns corroborate and bolster the claims and inferences made by the study authors, and those commentators who also make and defend these inferences? It appears they do not.

For instance, the authors claim that "Violent deaths were widespread." But were the coalition-caused deaths widespread? It now seems that coalition caused deaths were reported in only 7 of the 32 non-Falluja clusters, and in one of those clusters the authors concede that the man who died (shot by American troops) " may have been a combatatant."

More importantly, the authors claim that 95% of the coalition caused deaths were from air strikes. But in the 97% of Iraq that makes up the survey sampling excluding Falluja, only 12% of the clusters reported air strike fatalities, and none of these clusters seems to have reported more than 2 killed. This doesn't reflect the scenario being created by the authors and those who have hopped on their bandwagon, which is one of widespread, indiscriminate aerial bombing, obliterating civilian neighbourhoods and killing the non-combatant occupants of these neighbourhoods by the thousands. This certainly may have been the case in parts of Falluja, but as has been noted by many others, the study is attempting to create a carry-over effect with regard to bombing deaths for the other clusters.

This is extremely misleading, coalition bombing was not uniform across Iraq, and the proof is in the authors'own data. This is a cluster study grouping 30 or so homes in the same neighbourhood. If the bombing was occurring as implied by the authors and others, we should not be seeing deaths in 1's and 2's in the other 4 clusters that reported bombing deaths. One could reasonably expect that multiple deaths in more than one household should have resulted from a single air strike.

In much of the insurgency campaign outside of Falluja, the Americans have refrained from using air strikes in heavily populated areas, or used them only as a last resort.Often a single precision bomb was reported as the entire extent of a bombing attack.

Scott, you're playing the same dishonest game as the study authors, including the Falluja death figures when it suits you, then discarding them when it doesn't. The 61 violent deaths you refer to include Falluja. I'll ask you straight out. If the authors tell us the Falluja data is unreliable for extrapolation, but they're confident that AT LEAST 100,000 excess deaths have occurred without the Falluja data, are the authors being forthright and honest in suggesting that most of these 100,000 excess deaths that were caused by coalition violence were women and children? We know this is simply false, by a margin of 2 to 1.

In relation to your first point, concerning the reliability of the methods used to extrapolate the results of this survey, there simply is no way a single survey, conducted in an environment like Iraq, can be trumpeted as even reasonably accurate, standing alone. I don't need to be a statistician to realize that only multiple surveys, using identical methodology, can even begin to produce results worth considering. With that in mind, I still maintain that subsequent surveys could, and likely would, produce results that varied widely from this Lancet study, not only in the post-invasion numbers, but in the pre-invasion death figures also, as Heiko has pointed out.

"Finally, it has been my experience at least, that most of those who attack this study are ultimately out to justify the U.S. led invasion of Iraq. Thus, they want to show that Iraq is happier, healthier, and better off today than they were in 2002, and massive nationwide carnage does not fit too well with this agenda."

Well Scott, it has been my experience at least, that most of those who defend this study are ultimately out to discredit the U.S. led invasion of Iraq. So what the hell is your point, exactly? I believe that deaths have increased in Iraq since regime change. My objection to the Lancet study is that it is not an accurate depiction of the number or character of these excess deaths. How can it be, when it refuses to categorize a single violent death as that of an insurgent or terrorist, when the Americans estimate they have killed 25,000 of them?

Things are not as they should be in Iraq, and may never be. There was no reasonable prospect that Saddam's Baathist/tribal power base was going to be internally overthrown in the near future. Any attempt at overthrow would certainly have resulted in a protracted civil war, costing the lives of hundreds of thousands of Iraqis, with no guarantee of success. Foreign intervention was the only means to give Iraqis a chance to build a better society. The same Baathist thugs who formerly terrorized the Iraqi people are fighting the coalition to retain that power to terrorize. The foreign Islamist fundamentalists aiding them are the same people who believe in killing their sisters and daughters if they "dishonour" the family. The Lancet study attempts to pass off as innocent civilians the thousands of these butchers that have been killed by the coalition. Moreover, the Lancet seeks to place the blame for their victims' deaths on the Americans as well. In defending their study, be aware Tim and Scott, that you're doing the exact same thing.

Mike, you've got a much better chance of a civilised debate if you:
1. Use the "P" HTML tag to separate paragraphs; I too find Deltoid a little bit temperamental when it comes to line breaks, but this method works.
2. Stop making accusations that various people were "deceitful" or "deliberately dishonest" when you have no evidence for this.
3. Become a bit clearer about which of your criticisms have to do with things in the study (like estimates of the number of deaths) and which of them are about things which weren't in the study but in your opinion ought to have been (like characterisation of the dead as insurgents or otherwise).

I think you actually have a point that the attribution of deaths by violence to the coalition rather than insurgents seems counterintuitive. My guess is that the Iraqis are not fully aware of the extent to which the insurgent forces use mortars, and for this reason the survey respondents tended to attribute all deaths from explosions to coalition bombing. This would be a problem with the survey, although it would obviously not support a charge of dishonesty aimed at the authors; they have no basis for making ad hoc adjustments to the responses they received to account for hypothesised bias.
However, all your other points are just too confused to respond to; I'm not at all sure what specific claims you are making. Several of them appear to be based on misreadings of the study. For example, you say
are the authors being forthright and honest in suggesting that most of these 100,000 excess deaths that were caused by coalition violence were women and children?
The authors do not say this. They do not claim that 100,000 excess deaths were caused by coalition violence. The figure 100K refers to total excess deaths. Deaths by violence are a subset of excess deaths, and violence attributed to coalition forces is a subset of this subset.
If you're really interested in establishing the facts about what is likely to have happened on the ground in Iraq, then you need to turn down the emotional heat, stop throwing around accusations of dishonesty and (preferably) accept that the Lancet study is the nearest we have to comprehensive hard data. We can discuss where the holes are in the study, and we can discuss the points on which we think there are likely to be important differences between the results and reality, but it really is not legitimate to simply ignore the work that has been done.
I would also recommend to you, as I recommended to Heiko, that you seriously consider the possibility of an underestimate. It is much more difficult to have a good faith discussion with somebody about the strengths and weaknesses of the Lancet study if they seem to be always latching onto every bit of uncertainty as if it were evidence for one end of the confidence interval rather than the other.

Mike, you admit that I know more about statistics than you. I am baffled as to why you keep insisting that I am wrong on the statistical questions and you are right. I went to some trouble to present an explanation and experiments you could try with a spreadsheet without having to know statistics, but you seem to have just ignored this.
And yes, other people who do know statistics have presented arguments about small numbers without being told that they were wrong. But that's because their arguments are different from yours.

As for the rest -- I've read the study. I know what it says. You are misrepresenting it. If someone says that the study found 100,000 innocents had been killed by the Americans, then that someone is wrong.

On the air strike question, out of 33 neighborhoods one was devastated and 4 show light casualties. So we're supposed to assume the devastated one was a statistical freak though the authors say many other unsampled neighborhoods appeared equally hard hit and I'll repeat, the neighborhood in Sadr City showed no casualties and that doesn't seem to raise eyebrows. It's only a statistical fluke if a neighborhood is devastated, apparently. The most natural conclusion from this sample is that some neighborhoods (a few percent, maybe as few as 1 and maybe higher) are devastated by American bombing, while many others experience smaller attacks, or perhaps the people in those other neighborhoods were killed when away from home. As for the 12 killed by non-Americans, I think the largest single number was from crime, not from the insurgents. That makes sense--a recent story said there are around 700 murders a month in Baghdad, according to morgue statistics. Hell, if I go back to playing the game of taking the non Fallujah 21 deaths as an exact sample of what's going on, the breakdown would probably agree pretty closely with other sources--10-20 thousand from criminals, 10-20 thousand from insurgents, and maybe 25-30 thousand from Americans. And throw in the Fallujah cluster and you suspect the American number might be significantly higher.

One other point. The Lancet paper made the politically correct claim that most of the American civilian deaths came from airpower, not from individual soldiers shooting first and asking questions later. But Fallujah skews that--perhaps correctly, perhaps not. I haven't seen too many conservatives attacking the paper on that basis.

Tim, D Squared:

First off, I have posted at many blogs, and have never seen one that displays text in the manner this one does. I have no idea how to use the " P HTML " to split paragraphs, because no site I've been on does so.

Secondly, I made an error in typing my last post, with the sentence that states " .... are the authors being forthright and honest in suggesting that most of these 100,000 excess deaths that were caused by coalition violence were women and children? That is not what I intended to say. I am aware that the authors are claiming that most of the excess 100,000 deaths they extrapolate are violent deaths, most of these (approximately 85%)were at the hands of the coalition, and that separately, most of them (the coalition caused deaths) are women and children. This doesn't change the deliberate misperception that the authors are creating, and I have no difficulty in characterizing this as dishonest, particularly in relation to their selective use of the Falluja data. It is an indisputable fact that in the 97% of Iraq that makes up this survey outside of Falluja, men killed by violence outnumbered the women and children combined by 2 1/2 to 1.

As I've mentioned many times here, we don't know if most of the deaths in the non-Falluja clusters were in fact inflicted by the coalition. It may well be that this is not true, by a narrow margin. That the authors decline to reveal the breakdown of this data is highly suspicious, and if such a breakdown reveals what I suspect, than how can you say they were not being deceitful in both withholding the breakdown, and creating a false impression concerning the origin of deaths in the 97% of Iraq outside Falluja? We know from the data that the majority of the violent deaths, even if we include Falluja's total, are men, not women and children. Is that a conclusion noted anywhere in the text of this study?

D Squared, my arguments are straightforward. I'm saying that Tim and others are trying to protect the integrity of the study by confining the debate to the bubble of the statistical world. In my view, this defence fails when the study is scrutinized in another forum, utilizing common sense and the general overview we have of Iraq before and after regime change, and other factors such as existing media accounts of fighting and casualties, and coalition military practices.

As two examples of this, I cite the authors' own account of the various dangers they perceived the study interviewers faced in compiling the survey. It was clear that the danger originated solely posed by the insurgents, not the coalition. Yet the study brushes off any significant contribution to the death toll from insurgent killings.

I also point out the questions that arise from such a low number of bombing deaths among the 4 non Falluja clusters that report these. You offer a possible explanation that some of these deaths could have been caused by insurgents using mortars. If true, this only serves to further dilute the coalition culpability in the vulnerable to manipulation figure of 21 non-Falluja violent deaths. Additionally, what does it do to the authors' ability to project nation wide violent death figures, based on their claim that air strikes are responsible for a huge percentage of these deaths? The authors are already starting with a pitifully small number of non-Falluja bombing deaths to begin with.

Tim: I'm at a loss to understand your point concerning the use of a spread sheet to extrapolate further potential results by using hypothetical data. It's my belief that only actual data, compiled from additional identical surveys, would be relevant for a comparison to the data obtained from the original Johns Hopkins study.

I disagree when you say that others with statistics credentials are presenting statistical arguments concerning the dangers in extrapolating from very small numbers to very large numbers, but their arguments are different from mine. At it's core, that is my argument, that a figure such as 21 violent deaths is too small to accurately extrapolate to a nation wide figure, and more importantly in my view, far too vulnerable to dramatic fluctuations in the composition of that figure (gender of the victim, cause of death, culpability of death).

D Squared, correct me if I'm wrong, but haven't you been saying the same thing at Crooked Timberz?

Finally, D Squared, you're not accurately characterizing the premise the authors put forward when you make this statement:

"The authors do not say this. They do not claim that 100,000 excess deaths were caused by coalition violence. The figure 100K refers to total excess deaths. Deaths by violence are a subset of excess deaths, and violence attributed to coalition forces is a subset of this subset."

The authors clearly blame the lion's share of the excess death figure of 100,000 on violence. To illustrate this further, here's a quote from the study:

" It is surprising that beyond the elevation in infant mortality and the rate of violent death, mortality in Iraq seems otherwise to be similar to the period preceding the invasion. This similarity could be a reflection of the skill and function of the Iraqi health system or the capacity of the population to adapt to conditions of insecurity."

Are you not attempting to minimize, after the fact, the culpability the authors are placing on the coalition, to make these numbers more believable?

This is simply part and parcel of the intent of the authors to dump on the coalition. So Tim, I realize you may be technically correct when you state:

"If someone says that the study found 100,000 innocents had been killed by the Americans, then that someone is wrong."

However, the authors certainly intend that the reader arrives at the conclusion that a majority of the 100,000 were victims of the Americans.

Mike, you simply have no evidence for any of the claims you are making about the authors' intents.
I can't speak for Tim here, but I have seen quite literally hundreds of cases where people have decided to ignore the evidence in front of them because it was "just statistics" and instead rely on their pre-existing prejudices, which they described as "common sense". Both my own "common sense" and the evidence lead me to believe that people who do this are almost always making a mistake
You're right that I don't like the extrapolation of numbers at all. However, if one had to extrapolate a number, then the Lancet study does so in the correct manner, and is clear about the assumptions made. And my main point was that you can't escape the conclusions of the study by ignoring the extrapolated number; the hard finding is that the death rate rose, and that's not consistent with a view of the world under which the invasion made things better rather than worse.

Mike can't read the authors' minds, no; but this was in their news release:

The researchers found that the majority of deaths were attributed to violence, which were primarily the result of military actions by Coalition forces. Most of those killed by Coalition forces were women and children. However, the researchers stressed that they found no evidence of improper conduct by the Coalition soldiers. [...]

Excluding information from Falluja, they estimate that 100,000 more Iraqis died than would have been expected had the invasion not occurred. Eighty-four percent of the deaths were reported to be caused by the actions of Coalition forces, and 95 percent of those deaths were due to air strikes and artillery.

http://www.jhsph.edu/PublicHealthNews/Press_Releases/PR_2004/Burnham_Ir…

Not only misleading, but beautifully PC - "oppose the war, but support our troops." Not that any of it could be calculated, of course.

If you had been paying attention, GKL, you'd know it's not "their" news release. Which is part of the issue.

D

Mike,
They express their belief that study interviewers will almost certainly be killed if they leave their vehicles in or near Falluja, or if they're found with a GPS unit, it will mean execution. How then, does this reconcile with the study's conclusion that insurgent-caused deaths are inconsequential in the study's apportioning of the 100,000 excess deaths?

A good example of the sort of bizarre argument I was talking about. The majority of people in Iraq are not working for Western organizations, and are therefore not nearly as likely to be killed by insurgents. The survey workers are not representative.
Which is not to say, as someone pointed out above, that the civilians' reports of death rates due to coalition v insurgent violence is accurate- just that your criticism of it here is nonsensical.

Tim & Mike, I apologise for stepping on the line in my earlier post.
Wu

I'm afraid you'll have to enlighten me, Dano, as to why the news release titledIraqi Civilian Deaths Increase Dramatically After Invasionpublished on the Public Health News Center website of theJohns Hopkins Bloomberg School of Public Health announcing the study titled "Mortality before and after the 2003 invasion of Iraq: cluster sample survey" and quotinglead author Les Roberts, PhD, an associate with the Bloomberg School of Public Health's Center for International Emergency, Disaster and Refugee Studiesalong with co-author Gilbert Burnham, MD, associate professor of International Health at the Bloomberg School of Public Health and director of the Center for International, Disaster and Refugee Studiesand noting that joint funder of the study wasthe Johns Hopkins Bloomberg School of Public Healthand providing a media contact from theJohns Hopkins Bloomberg School of Public Healthis not the news release of the study's authors.

I'd also like to know what "issue" this is part of, as your latter point escapes me entirely (probably thanks to my lack of attention).

Heiko,
Hi Carleton, I don't think Mike is an experienced statistician, or that you are either for that matter (correct me, if you are and it's gotten lost in your heated rhetoric).
Well, I used to teach college stats, and I've used them a fair amount at work. But Im certainly not a statistic professor. OTOH, your rather absurd example of nuking Basra demonstrates that you yourself may have lost some objectivity in your pursuit of the matter. You say that the experimental design requires judgement on the part of the researchers, which is based on incomplete knowledge. Perhaps you would care to enlighen us with some examples of similar situations where experimenters possessed complete knowedge of a matter which they still wished to investigate?
That is to say yes, their design is based on judgement calls. But, other than your rather extreme and implausible hypothetical, you have not been able to critique the general design decisions. In fact, you said earlier ...it's not the methodology employed by the Lancet study I am criticising.
Your you don't need to be a statistician commonsense critique fails as well- and, I admit, Im puzzled as to why you would use it. Couterexample: Quantum mechanics makes some odd, counterintuitive statements about the world, but that does not make them incorrect. What I think, statistically, about those matters is that they are explained by a confidence interval that includes 8000k deaths as a floor (surely an underestimate). And the extrapolation isnt from 21 deaths to 60k, but from 21 deaths to 60k+- 55k, roughly. So, for your hypothetical layman, we've extrapolated from 21 deaths to "we're pretty sure there were at least 5000 and probably a bunch more." Since you've decided that we're using the common sense standard, IMO that passes.
Yes, the confidence intervals would be useless if the study were incompetently or intentionally badly designed. No, that doesn't change the criticism of estimates/confidence intervals, unless you've got a criticism of the design. But we've already covered that ground...
So, now you are complaining that Falluja was excluded from the general mortality estimate (you would've prefered that they included an obvious outlier? I can almost imagine your complaints...) but included in examining bombing totals? I can certainly see a rationale (WARNING: This is just my best guess)- while the Falluja cluster totals are an outlier, the individual deaths that occured there may still be representative of the ratios in other areas. They might not be (although Im not sure what the argument would be that they were not representative).Given the risks of gathering data, and the provisional nature of the study, this certainly seems to me to be the correct conclusion.
That's a judgement call that could be critiqued, but IMO all it does is suggest that further study is needed- which is what I believe that authors of the study have said. Certainly, that is no ground for criticism of the intent of the authors. Mike quotes the risks of gather data for his own purposes, but would've preferred that they ran those risks rather than including these numbers in their estimates, because the conclusions irritate his political sensitivities?
If your criticism is with the media's distortions and simplifications, that is a completely different matter. But that is not what is wrong with the study at all, and doesn't require all of the sniping about design decisions etc. The only thing wrong with the study is that it was done cheaply, and at great risk, and therefore doesn't have as much data to rest its conclusions on as anyone would like it to have.

Wu

D Squared:

I think you're trying to have it both ways here, when you state;

"You're right that I don't like the extrapolation of numbers at all. However, if one had to extrapolate a number, then the Lancet study does so in the correct manner, and is clear about the assumptions made."

I'm not sure you can justify a practice you find unsound in principle and theory by claiming they (the Lancet) still made the best of it.

What can be gained by extrapolating " a number " that may not be accurate or representative in the first place? Aren't you compounding the initial error, possibly by many thousands in some cases, regardless of confidence intervals? I see no reason why a single study sampling,in an environment like Iraq, and resulting in such small numbers, can inspire the confidence necessary for extrapolation. And this study is no exception, its results are all over the map, and largely because the small numbers involved make it prone to skewing and contradiction.

For example, if we take 100% of the sampling, the cause of death that eclipses all others among the deaths found post regime change is coalition air strikes (58). Forget about adding in the non-coalition caused violent deaths or coalition caused deaths other than air strikes. Coalition bombings alone account for 40 % of all deaths, and is a figure 3 1/2 times higher than the second leading cause of death post regime change, heart attack (18).

On the other hand, if we only consider the results from 97% of the survey, and exclude Falluja, air strikes now account for only 1/3 as many deaths (based on a number of 6 or 7) as those occurring from heart attacks. By eliminating 3% of the survey, we radically alter the figures that could be used for extrapolation. Even as a statistician, you surely must understand the frustration a lay person like myself has in seeing these numbers being extrapolated for all of Iraq.

D Squared, I realize I'm not a mind reader, but the circumstantial evidence that points to dishonesty on the part of the authors in how they framed their results is obvious. They provide us with much detail concerning the breakdown of the death figures, yet inexpicably leave out the vital information concerning where to place the non-coalition deaths. This is simply too much of a coincidence. I'd appreciate it if you'd offer an opinion of some substance concerning this, and also the potential impact it could have on the authors' contention that coalition forces accounted for most of the violent deaths. While I'm at it, I've yet to hear anyone even try to come up with a defence for the authors' chicanery on the Falluja numbers in relation to the genders of the victims. What is your take on this?

I agree with you that the death rate has risen in Iraq following regime change. The question is, by how much and at whose hands? I maintain this study is not adequate in and of itself to provide reliable answers to these questions. My belief is that the deaths of insurgents and Iraqi troops during invasion, combined with the deaths inflicted by insurgents, dwarfs the number of civilian non-combatants killed directly by the coalition. To what extent, who knows? We have no way of knowing the reliability of the American casualty figures for insurgents, and the pentagon has been very reluctant to provide an overall number.

I find the air strike figures and resulting extrapolation to be extremely prone to variation from study to study, and the air strike death toll is heavily relied upon by those who defend this study. It has to be, when you have airstrikes dominating the violent death figures. Surely you must see this. Only 5 of the 33 clusters (Falluja included) report bombing deaths. We're left with a death toll of 6 or 7 in the four non-Falluja clusters, and by your own admission, there's a possibility some of these deaths could be non-coalition. There's clearly huge differences between the extent and type of bombing in Falluja as opposed to most of the rest of Iraq. When your number one cause of death is whittled down well into the single digits, what business does a statistician have making extrapolations for an entire country? As I've stated before, further identical studies might deliver a clearer picture, but I suspect we'd end up with radically different figures across the board, from study to study.

Carleton:

Apology accepted. Please accept mine. Concerning your point that the insurgent caused deaths are pegged about right by the survey, I disagree. The insurgents are certainly killing many Iraqis intentionally, but we've also seen many others killed in car bombings and mortar attacks intended for the coalition, that missed their intended targets. We also don't know how many of the criminal homicides are in fact targeted political killings by the insurgents. We have no way of determining how many Iraqis were killed during the actual invasion, when used as human shields by the Saddam Fedayeen. I believe the overall number is much higher than you think.
Donald:

Any way you slice it, the air strike numbers are extremely unreliable in my view. It's clear that Falluja was extensively bombed, but as I've mentioned before, bombing was much more limited in anti-insurgency actions elsewhere. D Squared's explanation about mortars aside, if the coalition is going to be accused of indiscriminately dropping 500 pound bombs uniformly across Iraq, to sell a high extrapolated death number, then in fairness there should be some corroborating bombing deaths of note in the non-Falluja clusters. I don't see how 6 or 7 deaths in only 4 of the 32 non-Falluja clusters provides that.

Hi Carleton,

if we have a 6 sided dice, and role it 100 times, I can give you exact probabilities. The chance that the die throws add up to at least 100 is 100%, as is the likelihood that they won't exceed 600. And, I can calculate with perfect accuracy what the likelihood of getting any number in between would be.

Some commentators appear to be trying to make it sound, as if you could do this with the 8000-200,000 or so figure claimed for Iraqi excess deaths. We can't.

You appear to misunderstand me, when I say that you don't need to a statistician to be able to make valid arguments. Valid arguments can be counter-intuitive, and they can be made by non-statisticians.

I think that the authors don't make it clear that they are in fact including Fallujah for the distribution of deaths.

I also, and co-incidentally, heavily disagree with their and your judgement on the matter.

I would have thought that the interviewers that actually went into Fallujah were in fact Iraqis, not least because European looking foreigners would have had to be very brave indeed to interview anybody in Fallujah.

Even though I believe them to be Iraqi though, I think they had good reason to be considerably more afraid of insurgent activity than being killed in American airstrikes.

It's not at all absurd by Mike to bring this up. The Lancet authors can be critiqued for being too pro-insurgent, at the very least by being silent on points where they should be vocally condemning them, but aren't.

Donald johnson wrote: "So we're supposed to assume the devastated one was a statistical freak though the authors say many other unsampled neighborhoods appeared equally hard hit and I'll repeat, the neighborhood in Sadr City showed no casualties and that doesn't seem to raise eyebrows."
don, you are missing the point. Cluster sampling is very prone to giving large errors when you have a non-homogenous population, especially when coupled to low sample numbers. Unexpectedly low and high results are the two sides of the same coin; the basic is that cluster sampling is prone to systematic error. This is textbook stuff and well described.
j

Wu wrote: "your rather absurd example of nuking Basra demonstrates that you yourself may have lost some objectivity in your pursuit of the matter"
Not at all; Heiko's point is absolutely spot on. It is that cluster sampling is unreliable in the presence of a non-homogenous population, especially with low sample numbers. If you have clusters of high death rate, whether or not you over- or under- sample those clusters will have an enormous effect on your final average estimate of death rate.
small-scale cluster sampling is very unreliable, because you have no idea whether you are picking up a representative number of high death rate areas.
Not only is this widely accepted, you can create statistical models which demonstrate how this effect works very easily.
j

Heiko,
While some commentators may be making the study's numbers sound as absolute as you say, I don't believe that the study itself has done so- so I don't find that to be a legitimate critique of the study. Again, if you have a problem with the way that news media simplify and distort, welcome to the club. But that doesn't make the media pro-insugent (just lazy, and feeding the desires of their audience for simple stories), and it surely doesn't make the authors pro-insurgent.
Agreed, non-statisticians can make valid arguments, and valid arguments can be counterintuitive. What you suggested was that Mike's intuition was correct, but you failed to back that up with any statistical reasoning (other than the perfect knowledge red herring). That was neither- it is an attempt to use intuition as a substitute for argument.
Im curious as to why you think that the Fallujah death ratios would be so unrepresentative of death ratios overall that they ought to be excluded. What's your rationale for disagreeing so strongly?
Im not sure if your point about Caucs in Fallujah is directed at my earlier criticism of Mike's ancedotal 'evidence' that insurgent deaths must be higher, or if it's supposed to be an argument that gathering additional evidence wouldn't have been that risky. If the former, I was addressing that when I said that those workers would be in danger bc they were working for Western organizations- so I agree that Iraqis working for Caucs would be endangered as well. I just don't think that this is representative of the risks faced by most Iraqis.
And, the crux of the matter. I think that it would be perfectly reasonable of the authors to condemn the insurgents' tactics, but I cannot condone you making it a requirement that they do so or be labeled pro-insurgent. I believe that it is perfectly possible, and furthermore desirable, for the authors to merely be doing the best possible job to answer the question without expression bias towards either side.
I think your position is driven by an ideological need to find something to complain about in the study- to the point that you raised a critique (that they lacked perfect information before designing the study) that would not just apply to the actual Lancet study, but to any study performed at that time, regardless of design. That suggests to me that it is the very performance of such a study that irks you so, not any specific point about statistical methodology. Or, rather, it is not the study's bias that bothers you, it's the lack of bias against the insurgency.

Mike,
Concerning your point that the insurgent caused deaths are pegged about right by the survey, I disagree.
I very clearly said before "Which is not to say, as someone pointed out above, that the civilians' reports of death rates due to coalition v insurgent violence is accurate..." That is, Im not trying to claim that the survey's results were correct, just that they were consistent in metholodogy and didn't demonstrate any bias. Personally, Id like to see some other methodologies used in addition to asking surviving relatives to examine the same question. After that's been done, maybe we'll have more confidence in the resulting estimates.
You also said: I believe the overall number [of insurgent-caused deaths] is much higher than you think and My belief is that the deaths of insurgents and Iraqi troops during invasion, combined with the deaths inflicted by insurgents, dwarfs the number of civilian non-combatants killed directly by the coalition.
Here is the fundamental difference between us- I really don't have a strong opinion on the matter, and I don't think Im entitled to one. Nor are you, IMO. In fact, insofar as I have any suspicions, I share yours that these deaths were underreported by the study. But I wasn't there & the reports I've read have been all over the map & of varying reliability, so I really don't have the grounds for much of an opinion. An opinion at this stage of knowledge is more likely to be reflective of one's personal preferences than based on any serious analysis of the data. The fact that you choose to react to a study that challenges that opinion by attacking the motives of the authors demostrates exactly that sort of thinking.
Wu

Mike, you are also trying to have it both ways. When you start saying that you think this or that percentage of the deaths were "non-coalition" or "insurgents", then that's an extrapolative process, and one with a much weaker base than the base for the 100,000 number. If you are really serious about this critique, then let's just stick to talking about risk ratios. The Lancet study finds a risk ratio for total deaths associated with the Iraq war that is greater than 1 with more than 97.5% confidence. Nobody has seriously criticised this result, and it's important.
You are right; you're not a mind reader. Most of your "circumstantial evidence", if you look at it objectively, simply comes from the fact that the Lancet team were interested in establishing objective facts. Deaths are facts; someone's status as an "insurgent" is a judgement. Half of the statistics that you wish they had put into the paper are only relevant to your own judgements about the coalition's culpability, not to the subject of the paper which is the public health impact of the invasion of Iraq.
Finally, your belief that "the deaths of insurgents and Iraqi troops during invasion, combined with the deaths inflicted by insurgents, dwarfs the number of civilian non-combatants killed directly by the coalition" does not appear to be supported by the survey responses. It is also not obvious to me that in this sentence you are aware that the two categories you are talking about (violent deaths caused by insurgents and violent deaths caused by the coalition) do not exhaust the total increase in the death rate. You've left out the increase in non-violent deaths and the increase in criminal murders.

Hi Mike, I never meant to imply that you lacked a degree in statistics (I can't know whether that's true or not). Nor did I mean that if you do, it disqualifies you from expressing concerns. I did refer to some of your earlier remarks as "illiterate". Those words were uttered in the heat of the moment when I was irritated at the tone of your earlier posts. They were poorly chosen and I apologize for them, and my other heated comments that followed. It's up to me as much as you or anyone else to keep this blog civil and professional. BTW - For what it's worth, I don't have a degree in statistics either. I do have engineering and physics degrees that required me to take it, and I do know something about it. But I don't consider it my strong point, which is why I defer to Tim, Carleton, dsquared and others who are more knowledgeable than me when I'm in doubt. I'll address your comments specifically to me in posts after this one to keep things cleanly spaced.

One other thing - to put in a paragraph break, just type <"p"> at the end of a sentence (without the quotation marks) and it will put in an HTML line break. <"p"><"p"> will put in a space between paragraphs. Better yet, you can use <"br"><"br"> (again, without the quotation marks) because for what it's worth, the P Tag has been deprecating in HTML 4.

Hi Mike, Here are some thoughts on a few of your recent comments;

1) "the authors recount their concerns and experiences over the inherent dangers in conducting the studies. Yet all the dangers they describe are created by the insurgents. How then, does this reconcile with the study's conclusion that insurgent-caused deaths are inconsequential in the study's apportioning of the 100,000 excess deaths?"

Easy! The overwhelming majority of non-combatants in Iraq at present are Iraqi and as a general rule, most insurgents who are waging war against an invasion are doing so because they want to kill the invaders, not their own countrymen. Now a blonde haired, blue-eyed guy in a Washington Redskins tee-shirt, walking down the streets of Falluja carrying a GPS unit and a notebook looks a helluva lot more American than Iraqi, wouldn't you agree? In addition, Iraqi investigators who were carrying GPS units might easily be mistaken for Iraqi police supporting coalition forces. Hence, the danger. But there is certainly no reason to believe that insurgents want to kill anyone other than invaders or those perceived to be helping them. The reason that 2 out of 73 are reported as Insurgent related and the others are not, is because that was what the gathered data said - and data speaks a lot louder than "common sense" (which Einstein once defined as "the repository of prejudice instilled in a person by the age of 18").

2) "the coalition is responsible for the vast majority of all violent deaths, the insurgents virtually none"

This is false. The study never said insurgents caused no casualties. You yourself properly quoted it as identifying 2 of 73 death as insurgent related. As for the majority of deaths being caused by the coalition, one would think that the side with the Tomahawk cruise missiles, F-111 bombers, F-18, Panavia Tornado, and F-15E Strike aircraft, 2500 lb laser-guided bombs, Apache attack helicopters, A-10 close-support aircraft carrying 30 mm rotary cannons loaded with depleted uranium and armor piercing, exploding rounds, a bewildering suite of other air-to-ground weapons, tanks, Bradley fighting vehicles, and satellite intel would inflict more casualties than a bunch of local insurgents holed up in mosques and urban pockets with small arms and mortars. Now that's "common sense". Furthermore, Roberts et al. report that of 61 fatalities attributed to coalition forces, 58 resulted from attacks by helicopter gunships, rockets, or other forms of aerial weaponry. These ar, in fact, air strikes, and 58 out of 61 is a majority.

3) "the coalition is responsible for the vast majority of all violent deaths, the insurgents virtually none"

Again, false. The study said no such thing. You yourself properly quoted it as identifying 2 of 73 death as insurgent related.

4) "In the interview of Roberts reported in Medialens, as mentioned in this thread by Donald Johnson, Roberts reveals that 5 clusters reported bombing deaths. This is information that should have been readily available in the study report itself. While the authors still withhold several other key pieces of the puzzle, the revelation concerning the dispersal of the bombing deaths allows, through the process of elimination, a clearer picture of the breakdown of the non-Falluja deaths. It now appears likely that at least 10, and possibly all 12 of the violent deaths not inflicted by the coalition are contained in the 21 non-Falluja violent death toll."

Rule No. 1 - stick to the data, not anecdotes. Reports of interviews in Medialens from elsewhere in the thread about anecdotal comments (for which we have not been provided the context) are no substitute for the actual study data. Furthermore, even if this is true, it's irrelevant. The study gave no specific statistics about air strikes. It says only that violence accounts for a majority of the excess deaths after the invasion, and that air strikes account for a majority of the violent deaths. You have not demonstrated how this is inconsistent with bomb related deaths in 5 out of 33 clusters.

5) "It can also be reliably estimated that 6 or 7 is the number of deaths resulting from air strikes, spread among the 4 non-Falluja clusters that reported such deaths."

This is false. There is no data in the study or anywhere else to support this, or to support the claim that it is based on (that only 5 clusters reported air strikes). If you think otherwise, cite data that supports this, not anecdotes or someone else's context-free remarks.

6) "More importantly, the authors claim that 95% of the coalition caused deaths were from air strikes. But in the 97% of Iraq that makes up the survey sampling excluding Falluja, only 12% of the clusters reported air strike fatalities, and none of these clusters seems to have reported more than 2 killed."

Yet again, every word of this is false. A simple search of a PDF version of the paper for "air strike", "violence", or "coalition" turns up every remark the authors made about any of them. Nowhere does the paper say the 95 percent of coalition caused deaths were from air strikes. It simply says that a majority of excess deaths were the result of violence, and that it is likely that a majority of violence caused deaths were from air strikes. Nor does the study report at any point that 12 percent of the clusters reported 2 or less air strike fatalities. Once again we see the basic moral of the story being illustrated - stick to the data, not anecdotes.

7) "Scott, you're playing the same dishonest game as the study authors, including the Falluja death figures when it suits you, then discarding them when it doesn't. The 61 violent deaths you refer to include Falluja."

It seems that you haven't read my last post here, dated 2/12/2004. Heiko already pointed this out, and I readily acknowledged to him that I had done so. I then proceeded to make the same basic point using the 21 violent deaths recorded outside of Fallujah. Please read that post for details of that argument. The reason the Fallujah data was unreliable for extrapolation is due to, yet again, a principle of statistical mathematics. If a set of data points tracking some variable has, except for one or two points, a relatively small scatter, those points are called outliers. If you have a room full of 20 or 30 people who work minimum wage jobs, for instance, and Bill Gates walks in to room, now the average room income is $1 billion. Thus, our "statistics" tell us nothing about anyone in the room. Look at Roberts et al.'s Figure 2 (Page 5) and look at what the blue data (Fallujah) will do to the standard errors (and therefore, to the confidence interval) in April and August of 2004. Fallujah is an outlier that adversely impacts the standard errors and the characterization of the data. This has nothing to do with what "suits me" - it's mathematics, and it's unavoidably at the core of studies like this one (which is why "common sense" isn't good enough).

8) "This doesn't reflect the scenario being created by the authors and those who have hopped on their bandwagon, which is one of widespread, indiscriminate aerial bombing, obliterating civilian neighbourhoods and killing the non-combatant occupants of these neighbourhoods by the thousands."

No such statements are made anywhere in Roberts et al., and there have been no bandwagons started by the authors. In fact, they have been relatively silent about their work. They put it out there, with their methods, properly cited as all good work is, and left the arguing to others. The only "bandwagons" I see are those being sold at Right-Wing think tanks, editorials, and blogs trying to shoot the study down without looking at the methods. Me? Like most other folks at this blog, I'm just pointing to what the paper actually says and the methods it cites - which can be checked by anyone willing to do the work.

9) "I'll ask you straight out. If the authors tell us the Falluja data is unreliable for extrapolation, but they're confident that AT LEAST 100,000 excess deaths have occurred without the Falluja data, are the authors being forthright and honest in suggesting that most of these 100,000 excess deaths that were caused by coalition violence were women and children?

And I'll answer you straight out - every word of this is either false or irrelevant. The authors did not say that "AT LEAST 100,000 excess deaths" have occurred outside of Fallujah (why do so many folks think that stating things in ALL CAPITALS adds force or credibility to an argument?...). Yet again, back to the stated data and conclusions that are actually in Roberts et al. The authors estimate that conservatively, 98,000 excess deaths have happened in Iraq, excluding Fallujah, since the 2003 invasion of Iraq, violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths. Since the authors never said what you claim they did, there is no implication regarding their honesty. If you think otherwise, show me where the paper says so, or better yet, produce the actual data you think they're hiding (note the word data, not anecdotes, hypotheses, conspiracy theories, alleged "agendas real or imagined, or quotes from other bloggers or editorialists). As for women and children comprising most of the violence related fatalities, forthrightness of suggestions are beside the point - the data are stated clearly in their Table 2 (Page 4) and in their initial Findings and Interpretation. They did not say that a majority of deaths outside of Fallujah were women and children. They stated in their overall findings, and after having discriminated Fallujah from non-Fallujah data, that a majority of violence related deaths were women and children. Table 2 shows that there were 73 total violence related deaths including Fallujah. Later they state that 61 were coalition caused, and of these, 28 were children, and 4 were women. Simple math yields a majority of women and children, for their all-inclusive data. The table does show, quite clearly, that outside of Fallujah, women and children did not comprise a majority of violence deaths. Where's the dishonesty and the big propaganda conspiracy in that? All one needs to do to see what they are and are not saying is.. read the damn paper.

10) Quoting me first, you state that, " 'Finally, it has been my experience at least, that most of those who attack this study are ultimately out to justify the U.S. led invasion of Iraq. Thus, they want to show that Iraq is happier, healthier, and better off today than they were in 2002, and massive nationwide carnage does not fit too well with this agenda.' Well Scott, it has been my experience at least, that most of those who defend this study are ultimately out to discredit the U.S. led invasion of Iraq. So what the hell is your point, exactly? I believe that deaths have increased in Iraq since regime change. My objection to the Lancet study is that it is not an accurate depiction of the number or character of these excess deaths. How can it be, when it refuses to categorize a single violent death as that of an insurgent or terrorist, when the Americans estimate they have killed 25,000 of them? Things are not as they should be in Iraq, and may never be. There was no reasonable prospect that Saddam's Baathist/tribal power base was going to be internally overthrown in the near future. Any attempt at overthrow would certainly have resulted in a protracted civil war, costing the lives of hundreds of thousands of Iraqis, with no guarantee of success."

Yes Mike, many who defend the study are out to discredit the U.S. led invasion - and guess what, massive nationwide carnage does a pretty damn good job of doing that! Particularly when there is a well characterized cohort study with statistically sound methods behind that conclusion, and the best anyone can come up with against it is highly vocal "common sense" with no demonstrated flaws in the methods or data. That is "the hell" my point. Discredit the invasion? I don't need to. I can just point to the data in this study, and to the fact that G.W. Bush didn't even bother to read his own damn intelligence about the place before going there. It would be nice to think that he'd done at least some homework before putting American troops and Iraqi civilians through the meat grinder. The Lancet study does not refuse to categorize a single violent death as insurgent related. It classifies at least 2 in the starting data from which extrapolations were made. An attempt at overthrow would have caused a "protracted civil war"? What exactly do you think is going on their now? A company picnic? Hundreds of thousands? That's exactly what this study is coming up with! And it stands until someone - anyone - gets past "common sense" and takes down the math and the methodology with specifics.

11) "the Lancet seeks to place the blame for their victims' deaths on the Americans as well. In defending their study, be aware Tim and Scott, that you're doing the exact same thing"

What we're doing Mike, is the one thing you've refused to do so far - sticking to the actual data and methods and applying known principles of statistical science to them, as opposed to hand-waved guesses and/or hunches based on "common sense" that make little use of either. The methods used by the authors for their study are well known and clearly stated. Pages 5 through 7 discuss the strengths and weakness inherent in what they did and their rationale, including citations to similar and/or related works. The 98,000 excess deaths figure was derived using standard methods for such extrapolations, with other controls to help characterize the confidence intervals, etc. Relative risks were evaluated assuming a log-linear regression in which every cluster was allowed to have a separate baseline rate of mortality that was increased by a cluster-specific relative risk after the war, and the variation in relative rates was accounted for by allowing for overdispersion in the regression. These methods are described in detail in McCullagh and Nelder (1989). Average relative death rates were estimated with conditional maximum likelihood methods as described in Diggle et al. (2002). The bootstrapping methods used to obtain the confidence interval and the death rates (But with linear rather than log-linear regressions) are described in Efron (1979), with examples. The pre-war infant mortality rates they arrived at using their methods were checked against infant mortality rates under similar conditions in neighboring countries and agreed well with those estimates (WHO, 2004). That's for starters. Other sources from the paper characterize other methods they used and other data sources that can independently be compared with their own. This isn't "blame placing" Mike, it's sticking to science rather than empassioned pep-rallies.

I could go on and on. In fact, I already have, and I'm really only scratching the surface. There's a recurring theme here of loud, passionate claims based on "common sense" that cannot be verified anywhere in the actual Lancet paper, its data, or in any of the known and published mathematical methods required for such studies - the sort that can be found in any undergraduate statistics textbook. The reason Tim and I have "a steadfast determination to confine the debate within a statistics bubble" is because this is a statistical study - and I'm sorry to say Mike, but there is no getting around the mathematics and a thorough examination of the data characterization methods. This is science, not drama. A woman I work with has a sign up at her desk that I like. It say simply' Without reliable data, I'm just another person with an opinion." That is true. The devil is in the details, and all the passion, patriotic fervor, and "common sense" on this Earth are not worth the paper they're printed on if they can't demonstrated from the data and mathematics. Do you think that "the very low numbers of deaths from this study are not conducive to accurate nationwide extrapolations within the context of Iraq?" Then demonstrate it. The citations to the main methods of Roberts et al. are below. Read them and show where they're flawed. Were the deaths over-reported rather than under-reported? Then read their discussions about that on pages 5 through 7 of the paper and provide other data to refute it. Avoid conspiracy theories - they're well suited to secret societies, the Art Bell show, UFO magazines and the like, and the like, but they don't carry much weight as proof of anything. Mike, I appreciate your desire to see this sort of study done right. I share it too. But sooner or later, you're going to have to put flesh on the bones and do the work.

Without reliable data, I'm just another person with an opinion - however loud that opinion might be. Best wishes.

References

Diggle PJ, Heagerty P, Liang KY, Zeger SL. Analysis of longitudinal data, 2nd edn. New York, NY: Oxford Science Publications, 2002.

Efron B. Bootstrap methods: another look at the jackknife.
Ann Stat 1979; 7: 1-26.

McCullagh P, Nelder JA. Generalized linear models. London: Chapman and Hall, 1989.

WHO. Health action in crises. http://www.who.int/disasters/stats/baseline.cfm?countryID=62 (accessed by Roberts et al. on Oct 16, 2004).

Scott, most of your points look very correct to me, but the team members who did the fieldwork in Iraq were Iraqis.

Thanks dsquared. I did see that Iraqis led those teams, but I thought (incorrectly apparently) that at least some of the field workers were western, so I covered that base.

Scott wrote: "there is no getting around the mathematics and a thorough examination of the data characterization methods"
it is strange you say that, because there are several elements of the study that you are very reluctant to discuss
1) 19% of their attempts at verifying deaths failed
2) cluster analysis is known to be prone to systematic bias
3) the numbers involved are small, and this makes the final extrapolation less certain
4) the authors undertook post-hoc data manipulation
I just can't imagine why you don't want to discuss these issues, and their implications for the credibility of the study.
as another issue, what happens if you extrapolate their fallujah death counts up for the total population of fallujah ? Does it really give a number of 200 000 deaths ? Wouldn't that suggest that the methodology has gone wrong ?
j

Scott, D Squared:
My apologies for the delay in responding. I'm back to work after a week off, so the time for blog debates has decreased dramatically until my next days off. Scott, thanks for the kind words, and please accept my apology as well. Some of the blogs I frequent are bare knuckles forums, and bad habits are easy to pick up. And thanks as well for the paragraphing tip. My posts should be easier to decipher now.
The first thing I want to mention is something that struck me when I looked at the death table in the study again. We've been assuming the authors are correct when stating on page 1 that " Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths." Forget for the time being the air strike portion of the quote. The premise that violence accounted for most of the excess 100,000 deaths the authors are claiming occurred post invasion has been part of the foundation for much of the defense of the Lancet study, both here and elsewhere.
However, the post invasion death figures show that in the 97% of Iraq surveyed without Falluja, violent deaths do not exceed those of other causes. With Falluja excluded, we see 89 deaths reported post regime change, and 46 recorded prior to regime change. The difference in actual deaths is 43. 21 were from violence, while 22 were from non-violent causes.
Am I missing something here? Are the authors basing their claim that more deaths were caused by violence solely on an extrapolation derived from the much higher risk rate of death by violence after invasion? Is this a statistically sound practice, given the small numbers involved, and the fact that the violent deaths recorded are not the majority of the actual excess deaths recorded?
The second issue I want to address is my stubborn insistence that the distribution of the violent deaths not attributed to the coalition has a huge bearing on the credibility of this study. I've spent a lot of fruitless time searching on the Internet for any interviews where the authors might have revealed information concerning these 12 deaths and their distribution.
It is possible that all 12 violent deaths that were not coalition inflicted were recorded from the non-Falluja clusters. If this is the case, then the authors' generalized statements concerning the characterization of the violent deaths, and their accompanying extrapolation, are seriously compromised. Based on the known non-variables concerning the composition of the entire number of violent deaths in the study, at least 7 of the 12 non-coalition violent deaths are definitely from the non-Falluja clusters (I explain this further down).
Here are some examples of the problems the placement of these 12 deaths would cause for the study:
If the 12 violent non-coalition caused deaths are all outside of the Falluja cluster, then they outnumber the coalition-caused violent deaths by a 12 to 9 margin. On what basis then, can the authors claim, as they have, that " Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths." Without Falluja, this is simply a false statement, if the 12 non-coalition deaths are outside of Falluja. It is beyond dispute that the authors have offered up the 100,000 excess death figure while explicitly omitting the Falluja data. Roberts did so again in the e-mail to Medialens, stating "It happens, that the one place with a lot of bombings, Falluja, and we excluded that from our 100,000 estimate....thus if anything, assuming that there has not been any intensive bombing in Iraq."
If this is the case, violent deaths not caused by the coalition actually may outnumber violent deaths from air strikes by nearly 2 to 1 in the 97% of Iraq surveyed outside of Falluja. This is a huge contradiction of the " Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths," statement, and what are the repercussions to the assumptions the authors make?
We also have to consider how unrepresentative of events on the ground the data becomes if we assume these 12 deaths are non-Falluja deaths. To set the stage for this, you have to refer to the death table in the study.
If all 12 deaths are outside of Falluja, that means all 52 deaths in Falluja were bomb related, and that leaves only 6 bomb deaths possible for non-Falluja clusters. The best I can figure, and I can't do it with any certainty, but 4 of these would be children, 1 woman, and 1 man. We have 3 coalition shooting victims plus the 12 non-coalition victims that would then make up the rest of the 21 non Falluja violent death toll.
Obviously, 5 of the bombing victims (the women and children) and all 12 of the non-coalition victims have no chance of being insurgents. 2 of the 3 shooting victims were described in the study as being definitely non-insurgent. That leaves only the 56 year old man who was shot and who "may have been a combatant," and a male adult bombing victim whom we know nothing about. My point is, we may well have zero deaths out of the 21 violent deaths occurring in all of Iraq outside Falluja that can be categorized as an insurgent. In the casualty distribution I propose, the authors have a tough time using the "could have been combatants " equivocation, because only 2 have any chance of qualifying, and 1 (the bombing death) could just as easily be a non-combatant. We simply have no idea.
I've been complaining throughout this thread that the authors won't commit to any insurgent deaths, when all along, in the violent death figures that really count, the non-Falluja deaths, the authors may well have had only 1 likely candidate to work with, and he's a maybe!
So what does this do to the credibility of an extrapolation using the 21 violent death number? We know the coalition has killed many thousands of enemy combatants. As I've already mentioned, having crunched the numbers again, I now realize that the study numbers may actually preclude all but 1 or 2 having any chance of being labeled as insurgents. It all depends on where these 12 non-coalition deaths occurred. My question to both of you is, can we have any faith in a nation wide extrapolation of violent deaths from the 21 number if it isn't capable of generating any insurgent deaths other than a single " maybe," and not because the authors are consciously avoiding it, but because the study's own data doesn't allow it! Even if we allow the 56 year old man to be counted as an insurgent death, he's 1 out of 21. This in no way can be accurate for overall insurgent deaths if extrapolated to a nationwide figure.
If this problem arises with insurgent deaths among the 21, why can we express confidence in the other numbers that make up this 21 figure, such as the 6 bombing deaths, the 3 shooting victims, etc? Because that's exactly what the study and its defenders are doing. The 21 figure is still just the sum of its parts. If there are questions concerning the reliability of elements of the 21 figure for extrapolation purposes, how can we have confidence in an overall extrapolation, working out to around 60,000 excess violent deaths? What does this do to the confidence that a 100,000 excess death number can be extrapolated from data that the authors tell us is derived from a belief that coalition air strikes killed more people violently than any other cause, if air strikes killed only half as many people as non-coalition causes?
In conclusion, I feel very strongly that this 12 non-coalition issue needs to be addressed, because it has the potential to seriously undermine this study, for the reasons given above.
Scott, D Squared, I'm wondering if you could reply in two respects to the issues I raise concerning the 12 non-coalition deaths. First, I'm asking you to set aside entirely the question of whether all 12 non-coalition violent deaths occurred outside Falluja. Could you then provide your opinions in relation to the arguments I make concerning the harm I believe this causes the study's integrity if we assume that all 12 (actually even 11 still accomplishes essentially the same thing) non-coalition inflicted deaths occurred in the non-Falluja clusters. Second, could you comment on the likelihood that all 12 (or even 11) of these deaths are in the non-Falluja clusters.

Given the length of your post Scott, I'm not sure where to start, and I don't have the time to cover everything. I guess I'll begin with what surprises me the most, your pique over my reference to the Roberts interview from Medialens, concerning the number of clusters reporting bombing deaths. I'm not sure what surprises me more, your questioning whether it is genuine, or your assertion that it is irrelevant, and because it is not included in the study's text, it is somehow out of bounds in the context of this debate.
Here is the relevant text from medialens, along with the link to the entire article:
We again raised these queries with the report authors. Dr. Roberts replied:

"Point 1 is true and it is not a mistake on our part. We would have had a more accurate picture if we conducted a 'stratified' sample, with some in the high violence areas and some in the low violence areas. But, that would have involved visiting far more houses and exposing the interviewers to even more risk. Secondly, we do not know how many people are in the 'high violence' areas, so this would have involved large assumptions that would now be criticized.

"Most samples are taken with the assumption that all the clusters are 'exchangeable' for purposes of analysis. The difference between them is considered in the interpretation of the data.

"Point two, assumes bombing is happening equally across Iraq. There is no such explicit assumption. There is the assumption that all individuals in Iraq had an equal opportunity to die (and if we did not, it would not be a representative sample). It happens, that the one place with a lot of bombings, Falluja, and we excluded that from our 100,000 estimate....thus if anything, assuming that there has not been any intensive bombing in Iraq.

"Finally, there were 7 clusters in the Kurdish North with no violent deaths. Of those 26 randomly picked neighborhoods visited in the South, the area that was invaded, 5 had reported deaths from Coalition air-strikes. This, I suspect that such events are more widespread than the review suggests." (Email to David Edwards, November 1, 2004)

It apparently is not an interview, but an exchange of e-mails between Roberts and the Medialens journalist, David Edwards. The e-mail seems to be verbatim, and there is no question that Roberts is clearly stating that 5 clusters experienced deaths from aerial bombing. I see no grounds to believe this e-mail from Roberts isn't authentic. Medialens is staunchly defending the Lancet study, there is no reason to believe they would mis-report anything Roberts said. This is clearly beyond an " anecdote," it is " actual study data," whether it found its way into the study or not, and I see no reason why it should be dismissed.
"The study gave no specific statistics about air strikes. It says only that violence accounts for a majority of the excess deaths after the invasion, and that air strikes account for a majority of the violent deaths. "
Scott, the study does give us specifics about air strikes. It tells us that 58 of the 61 coaltion caused deaths were the result of aerial bombing.
This leads me directly to your point number 6. I won't repost it, except to note you claim " every word " is " false." This is simply not true. As you can now see, 58 deaths out of the 61 attributed to the coalition is 95% of that particular death toll. Four clusters (the non-Falluja ones reporting bombing deaths) out of 32 is 12% of the clusters. The only point you may be correct on is my calculation that those four clusters experiencing bombing deaths outside of Falluja all experienced no more than 2 such deaths. That may not be correct, for the reasons I give below.
Why would you suggest that anything providing a clearer picture of the distribution of deaths in this study is out of bounds? Knowing that there are 5 clusters reporting bombing deaths provides another non-variable element concerning the distribution and interpretation of the violent death figures.
The importance of this rests with my argument made above that the composition of the 21 violent deaths in the non-Falluja clusters has a great bearing on the study's integrity for purposes of extrapolation.
By knowing the number of clusters reporting bombing deaths, along with the data the study provides concerning the distribution of deaths of coalition culpability (28 men killed, 26 by bombing, 2 by shooting; 28 children killed, all by bombing; 4 women killed, all by bombing; and 1 elderly man killed by shooting), there are enough non-variables now established to state the following:
At least 48 of the violent deaths in Falluja have to be from coalition bombing. That means the maximum number of bombing deaths in the 4 non-Falluja clusters is 10 (to total the 58 deaths attributed to bombing). These two figures (48 and 10) are only possible if each of the 10 clusters reporting a non-bombing violent death recorded only 1 violent death each, and no more. They are also only possible if none of the 3 coalition shooting victims are from the Falluja cluster. The number of Falluja bombing deaths has to increase by 1 (with a corresponding decrease of 1 in the non-Falluja bombing toll) for every non-bombing cluster that reported more than 1 violent death. If any 1 of the 3 coalition shooting deaths shows up in the Falluja numbers, it automatically raises the number of non-coalition violent deaths in the non-Falluja clusters by 1.
I realize this is convoluted, and regret having to go into this kind of exercise, when the authors could simply have told us where the non-coalition violent deaths occurred.
The point I'm trying to make with the above is, at least 8 of the 12 non-coalition caused deaths have to be recorded among the 21 non-Falluja violent deaths. I believe it is higher, because the only way it isn't higher is if the Falluja violent deaths contain none of the 3 violent deaths caused by coalition gunfire, and the other 10 clusters recording violent deaths each recorded only 1 violent death exactly. I don't think this is likely, so the number 8 is probably low.
I'll just briefly touch on some of your other points, Scott:
Your point number 1, concerning the number of insurgent caused casualties and my belief that the study underestimates their numbers. All I can say in this regard, is it sure seems to me the insurgents are killing a lot of their countrymen, there were 30 more yesterday, and a 3% figure pegged from the study data seems low to me. I see no way of resolving this, as much of it rests on media reports and perception.
Your points number 2 and 3, concerning your objection to my statement;"the coalition is responsible for the vast majority of all violent deaths, the insurgents virtually none"
A poor choice of words on my part. 3% isn't "virtually none," but I think you'll agree there is no comparison between 61 deaths versus 2 deaths.
Your point number 7, concerning my complaint about your use of the Falluja data. I had posted it as a re-post of the one Tim deleted. In the meantime, Heiko and you sorted this out. My post came after, if I had read the others, I would have noticed this was already addressed. I understand the outlier concept, it seems to be one of the few easily grasped principles of this wonderful science of statistics.
Your point number 8 involves this statement from me:
"This doesn't reflect the scenario being created by the authors and those who have hopped on their bandwagon, which is one of widespread, indiscriminate aerial bombing, obliterating civilian neighbourhoods and killing the non-combatant occupants of these neighbourhoods by the thousands."
Many of those defending the Lancet study are trying to create the impression that an excess violent death toll of 60,000 can be reliably extrapolated on the basis that the number one contributer to this number is coalition air strikes. My point with the above statement is, if the death by bombing numbers are as low as I suspect they are in the 4 non-Falluja clusters, it calls into question whether we're seeing 500 lb bombs being routinely and indiscimintely dropped across much of Iraq in civilian areas. That's what has to happen to reach this 60,000 toll, based on the authors' own claim that the air strikes are doing the killing. In my view, it's yet another reason why we need to pin down where those 12 non-coalition violent deaths belong, because it also allows us to get a hard number for the bombing deaths in the non-Falluja clusters.
Your point number 9, where you complain; " And I'll answer you straight out - every word of this is either false or irrelevant. The authors did not say that "AT LEAST 100,000 excess deaths" have occurred outside of Fallujah (why do so many folks think that stating things in ALL CAPITALS adds force or credibility to an argument?...). "
The emphasis is warranted. When the authors estimate, as you note, that conservatively there were 98,000 excess deaths, they're expressing confidence that there are not only likely 98,000 excess deaths, but probably more. The authors make this assertion several times. That's why the word " conservatively " is used by them. They want the reader to know that their confidence in the 98,000 figure is based on their belief that the number is actually higher.
On this next quote, I'm gonna go easy on you, but you're starting to get a little snarky here:
"They did not say that a majority of deaths outside of Fallujah were women and children. They stated in their overall findings, and after having discriminated Fallujah from non-Fallujah data, that a majority of violence related deaths were women and children. Table 2 shows that there were 73 total violence related deaths including Fallujah. Later they state that 61 were coalition caused, and of these, 28 were children, and 4 were women. Simple math yields a majority of women and children, for their all-inclusive data."
Scott, your "simple math" has let you down. It does not yield a majority of women and children dying violently, for their all-inclusive data. There are 73 violent deaths. 40 are men (the 2 elderly being male as identified in the text), while 33 are a combination of women and children.
When you consider that more men died violently in the entire study, I don't know how you can say the authors are not being dishonest here. The number they're really pushing in this survey is 100,000 excess deaths. It's based on data that excludes Falluja. The violent death toll outside Falluja is made up of 15 men and 6 women and children. I don't think I need to make this any clearer.
Point number 10;" Yes Mike, many who defend the study are out to discredit the U.S. led invasion - and guess what, massive nationwide carnage does a pretty damn good job of doing that! Particularly when there is a well characterized cohort study with statistically sound methods behind that conclusion, and the best anyone can come up with against it is highly vocal "common sense" with no demonstrated flaws in the methods or data."
Well, that's all still up for debate, as far as I'm concerned (the statistical soundness of the study). Concerning the rest of your argument, the coalition would likely be dealing with an insurgency regardless of the quality of the post-war planning. The Iraqi people are far better off having the coalition take apart the Baathist power structure left in place, even with the death toll we now see (whatever that might be. For you to suggest that "hundreds of thousands " is exactly what we're seeing now, based on this study, is an inaccurate statement )(See, I'm still playing nice, temper under control, etc......).
Point number 11; "What we're doing Mike, is the one thing you've refused to do so far - sticking to the actual data and methods and applying known principles of statistical science to them, as opposed to hand-waved guesses and/or hunches based on "common sense" that make little use of either. "
That's fine with me Scott, but let's get all the actual data, not just the data the authors wanted us to see. I believe what we don't know has the potential to dramatically change how this study holds up.

What did I do wrong with the paragraph breaks? Do I need a at the beginning of each paragraph as well?

I believe what we don't know has the potential to dramatically change how this study holds up.

Well, I guess that sums up your argument. Very Rumsfeldian. Alas, it's the stuff you don't know you don't know that's at the root of the spectacular amount of wreckage you've generated on this topic.

HTML tags for breaks are [ br ] you need two to do a skip line. I use [ p ] for that purpose (of course using < and >) as the keystrokes are fewer.

Rumsfeldian. I like it. There are a lot of unknown knowns around here.

Best,

When you read the Lancet paper, they estimate 98K (+/-90k) deaths for the whole country excluding falluja ! For Falluja, the authors estimate that there would be an additional 200 k deaths !
Surely we can all agree that the Falluja results are a cause for concern about the precision of their overall finding !
j

Hi Mike. Thanks for the kind words, and yes, I accept your apology too. I know what you mean about bare knuckles blogs. These days I just stay away from them because I have no confidence in my own ability to solve more problems than I create. As you can see, I have enough trouble doing that in basically civil blogs like this one :).

Your line breaks seem to be working fine. Like other folks said, the P Tag should create an actual paragraph break. But for some reason, when I tried it here I only got a line break. ??? Anyway, a regular BR Tag <"br"> (but without the quotes) is just like hitting "Return", and two of them (<"br"><"br"> but again, without the quotes) gives a space between lines, which is the same thing as a P Tag. That always works for me here.

Thanks for your comments about my responses. I'm kind of up to my neck here in a project, and in pre-Holiday stuff, but as soon as I can (hopefully within the next few days), I'll get back and share some more thoughts about them. Thanks again, and best!

BTE Mike, You are right. I was a little "snarky" in my post from the other day. I'll refrain mightily from that in the future!

Hi Mike,

looking through your numbers, I have some difficulties following your logic entirely.

So, we assume that it is true that there were bombings in just 4 clusters outside Fallujah. We know that 15 clusters recorded at least 1 violent death, and 21 violent deaths were reported outside Fallujah.

If 14 clusters report at least 1 death each, we've got 7 we can freely distribute. Therefore the 4 clusters outside Fallujah reporting deaths from bombing could contain up to 11 people.

However, we also know that deaths attributed to coalition forces were composed of 58 bombing deaths and 3 shootings of men.

Therefore, a minimum of 7 clusters outside of Fallujah must have had 1 non-coalition caused violent death. Why? Because 10 clusters reported non-bombing related deaths, each at least 1, and only 3 could have been caused by coalition related shooting incidents.

This looks like we could have a maximum of 4 child bombing deaths, 2 woman bombing deaths, 1 elderly bombing death and 4 male bombing deaths outside Fallujah. And a mimimum of 4 children, 1 woman and 1 man (21 less 12 non-coalition caused less 3 coalition shooting is six, all 28 children were killed through coalition bombing, therefore also the ones excluding Fallujah, 1 woman outside Fallujah could have died violently non-coalition attributed).

If all non-coalition caused fatalities and all coalition caused shootings were outside Fallujah, we've only got 6 bombing deaths, in 4 clusters (which could be 1 in 3 clusters each, plus 3 deaths in the fourth, or 2 clusters with 2 deaths each and 2 clusters with 1 bombing death each - there are no other possibilities to distribute 6 deaths in 4 clusters with a minimum of 1 each).

If coalition caused shootings happened in Fallujah, the minimum number of violent fatalities outside Fallujah attributable to non-coalition causes goes up. If all the 3 shootings were in Fallujah, 10 of the 12 non-coalition caused violent fatalities would have to be outside Fallujah.

We'd then have something between 9 and 11 bombing related fatalities outside Fallujah, and between 10 and 12 non-coalition caused fatalities.

Anyways, it's pretty clear why the authors don't split the numbers down any further. No matter how they slice it, they have to base their claims (most killed by the coalition were killed by bombing) and (most of those were women and children) on Fallujah. The number of deaths from bombings and of women and children outside Fallujah is simply too low for such conclusions to be vaguely credible.

For example, if we look at one possible scenario (3 children dead in one bombing, 1 adult, 1 woman and 1 child each in other clusters), one misperception of the guilty party (the 3 children were killed after an islamist terrorist bombing rather than a coalition bombing), could halve the number of innocent bombing victims of the coalition ex Fallujah.

Also, none of these 6-11 bombing deaths ex Falluja may be verified with death certificates, nor has the attribution of the deaths to the coalition in any of those cases been checked (again 1 single former Baathist interviewed who would have falsely attributed an insurgent mortar to the coalition, could easily be responsible for half the number of bombing deaths ex Falluja).

Let me summarise what I think about this study at the moment:

It tells us nothing we didn't know already. The only thing that can be reliably concluded from the study is that violent death has gone up, when comparing the war and aftermath with the period immediately preceding it.

Tim summarises the study as "things have gotten worse thanks to our decision to invade, not better". And if he defines "worse" as violent death has gone up, which we knew already before the study got published, he's clearly right. But then there is no need to defend the study, because that's a conclusion that the study is not being attacked for.

What the study is being attacked for is principally the claim that 100,000 is a conservative number with most of the excess death due to coalition bombing related fatalities of women and children.

That's completely unsupported outside of Fallujah, and the data for Fallujah itself are highly questionable.

Tim's summary of the main finding of the study (things have gotten worse, not better), is not the same as the authors' recommendation, namely that the coalition should reassess the use of aerial bombardment due to the widespread death of innocent civilians being caused.

A few other points:

1.) The infant mortality numbers are very shaky (to rephrase one of my arguments: 3 of the infant deaths may be from bombing in Fallujah, 3 may be due to inability to get to a hospital, also possibly in Fallujah - without these 6 infant deaths the supposed increase pretty much disappears)

2.) The number of deaths from crime clearly has gone up. But again we knew that already.

3.) The study's numbers on insurgents/Baathists, and their victims are unreliable.

I am amazed at the lengthiness of this blog now. I've re-read the paper again and have to say that it is summarised in such a way as to make it difficult to see a few points. I think it would be best if the authors released the raw data so we don't have to make all these guesses about how the numbers break down.

I am somewhat skeptical about this paper. it comes on the back of a number of papers in the Lancet that have annoyed scientists in the UK. Notably there was the infamous MMR study that should never have passed peer review and a health scare on GM potatoes that was excoriated by the Royal Society. Both were topical and seem to have been published due to this. This study too is politically motivated research. I do not think that there are great problems with the methodology. You have to make the best guesses with the data involved-- the problems to me lie in how reliable we believe surveys to be, particularly when very rare events make huge extrapolated differences.

Again I return to the point that the major flaw with surveys whether they be exit polls or consumer surveys is that people lie. A small bias does not matter against an even background of respondents but makes a huge difference to the reporting of rare events.

If the findings were consistent with other sources then we might have more confidence. Most excess deaths come from US bombing apparently. Yet we simply have not seen this level of bombing during the time period in Iraq. Here in the UK we see Al-Arabiya, Al-Jazeera and the BBC all of which are commited to showing all the news they can. Whilst the last maybe restricted in its access around Iraq the former are not and try and find as much carnage as possible. I am sure Al-Jazeera were as surprised as anyone to hear there had been 10s to 100s of thousands of deaths caused by US bombing they had missed. They clearly haven't been doing there job very well. We do see lots of insurgent or resistancy related deaths on the BBC (maybe this is biased too) yet this is hardly reflected in this survey at all. Very strange. As has been pointed out a number of times Iraq is dangerous and the surveyors wished to cluster sample to minimise travel. Are they really claiming that they were scared of air attack and not criminals or insurgents.

Let me clarify. I think Iraq is a bloody mess that should have been best avoided. But have we really been missing hundreds of people killed by US bombing every day?

Hi Stephen,

I am in broad agreement with you. I am also sceptical about the reputation of the journal after its recent lapses, most notably the MMR debacle.

Having access to sciencedirect as part of my job, I've done a search on Iraq and mortality, and the Lancet came up several times. The articles thrown up largely weren't research, but editorials, and reveal pretty clearly where the Lancet stands. Not only that, they are the only medical journal in sciencedirect that's taken such a clear political position based on what my search came up with (Elsevier is the largest publisher of scientific journals, so it's a fair proportion that are in this search).

I also agree with your doubts about the bombing deaths. That conclusion of the Lancet study (and sadly the one they identify as their key finding in terms of policy relevance) appears to be baseless, and is in contradiction of other evidence.

Somebody defending the study has made the point that loads of people die in Australia every year, yet we hardly read anything about them in the papers.

However, as somebody else here has commented, we would hear about tens of thousands of Australian deaths, if those deaths were brought about through bombings.

Those will be reported and it is just not credible that in excess of 90% would not be. There is an enormous incentive for anybody in Iraq opposed to the coalition presence to bring every single bombing to the attention of the likes of Al-Jazeera.

If anything, press accounts are likely to represent a significant overcount in my opinion, as journalists will be lied to (about the numbers killed and the attribution of the bombing to coalition forces or terrorists).

I am of the opinion that the numbers published by the Iraqi government are the most reliable, and that by far and away the greatest concern where coalition military performance is concerned are accidental shootings of innocents after IED explosions and while soldiers are under attack, and my main recommendation would be for communication to be improved through forcing soldiers to learn Arabic, and through arranging extensive contact with Iraqis in safe surroundings, to ensure that those language, cultural and empathy skills can be practised extensively.

Stephen
you are spot on correct. People suffer from recall biases- they lie, and fail to recollect correctly.
in this study, 19% of the verifications failed, and they only attempted verification in a small proportion of their putative fatalities. So there are very good grounds for believing that there could be substantial recall bias in this study.
even worse, the Falluja numbers extrapolate to 200 000 deaths in the falluja area alone. Presumably we would miss the 200k bombed people in falluja ?
j

Mike; your post deserves a detailed reply and will get one as soon as I have time.

Heiko; although Fallujah was an outlier for calculating the relative death rates, you can't simply ignore it and treat it as uninformative. Al Quaeda has never killed anyone in the USA, as long as you exclude the outliers.

Stephen, so far from being "not restricted", Al-Jazeera's representative in Iraq was asked to leave by the Allawi government in August and has not been allowed back. Al Jazeera currently has no camera crews in Iraq. Are you beginning to see the dangers in making assumptions without checking?

dsquared wrote:"Al Quaeda has never killed anyone in the USA, as long as you exclude the outliers."
yes, dsquared, you are getting the general gist of the problem with cluster sampling; that it is unreliable.
but it is worse than you suggest. With Iraq, we do not know what the true distribution of mortality is. So the results obtained from sampling may be correct, or may- as Heiko points out- have missed another major area of mortality. The problem is that we just do not know, and cluster sampling is inherently unreliable.
By contrast, your comparison with the US is particular interesting. We only know that Al-quaeda has killed someone in the US because there is excellent recording of deaths. The ~4000 people who died in the twin towers disaster would not be picked up in an epidemiological cluster analysis for the last decade, simply because they are such a small, and geographically isolated, proportion, compared to the whole american population.
I am waiting to see how you manage to tell us that Falluja was informative, but there were not 200 000 deaths there.
J

I'm not sure how the authors can get around this. It's difficult for them to claim that coalition bombing deaths rose at a much higher rate (obviously, from nothing pre-war) than other causes of violent deaths, because it appaears ALL causes of violent death have gone up.
I believe that was the initial assertion.

The only way around this is to have more studies. Denunciation of the figures is quite problematic without figures of one's own for comparison.

This paper is reminiscent of the Nature Chapela GMO gene-flow paper. Subsequent studies proved him right.

Best,

D

Scott:

I was just yanking your chain with the " snarky " comment. Nothing to apologize for on your part.

D Squared, Scott, Heiko;

The only way to get to resolve the question of the distribution of deaths from the study data was to contact the authors, so I did.

I have to say, I was impressed that the two I contacted (Gilbert Burnham and Les Roberts) immediately replied to a nobody such as myself. Here's the e-mail from Roberts:

Dear Mr. Harwood,

Yes, all 12 non-coalition violent deaths happened outside of Falluja. (1 Kut, 1 Thiqar, 1 Karbala, 7 Baghdad, 1 Diala, 1 Missan, Note Baghdad is about 3-7 times greater in population than these other Governorates so the rates are not so different)

Bombing deaths:

Thiqar- M5, M2, F22 (one family)

Thiqar (different village) - M27

Missan - 1mo. & 6mo. in same households (often there are multiple sons with wives under the same roof - interviewer did not record the gender of the infant)

Falluja - 10 girls<12 years, 13 boys<12, M14, 25 adult males, 3 adult women (adult defined as 15 - 59).

I hope this helps.

Cheers,

Les

At least now, the mystery of the configuration of the non-Falluja death totals is cleared up.

We now know that 6 of the coalition bombing deaths occurred among the non-Falluja clusters. We also know that the 3 shooting deaths attributed to the coalition come from these same clusters, as do all 12 of the violent deaths not directly inflicted by the coalition.

As a result, the following observations can be made from the data that constitutes the 97% of Iraq surveyed outside the Falluja cluster:

The data from the 97% of Iraq sampled in the survey that excludes Falluja recorded 89 deaths of all types during the post-invasion period.

This is an excess of 43 deaths, over the 46 deaths recorded from the pre-invasion portion of the survey.

Of the 43 excess deaths, a narrow majority (22 versus 21) were from non-violent causes.

The most prominent, defining, and well known quote from the study is contained on page 1, when the authors assert that " making conservative assumptions, we think that about 100,000 excess deaths, or more have happened since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths."

The 6 deaths from coalition bombing in the non-Falluja accounted for only 6.7% of the deaths from all causes that were recorded during the post-invasion period.

These 6 bombing deaths account for only 13.9% of the excess deaths (43) recorded during the post-invasion period.

The 6 bombing deaths make up only 28.5% of the violent deaths (21) that were recorded during this same period.

These three percentages clearly are in stark contrast to the claims made in the above statement from the authors.

Some further observations concerning this same subject:

The authors leave the impression with the above statement that coalition air strikes were far and away the leading cause of the excess deaths. However, the 6 non-Falluja bombing deaths are still 1 less than the number of men and children who died of heart attacks and strokes in the post-regime period (7). The study measured no such deaths from heart attack and stroke among men and boys before the invasion.

Moreover, the 6 bombing deaths are exactly half of the number of individuals who died from non-coalition violence (12). This comparison, perhaps more than the ones I mention above, stands out as a direct refutation of the authors' statement. It also best makes my point concerning the importance of knowing the missing non-variables concerning the distribution of the violent deaths.

When you look at the individual components of the death table from the study, you begin to realize how fragile they seem when standing on their own. We have 2 non-coaltion caused violent deaths that are categorized as "unknown origin." We have no idea how they met the criteria necessary to be called violent deaths, since we don't know how the individuals died, but coalition bombing deaths, which are supposed to be the dominate cause of death, only outnumber these 2 "unknown origin " deaths by a factor of 3!

D Squared, I realize that Falluja can't be completely dismissed simply because it's an outlier, but the authors have clearly stated their 100,000 execess death figure, largely based on coalition bombing, holds true without Falluja. If the bombing deaths outside Falluja don't support the authors' claims, can we still have confidence int their 100,000 death estimate, if the primary contributor to the number is not the leading contributor it is claimed to be?

I'm not sure how the authors can get around this. It's difficult for them to claim that coalition bombing deaths rose at a much higher rate (obviously, from nothing pre-war) than other causes of violent deaths, because it appaears ALL causes of violent death have gone up. There was only 1 violent death for the entire survey prior to invasion. We have no idea what category it fits in. If it wasn't a criminal homicide, then the rate of homicide went up more than the rate from bombing deaths, according to this study.

What do you all think?

Mike, thanks for getting the breakdown of the figures. The pre and post-invasion periods were of unequal lengths, so you would need to adjust for that when working out how many excess deaths there were. I think you get about 60% from violence (not counting Falluja).

If I was the authors I would reason as follows:
Iraq is made up of normal regions, extremely violent clusters (like Fallujah) and the Kurdish North (where the death rate fell).
In normal regions, the excess deaths are split roughly half and half violent/nonviolent.
In highly violent clusters, the excess deaths are massively biased toward violent deaths.
Therefore, in Iraq as a whole, the excess deaths are mainly from violence. Although Fallujah was an outlier, a more comprehensive study would have sampled Sammarra and Najaf, so Fallujah cannot be taken as completely unrepresentative.

Dano said: "Denunciation of the figures is quite problematic without figures of one's own for comparison."
not at all; all you have to say is that the methods underlying this study are not capable of the precision claimed. It is like using a pair of kitchen scales, and claiming precision to three decimal places.
"This paper is reminiscent of the Nature Chapela GMO gene-flow paper. Subsequent studies proved him right. "
this is the science thing; the lancet paper may be quite correct when they come to a total of 300 000 deaths in all Iraq; I don't know for sure, but I have my doubts on this finding, because the methodology is so flaky. I am not asserting that the finding is wrong.
j

I agree with your main point here, but I think that you're arguing in a systematically biased way with respect to the possibility of an underestimate rather than an overestimate.
We know that one accident of the kind you describe actually happened; the Sadr City cluster had zero deaths. We also know that because of the grouping process, Samarra and Najaf were not sampled at all. That's three separate highly violent regions which contribute nothing to the estimated death rate; in my previous post I referred to highly violent "regions" not "clusters", because I was attempting, (not very clearly), to make the point that there were lots of highly violent areas which weren't sampled.
Meanwhile, you're arguing on the basis of figures which exclude the one highly violent region which was sampled. The ex-Fallujah figures can't be taken to stand for "97% of Iraq" because they aren't representative of Sadr City, Najaf, Sammarra or Ramalla. These four cities would be equal to [DD rough guess] 15% of Iraq (based on the factoid that Sadr City has about 2m inhabitants, which is from memory and could be out by an order of magnitude). They are also likely to be disproportionate contributors to the death rate. I'd argue that in looking at causes of death, it makes much more sense to look at the cum-Falluja data than the ex-Falluja data, which is what the authors did.

D Squared:

I have to take issue with some of your latest post.

"In highly violent clusters, the excess deaths are massively biased toward violent deaths."

We don't have that many " highly violent clusters," to begin with. More than half, 18 of 33, reported no violent deaths whatsoever. Even excluding the 7 Kurdish clusters in the north that failed to tally a single violent death, we still have 11 of the 26 remaining clusters that did not report a violent death either.

Finally, we only have 21 violent deaths spread among 14 non-Falluja clusters. There simply aren't enough violent deaths to go around to create a " massive bias toward violent death" in any cluster other than Falluja.

You may have a point if we look at the governorates that encompass Falluja and Baghdad, but not the individual clusters themselves.

I'd also like to draw your attention to an earlier portion of this thread, where I advanced the theory that the non-Falluja bombing death numbers may well be quite small, as low as 6, and this was inconsistent with the large death toll from the type of massive bombing that the Lancet study authors and its defenders were implying must have occurred. To put this another way, the Falluja cluster sample reported deaths more consistent with the type of bombing that the Lancet believes to be fairly common across Iraq other than the Falluja area.

At the time, the rebuttal to my point was two-fold; From Scott, he challenged the supposition that there were only 6 bombing deaths in the non-Falluja clusters, voicing the possibility that the numbers could be significantly (for statistical purposes) higher.

From you, D Squared, came the suggestion that possibly some of the non-Falluja bombing deaths (however many there were) could have been erroneously attributed to the coalition when in fact they were caused by insurgent mortars.

But we now know that in fact there are 6 bombing deaths from the non-Falluja clusters. Further, we know that only 4 of the 32 non-Falluja clusters reported any bombing deaths. I believe this supports the point I made earlier, that bombing in Iraq was, and is, far from uniform, and in many areas was not, and is not, widely used by the coalition.

To further illustrate the argument I've made in the past, that the elements of the violent death toll are extremely fragile to manipulation, let's assume that the one Thiqar cluster that reported 3 bombing deaths was substituted for another Thiqar neighbourhood that actually had 15 bomb deaths, something more consistent with (but still only a fraction of) the Falluja results. Such a bombing in Thiqar would represent what many have come to assume the coalition bombing as depicted in the Lancet study must look like; a significant part of the neighbourhood devastated, with multiple deaths in multiple homes within the neighbourhood. By changing that single bombing, one event, we drive the non-Falluja bombing detah rate up by a factor of 3, and create an extrapolated death toll of well over 50,000 from bombing alone. Of course, this puts the Lancet 100,00 excess death toll from all causes off the charts.

Now, let's do the same thing with the Thiqar triple fatality, but instead, the survey picks another neighbourhood that reveals no bombing deaths. Now we're only down to 3 bombing deaths outside of Falluja, for 97% of Iraq. The impact on the extrapolated excess death toll, and the weight of criticism of Coalition culpability, is dramatically skewed from the data we currently have.

Neither of the above possibilities is outlandish or unlikely, given the overwhelming number of clusters that experienced no bombing deaths.

The difference in sampling a single neighbourhood other than those actually surveyed has the potential to cause huge swings in the overall extrapolations. I hate to keep bringing it up, but that's what small numbers are vulnerable to.

dsquared argues that you should analyse the data with the Falluja cluster in.
the authors specifically exclude this, because they say that the cluster is statistically unreliable. The numbers would extrapolate to 200k deaths (+/- confidence interval), and the authors could not contemplate justifying such numbers.
so when you argue for including the falluja data, you have to accept that the authors themselves believe that data is not reliable for the purposes you wish to use it.
j

D Squared:

I don't think I was neglecting the possibility of an overestimate being just as likely as an underestimate, at least not in the last post. To be sure, the thrust of my argument from the beginning is that I believe the 100,000 excess death figure is too high. But when it comes to this particular sampling methodology, I do believe that the actual study result could represent an underestimate, and I'm firmly convinced that identical studies could give us much higher extrapolations than what we have with the actual study. It goes without saying that I don't believe this enhances the credibility of the results of the actual survey.

If you look back at my previous post, I insert two hypothetical changes to the triple bombing fatality in Thiqar. One allows for a different neighbourhood that reports no bombing deaths, the other hypothetical supposes 15 deaths in a different neighbourhood. One creates a much lower death extrapolation, the other a much higher extrapolation. I think I was giving each (the underestimate and overestimate) equal billing.

The problem with using the actual study data to determine the probability of an overestimate or underestimate for the actual study is the shakiness of the totals for the unique causes of death that make up the lion's share of the 100,000 excess death figure. As I've pointed out here and in the next thread, the actual data appears to grossly underestimate the number of insurgent dead, and provides extrapolations on bombing deaths which do not appear to corroborate the authors'own conclusions. Because these two causes of death are so significant to an overall excess death calculation, the study's apparent inability to provide reliable estimates for these reflects badly on the reliability of the 100,000 figure (and yes, it may be actually low balling! I just can't see it).

"I'd argue that in looking at causes of death, it makes much more sense to look at the cum-Falluja data than the ex-Falluja data, which is what the authors did."

D Squared, I just can't see a viable case for this. The reason that jumps out at me is the fact that including the Falluja numbers blows the confidence interval away. Including Falluja's deaths unavoidably leads to a nationwide extrapolation of 300,000 excess deaths. The authors wanted no part of this, and are repeatedly on the record that their 100,000 estimate can be defended without relying on the Falluja data. The confidence interval can't survive the re-introduction of the Falluja data.

As I mentioned in an earlier comment, we have an unequivocal expression of this from Dr. Roberts in the Medialens e-mail exchange:

"It happens, that the one place with a lot of bombings, Falluja, and we excluded that from our 100,000 estimate....thus if anything, assuming that there has not been any intensive bombing in Iraq." (the elipse isn't mine, that's how the e-mail reads)

As for Najaf, Samarra, Sadr City, etc, we really can't project how sampling these areas might affect the outcome of a different survey. My suspicion is that large scale civilian casualties may well be specific to unique and limited geographic areas. I see this as a major limitation of the survey methodology. Sampling different clusters in the high violence areas not originally sampled may still provide a very misleading overall picture. Samplings from Samarra or Ramadi may not reveal a Falluja-like death toll, but what if each provided somewhere in the neighbourhood of 15 coalition-caused deaths? It's unlikely the authors would have excluded them as outliers. Now you've got a much larger extrapolated death toll than 100,000. But could these clusters, AS A WHOLE, provide the necessary numbers to fill in the tens of thousands of extra bombings? Targeted bombing may have hit certain very narrowly defined parts of the violent governorates. A survey that randomly selected 1 or more of the hard hit neighbourhoods in these governorates might reault in violent death extrapolations the governorates are simply unable to even remotely sustain.

Hopefully this makes sense, I'm getting tired!