The Lancet authors have released the data to other researchers:
Six months have passed since the publication of the study and we feel the time is now right to make the data set available to academic and other scientific groups whom we judge have the technical capacity to objectively analyze the data. It is our desire that the data be used in a way that will advance the understanding of how to best assess mortality during conflicts and to improve the protection of those caught in conflict. Although conflict is inextricably intertwined with national and international politics, it is our very strong hope that the analysis of these data and the broader discussion of mortality in conflict can be conducted above the short-term political controversy.
In respect to major ethical, as well as personal safety concerns, the data we are making available will have no identifiers below Governorate level. All other data are available for review.
However, they have added some conditions:
These data will be released on request to recognized academic institutions or scientific groups with biostatistical and epidemiological analytic capacity.
The data will be provided to organizations or groups without publicly stated views that would cause doubt about their objectivity in analyzing the data.
The first condition seems reasonable, but the second is unnecessary. Science reports (subscription needed):
But at least one researcher has been turned down: Michael Spagat, an economist and expert on conflict studies at Royal Holloway, University of London, in Egham, who has been a proponent of the street-bias idea. Burnham, e-mailing from Jordan, declined to explain which criteria Spagat did not meet; co-author Les Roberts, now at Columbia University, says he wasn't involved in the decision but that Spagat "would not meet the criteria by multiple measures."
Spagat calls the policy "deeply flawed," adding, "If we do something dumb or nonobjective with the data, qualified people should be able to expose our stupidity." The decision also puzzles David Kane, a fellow at the Harvard Institute for Quantitative Social Science who has received the data set even though he says he posted comments on a Web log last fall that raised the possibility of fraud. Denying some critics access "is ridiculous," Kane adds.
I think it's fair to deny Spagat the data on the grounds that he doesn't know how to analyse it, but not on the grounds that there are doubts about his objectivity. Not that he would have been satisfied with the data in any case:
Others are concerned that the group's decision to withhold information such as main street names and the sampling protocol has made it impossible to detect street bias or other potential problems. More details on the interviews "are necessary if the authors are to lay to rest intimations of 'fabricated' data," says Madelyn Hicks, a psychiatrist and public health researcher at King's College London. Burnham says his group "envision[s] no additional release of materials."
I don't recall ever seeing a release agreement quite like this. Serious analytic credentials, I can see. No second-party release, I can see. Written request on organizational letterhead with signed agreement, I can see. I've done all of that before. I've never seen
That's seriously off-putting.
Why am I not gasping with amazement?
Spagat: "If we do something dumb or nonobjective with the data, qualified people should be able to expose our stupidity."
Well yes, but that's not going to stop the Times from printing another hatchet job based on your "work", is it Professor?
There is a real problem here (though I don't think Burnham has found a very good solution). The biggest potential payoff from mortality studies is that they may alert the public to the scale of an ongoing disaster. That possibility doesn't exist if the message is drowned out by guys like Spagat. The scientific process works slowly. To the extent that the JHU team are motivated by a desire to save lives, they are right to try to protect their work from hacks. Long-term vindication is not enough.
Given the Lancet's strongly prejudical views of John Howard and others on that side of politics (recently discussed in the press )isn't it a bit rich for him to seek 'objectivity' in others.
Isn't the Lancet's editor a blatantly left-wing partisan whose views on Iraq and a whole set of social issues entirely predictable and anything but objective?
That's not to say he is necessarily wrong. But when science gets mixed up with politics credibility fails. Is Lancet really a reputable scientific journal?
Err, Harry, the Lancet was just the journal where the study was published. The Lancet is not involved in any way with the release of the data.
Howard dismissed the study with this gem:
>Well, I don't believe that John Hopkins research, I don't. It's not plausible, it's not based on anything other than a house-to-house survey. I think that's absolutely precarious.
Strangely enough, other house-to-house surveys like the Ausralian census are OK with Mr Howard.
Oh dear. This is daft and counterproductive. Kevin makes good points, but I can't believe that this sort of rinky-dink stipulation is the way to deal with it.
House to house surveys are interesting. I wonder if they could do a study that looked for differences in some similarly measurable quantity in a conflict zone, non-conflict zone, or simply across cultures, to see if something that should essentially measure the same does not due to some other influence. Or if house to house surveys can provide uniform results in spite of those differences.
Err, Tim, didn't I read Les Roberts decrying objectivity in epidemiologists saying they all had good reason to dislike the subject of their study, including the Iraq war, and that he had resultantly rushed to publish every study he'd ever done? How objective is that?
"In respect to major ethical, as well as personal safety concerns, the data we are making available will have no identifiers below Governorate level. All other data are available for review."
Heh heh. How nice of them not to identify the streets involved due to "ethical concerns" despite their stated method of getting little kids to run around the neighborhood notifying everyone that there was a poll coming through and what it was about. What a lovely nebulous non-claim.
To the extent that they desire to save lives, they should make their research public and open to everyone and shouldn't rush publication, making little mistakes like confusing casualties and deaths along the way. That they don't make it open to their severest critics, failed to release their data until after many attacks on the objectivity of their study, and are openly non-objective on the topic of the Iraq war is an indication that the partisan nature of contemporary politics has infected JHU epidemiology research.
Oh and BTW, the Nature article still plainly contradicts several claims the authors have made, even after their varied retractions and 'modifications'.
"Why should I make the data available to you, when your aim is to try and find something wrong with it".
Where have I heard this before?
Think, think.
John 'A', do you know, just like you, I think I have that *deja vu* feeling all over again, again! Semi-scientists, reluctant to release data, tight little groups who all agree with each other ... no, don't tell me ... it'll come to me in a minute ...
If Les Roberts lost the data somehow, would the he
meet the requirement:
The data will be provided to organizations or groups without publicly stated views that would cause doubt about their objectivity in analyzing the data.
?
"Semi-scientists, reluctant to release data, tight little groups who all agree with each other ."
And self-deluded far-right non-scientists ready to denounce anyone and everyone and any scientific discipline that conflicts with their prejudices.
I'm sure said non-scientists will be as successful in this are as they have been in the area of global warming (i.e. not at all).
Ian, in a no-doubt vain effort on my part to approach the sort of scientific exactitude you require, may I ask you how far "far" is, as in, say, "far right"?
I only ask because it has just the teensiest-weensiest hint of a 'denunciation' in it, to say nothing of a hint of 'prejudice'!
"I think it's fair to deny Spagat the data on the grounds that he doesn't know how to analyse it"
I do not. We aren't talking about an unpublished study. If they got it published then I think (if they are in fact "scientists") they have an absolute obligation to make all of their data used (save that which is obviously protected for privacy reasons) fully available.
"And self-deluded far-right non-scientists"
Well, I'm not exactly far-right, and I may or may not be "self-deluded", but I am in fact a real live scientist with publications in various peer-reviewed journals. Virtually every journal I have ever published in made it an explicit requirement that any novel reagents used in any study they published had to be made freely available (though not necessarily available for free) to anyone who requests it. I have always assumed that was to permit others to verify your results. Refusal to allow anyone access to raw data should immediately be considered an involuntary retraction and be treated as such by any and all (whether they are scientists or not).
Prejudice David is to pre-judge, i.e. to judge someone on first sight on the basis of superficial appearance.
At this point, sadly, I've had more than enough exposure to the blatherings of the pro-war anti-environment crew (at this point you start telling me how you're a three-dimensional human being with a rich meaningful life outside the internet who loves puppies and walks in the country).
Politics aside, real science has to be reproducible. If that involves making the data available, then the data has to be released.
Doesn't matter if you're a wingnut or a moonbat, you've got to release the data or withdraw the paper.
"Semi-scientists, reluctant to release data, tight little groups who all agree with each other."
Yes, that's been proven here - all of us who generally agree with the Lancet paper conclusions have closed ranks and expressed agreement with the decision to not share data.
Do the denialists every say anything accurate?
Well, bottom line:
the data is the result of the hard work and, in fact, risk of life by the Roberts group, and anybody who thinks that it should be given to every Cadillac-driving welfare queen who just wants to steal the hardwork of the productive members of society is just a socialist, so there.
I'm with z and Brian. To explain a bit further. By releasing first to the academic community and neutrals/favorables, the data will get an honest working over anticipating the broiwn fecal matter storm once it is generally released. I am sure that Roberts & Co know that there will have to be a general release at some point, but given the obfustication and just straight out lying that greated the paper one does not want to give a lead to the political opponents of the study.
As Phil Jones said, he is under no obligation to give a helping hand to those who have only an interest in destroying his work.
The Lancet team should follow whatever the standard scientific procedures are in cases like this. If that means releasing the data to anyone competent, even someone with suspect political motives, then that's just how it is. All they've managed to do with this approach is to give Spagat another platform.
That said, unless someone really does find evidence of bias or fraud, this release of the data won't prove anything. The argument will continue. The real way to test the claim that 600,000 have died is to replicate the study. Surveys on a large scale continue to be done, so it can't be argued that this is impossible. Yet nobody does it--Lancet 1 is followed nearly two years later by Lancet 2 and nobody else could be bothered. My impression is that in science, controversial scientific claims are supported or refuted by doing more studies, not by endlessly rehashing arguments over the original study.
I am pleased to be on the same side at dsquared and Tim on this dispute! "Daft", "unnecessary" and "counterproductive." Indeed. Give Spagat the data. One mystery is that they have given me the data and I, as Deltoid regulars know, have, on occasion, been critical of the study. Why me and not Spagat?
One hypothesis is that the authors think I am an idiot and, therefore, unlikely to find anything wrong. Spagat, on the other hand, is viewed as a threat. Any better guesses?
"By releasing first to the academic community and neutrals/favorables, the data will get an honest working over anticipating the broiwn fecal matter storm once it is generally released."
If we try to set aside the issue at hand and speak about principles, you seem to be saying that each side of a scientific debate should withhold data from anyone who disagrees with them for an unspecified period. If I understand you correctly, anyone who doesn't agree with ones point of view can be filed under "dishonest" leaving those who agree or are undecided to give the data an "honest" going over.
I think I'm going to take some convincing that this is going to be a valuable way of advancing scientific knowledge.
I should, perhaps, repeat here what I wrote over at my place:
"I must again repeat my respect for Tim Lambert, a man with whom I would probably have difficulty agreeing the time of day. Politically, I suspect, he would have considerable sympathy with the writers of the original report published in the Lancet, but he's unafraid to point up humbuggery where-ever it appears."
And that goes for everyone who supports his stance, from where-ever they happen to be standing!
I assume all you Lancet critics would go along with the Lancet call for another independent study? And you'd no doubt agree that the fact that the US/UK/Iraqi government hasn't funded an independent study probably indicates they have something to hide? After all, even, say, 200,000 excess deaths, while it would discredit Lancet 2, would also make the US/UK/Iraqi governments look like incompetent war criminals and liars.
Honestly, how you people can argue about bugger all for ever, beats me!
It's perfectly plain that a very large number of people have been killed in Iraq post-Saddam.
It's also perfectly plain that there was a good chance, given the recent wars between Iraq and Iran, and given Iran's determination to go nuclear, that had Saddam remained, zillions more than that would have perished if war had broken out again between them once again.
Similarly, given that Iraq, like Jugoslavia, was a nation of disparates held together by an iron fist, the internecine warfare that we see now would have happened anyway even if Saddam had died in his bed and his incompetent sons attempted to take over.
Shit happens! It just tends to happen more often, and tends to be several factors more smelly in dumps like Iraq than elsewhere. In the circumstances, measuring shit and then arguing about it is, er, well, pretty shitty!
#24
Sorry, I'm new to this debate. What reasons are given for the demand for another study? If Lancet 2 were found to be flawed, then it might well be reasonable to commission another survey, but we're not there yet are we?
[One mystery is that they have given me the data and I, as Deltoid regulars know, have, on occasion, been critical of the study. Why me and not Spagat?]
I think the problems with Spagat are 1) he has a really bad reputation in human rights circles - he made some very serious charges against Human Rights Watch with respect to Colombia which he hasn't backed up as far as I can tell and 2) his "main street" theory means that he might be a risk with respect to compromising the anonymity of the survey respondents. I still think that he shouldn't be shut out of the data but my guess is that this is the reasoning.
"We aren't talking about an unpublished study. If they got it published then I think (if they are in fact "scientists") they have an absolute obligation to make all of their data used (save that which is obviously protected for privacy reasons) fully available."
I don't think that they are obligated to make the primary data public. There is certainly no scientific rule that mandates this. However, in view of the controversy regarding this result, I think that they should make the data available to other legitimate academic scholars. And I agree with Tim Lambert that excluding people who have expressed an opinion based upon already-published results is inappropriate and counterproductive. Once a study is published, anybody is entitled to criticize it.
"It's also perfectly plain that there was a good chance, given the recent wars between Iraq and Iran, and given Iran's determination to go nuclear, that had Saddam remained, zillions more than that would have perished if war had broken out again between them once again."
Bullshit
We bombed him into the stone age. You don't build a nuke by pounding rocks together.
"given Iran's determination to go nuclear"
It's also perfectly plain that there was a good chance that that could have been averted, had the Forces of Good not gotten themselves in somewhat of a pickle.
Bishop, the issue is that the Lancet studies were limited by the size of the sample. To get a larger sample and narrow the confidence limits would require the acquiesence and help of the occupation and Iraqi authorities. They have no interest in doing this probably because they are certain of the results.
I don't know what neverworld the rest of you live in, but data is first shared with collaborators. You don't immediately release them to every tom dick and ross who wanders across the road. Even after your collaborators have mined it, release to others often requires payment for the time and effort spent on explaining it to those who require help. Most often you just refuse to do it because given someone thick, angry and dogged you will spend all your time holding their hands. Then you have the issue of the proprietory nature of the data.
I know this is in conflict with what you learned about science in fifth grade, but there are lots of things you learned about how the world works in fifth grade that just ain't so.
Bishop Hill said: "Politics aside, real science has to be reproducible."
So what's stopping you?
Hey, if even a fraction of the negative stuff some have claimed about the Roberts study is true, the Roberts data is crap and/or fabricated anyway.
So why would you want to use it?
Get your own brand new "clean" data (not corrupted by mainstreet bias, etc) to do your own study, man.
Don't let a few little roadside bombs get in your way. Get on it. Hop the next flight to Iraq.
As an aside, I find the implication that scientists are somehow "required" (by the "World Socialist Data Sharing Committee"?), to share all their data with anyone who wants it utterly hilarious.
Yessirree. Everyone knows scientists take an oath when they get their science merit badge: "I solemnly swear to share all my data with any and all fraud-charging morons who ask for it, so help me Dawkins".
Let us simply ignore for the moment that some of the greatest scientists in history hoarded their data for years -- Isaac Newton, for example. But as everyone knows, he was just trying to hide his arithmetic errors -- that and cover up a vast left-wig conspiracy.
"As Phil Jones said, he is under no obligation to give a helping hand to those who have only an interest in destroying his work."
Phil Jones is either just wrong or should be ignored completely. He's claimed variously that he needn't provide it because it has been superceded while citing it after this claim and despite it being used as a cite by other studies and that he won't provide it to someone trying to destroy it and now that he can't find it.
Either his work is sound or not. If he is going to put it into the public arena, he ought to justify his claims by showing his support. Jones is claiming the privilege to assert claims which are affecting policy without providing evidence to any and everyone it will affect. Every claim he tries to justify thus should be rejected prima facie. The acceptance of his claims sans evidence is contrary to sound, replicable science.
Plus, the general failure of climatologists to archive the data from their studies is a scandal.
While everyone's calling for more surveys, how about a survey of Iraqi refugees in Kurdistan and neighboring countries? Surveys like this are very common in conflict situations (e.g., surveys of Darfur refugees in Chad) and it's a lot easier to conduct surveys away from the war zone. Since a large fraction of these refugees fled Saddam before the war, such a survey would probably capture more pre-war violent death than did the Roberts surveys.
"Similarly, given that Iraq, like Jugoslavia, was a nation of disparates held together by an iron fist, the internecine warfare that we see now would have happened anyway.."
Yes just look at the civil wars that didn't break out in Iraq during the British occupation; with successive changes of ruler during the monarchy; during the British re-occupation during World War II; after the coup that overthrew the monarchy; after the subsequent Ba'athist coup and after the defeat of the Ba'athists in the Gulf War.
Oh and as for Iran's desire to go nuclear, (assuming it exists and is more than a delusional fancy), do you think maybe it has something to do with a certain nuclear-armed super-power which has invaded two of Iran's neighbours in recent years; has a large military presence in at least two others (Pakistan and Dubai) and has a publicly declared policy of "regime change".
I'm sure if China invaded Mexico and Canada and installed governments friendly to it in those countries the US wouldn't regard that as a hostile act.
It has in fact taking the combined efforts of the Coalition (unwittingly via their ham-fisted divide-and-rule policy of favoring the Shia) and Al Qaida over several years to get the current rather desultory low-key civil war going.
Eli
Thanks for this. If the sample sizes were too small then I would have thought it makes perfect sense to commission a larger study.
JB
If a second study was performed and reached different conclusions from the Lancet study then we would not be much further forward. We would still have to examine the Lancet study and test whether its data and methodology were correct. It makes much more sense to test the Lancet study first. After all, it might be unassailable.
Bishop Hill said:
and:
The first is, of course, the answer to the second.
But there has already been another statistical study which invalidates Lancet 2. This was the ABC/BBC poll last month which included the question whether anyone in the same household as the respondent had been physically harmed by violence since 2003. 17 per cent answered yes. Given that the World Bank gives a figure of 7.7 people per household in Iraq, this means that the total number of all killed, injured, beaten up or whatever in Iraq, whether civilian, military or terrorist and whether harmed by terrorists, criminals, Coalition forces or whatever, cannot be more than 2.5% of the population. This is the figure that Lancet 2 gives for civilian deaths alone. Given that the ABC/BBC poll was larger in number of respondents, had ten times the data clusters and was one of a series of similar polls whose results have been universally accepted as statistically valid, it must be regarded as much more reliable. Therefore the Lancet 2 study must be overestimating total deaths by at least three times (the figure of 2 wounded to 1 killed even in conflicts with zero medical infrastructure is widely accepted) and possible much more (US wounded to killed are 8 to 1, albeit Iraqis surely would not have such a high ratio). That represents an enormous discrepancy. Since Lancet 1 was at least a 4 times overestimate (measured against the UN Development Agency study carried out at the same time) and other casualty estimates are between 1/10 and 1/4 of the Lancet 2 totals this would be no surprise.
John, i don't understand your maths. So, out of a household of 7.7, what was the average number killed, injured or other? Surely in some households everyone might have been killed, leaving no-one to tell about them, and in other households one person might have been slightly injured. Merely quoting 17% of households had someone harmed by violence tells us nothing about total numbers involved.
What do you make of the 90% death certificates produced? Do you think they faked them?
dsquared speculates on two reasons why the Lancet team would give data to me rather than Spagat.
I am fairly certain that 2) is not correct. The data have already been thoroughly scrubbed. Although I have not done a thorough analysis, I do not think that there is anyway to identify individuals. I have no information on 1), but, if that is the reason, than the Lancet folks are lying since they claim that the reason for denial is one of their official requirements. Being on the good side of the human rights community is not so listed.
Perhaps dsquared and Tim can talk some sense into Roberts et al. Not giving Spagat the data hurts their cause far more than it could possibly help it.
Also, for those who are curious about replication and scholarly standards, a good starting place is here. And here (scroll down) is a long list of journals, most of which would have refused to even consider publishing the Lancet articles because the authors refuse(d) to share the data. A typical example is the American Economic Review:
The Lancet does not have such a policy, unfortunately. I trust results published in AER much more than those published in the Lancet. Shouldn't we all?
...what? You seem to be assuming that, wherever someone answered 'yes,' everyone in that household was questioned and also answered 'yes.' Have you actually read anything about the methodology of the poll in question?
Bishop Hill:
Perhaps you did not read what Tim said in his original post: Roberts et al have agreed to give their data to other legitimate researchers so that they may "test whether its data and methodology were correct."
The fact that Roberts won't give the raw data to a few know-nothing hacks who are completely out of their league and who seem bent on merely smearing Roberts is irrelevant from a scientific standpoint.
Since when have epidemeological studies carried out in a warzone been required to follow the same procedures as economics journals?
John Riddell, I was invoking that very same ABC poll a few weeks ago. It neither proves nor disproves Lancet, as various people pointed out. It seems not to have occurred to you that a household could have suffered more than one casualty. Incidentally, Lancet 2 didn't count "civilian" deaths. It counted all deaths of people in the households they checked.
I played with the numbers and thought that a death toll in the low hundreds of thousands (a bit too low for Lancet 2) seemed most reasonable for the ABC poll, but playing with the numbers is a pretty good description of what I did. It didn't prove anything.
What we need is a survey on that scale devoted to determining exactly how many people died violently in each household, which that poll didn't do.
Oh dear oh dear. Several people reading this don't understand basic probability theory. If 2.5% of people in Iraq are casualties then 97.5% are not. Hence the odds of everyone in an Iraqi household of 7 or 8 people NOT being casualties are 0.975 raised to the 7th or 8th power. The residual number is the proportion of households where at least one person is a casualty - and a small proportion will have two, three or even seven casualties. Do the maths yourself. 0.975 raised to the seventh power is just under 0.84. 1 minus 0.84 is 16% of a sample. The numbers are unfortunately unanswerable and do not support the Lancet 2 study - unless we assume that everyone who is a casualty in Iraq is actually dead and nobody is wounded.
Err, John you seem to be assuming that the probabilities are independant, which is obviously untrue.
JB
I haven't disputed that the data is being released to some other researchers.
I'm not happy though that they seem to want to pick and choose who they release the data to. When you say that these people are attempting to smear them, what precisely do you mean? If they are trying to prove them wrong, how can this be interpreted as a smear. Either the science is sound or it isn't.
Clearly Spagat appears to be a bona fide academic from an important institution. How can he be ruled out as a "know nothing hack"? You can't seriously argue that the search for scientific truth will be advanced by withholding data from those you think will disagree with you?
Bishop Hill said:
"I haven't disputed that the data is being released to some other researchers."
Then why did you bring up the subject of reproducibility?
There are lots of people around who have a background in epidemiology and (unlike Spagat) are therefore qualified to review the Lancet study -- and Roberts has already said that these people will/i> get the data.
If having qualified people review the data and attempt to reproduce the results were really what you were after, that would certainly have made you happy.
But you as much as admitted above that you just don't like the fact that Spagat (an economist, not an epidemiologist) did not get the data.
Boo hoo to you.
JB
I brought up the subject of reproducibility because that's the reason the data should be released. As I said in my last post, you can't pick and choose who you release it to either. This will end up with people only releasing data to those they think will not look at it too hard - which will not help advance the search for scientific truth. The idea that only an epidemiologist is allowed to look at the data is also quite wrong. Why should an economist or a statistician look at the data too? They might bring new insights. Science doesn't exist in neat single discipline pigeonholes - much, if not most, interesting work now takes place in interdisciplinary fields.
I don't know anything about Spagat or his views. He is a bona fide academic from a bona fide institution. If he's interested, give him the data. Frankly, the data should be up on the internet for anyone to do with as they wish (suitably cleansed to prevent identification of the subjects, of course). That's how we will get closer to the truth, whatever it is.
I trust results published in AER much more than those published in the Lancet.
David, this remark more than anything else makes me question your sanity!
As I say, I think that the JHU team have gerrymandered a set of criteria to exclude Spagat, for reasons which I find understandable but in the final analysis not legitimate.
I think "reproducibility" is a red herring here btw (this is the point on which the Lott/Levitt lawsuit turned, appropriately enough). If you have the data, you can check that the calculations were made correctly, and you can see if the results are robust to other modelling approaches. That's all. You might be able to run a few simple checks like Benford's Law, but you are not really going to be able to tell anything at all about the validity of the data themselves, other than what one can observe on the evening news which is that they are at least plausible.
I propose as a decent rule of thumb by the way that anyone who proposes to make any important point at all dependent on "the well known and valid ratio of n wounded to 1 killed" (where n is a number between 2 and 10, usually), can safely be dismissed as not having anything valuable to contribute. The source of this ratio is a couple of throwaway lines in histories of the Second World War, and more likely, rulebooks for miniature wargames. In any case, it cannot possibly be applied to the kind of violence seen in Iraq at present, which does not resemble a battlefield at all - when someone is murdered with an electric drill in the skull, it is unusual for three bystanders to be caught in the cross-drilling and merely wounded.
Bishop Hill said:
"you can't pick and choose who you release it to either. This will end up with people only releasing data to those they think will not look at it too hard - which will not help advance the search for scientific truth."
That's pure speculation on your part.
It's also demonstrably false, since David Kane was given the Roberts data in this case -- and Kane has been critical of the Lancet study, including speculating about the possibility of fraud.
John Riddell, fleshing out what Tim said, did it occur to you that if a mortar shell or a death squad pays a visit to a house and kills one person, the odds of some other person in that house also being killed or wounded are significantly higher than some random person in Iraq?
Or if more than one person in a family goes shopping (it's been known to happen in the US and I presume also in Iraq), then if one gets hurt in an explosion, the family member standing next to them has a higher than average chance of getting hurt?
The poll found 1 household out of 6 had someone physically harmed. That suggests around 500,000 households with at least one dead or wounded. What the average number of casualties per harmed household is I don't know and neither do you. You won't get the answer using your misapplied basic probability theory.
JB
The fact that Roberts has released the data to Kane doesn't prove that others won't refuse to do so on spurious grounds in future.
If the accepted standard is concurrent archiving in a public database or failing that, release to any interested party on request, then there is nowhere for anyone to hide.
Regarding criticisms that the deaths in Iraq are likely to be correlated by being concentrated on specific households (ie that there is likely to be disproportionate concentration in households due to the nature of the conflict) I don't think the evidence supports that at all, regardless of Tim's comment that they are obviously correlated. The one thing we do know about Iraqi casualties is that they are disproportionately male - Iraq Body Count gave an analysis some time ago suggesting 84% male. That does not suggest households getting wiped out en masse - that suggests individual male members of the household getting killed whilst in the streets - and frankly you would expect Iraqi men to try to protect their women and children by keeping them off the streets as much as they could. If they were correlated half the deaths should be children, as half of Iraqi household members are children. They are not (see above). Iraqi military/terrorist deaths are always going to be single men of course, and the pattern of sectarian/criminal killing in Baghdad would strongly support that - essentially all the deaths with very few exceptions are single males kidnapped and killed from the streets. This argument just does not recognise the enormous implausibility of the Lancet 2 numbers compared with the ABC/BBC results. Lets tackle the problem the other way. If Lancet 2 is correct then 2.5% of the Iraqi population are dead and (by extension) at least 5% must be wounded, making at least 7.5% physically harmed in some way (if the harmed to killed ratio is higher the problem gets even bigger). Again, basic probability suggests the answer to the question "has anyone in this household been harmed" should be about 44% not 17% (with no correlation between injuries of course). This discrepancy is far too large to be explained by anything other than systematic error in the Lancet 2 study I'm afraid. This issue with the "missing wounded" was in fact one of the main criticism raised of the Lancet 2 numbers by teams such as Iraq Body Count when they first came out - along with the many other problems with the numbers (like the absence of death certificates in anything like the numbers to justify the Lancet 2 figures, even assuming major bureaucratic failings). The trouble seems to me that the Lancet 2 study is trying to measure a very small effect (number of deaths at the 1-2% of the population level)when the statistical errors in their numbers are of the same order of magnitude - a fact they recognise by allowing that their range of deaths is the same size (around 600,000) as their mid point ( about 650,000). The ABC/BBC news poll is much more likely to be reliable as it is measuring an effect large enough to be really noticeable and where errors of 1 or 2% in the result do not really impact on the implications. I stand by my earlier statement - ABC/BBC clearly invalidates Lancet 2. The one downside to it as a check is that "physical harm" is such a broad term that whilst it can clearly be used to cap the highest plausible numbers of deaths it can't be used to give a very precise number. But since reasonable killed to harmed ratios end up giving you similar numbers to other methodologies excluding the Lancet 2 numbers I'm not sure this is a handicap. ABC/BBC supports the other methodologies, Lancet 2 is a very clear outlier.
But John, IBC only counts those violent deaths reported in the media. Which is not by any stretch of the imagination all of them.
Secondly, in a household with say 7 people, parents and a grandparent and some children, would you be really surprised if the father and son got killed whilst out shopping? This alone would double your possible deaths and injuries to, as far as I can calculate (2.5% of 26 million is around 650,000) to over a million. Which to me seems well into Lancet teritory.
Anyway, nice of you to totally ignore the posters pointing out that your methodology is completely invalid for the situation.
Too many assumptions, John Riddell. I played with the numbers a bit, as I said before, and what I thought most plausible was too high for IBC and too low for Lancet, but it proves nothing.
And if you look at the rest of the poll it suggests higher levels of violence than the IBC results--more current fear of US violence than seems warranted, for instance, given that press figures only gave 370 civilian deaths caused by the US in the third year of the occupation. About 16 percent (from memory) feared US troops over anything else, which is way out of line with what IBC statistics would suggest past the opening two months (and aside from Fallujah during the two US attacks).
Gotta go. This argument could go on forever and prove nothing. If someone did another survey specifically designed to measure mortality maybe we'd get somewhere.
I'm back briefly. Specifically, John Riddell, you're still ignoring that death squad deaths might run in the family, outside excursions might be taken by multiple members of a household, insurgent support (with accompanying casualties) might be a family affair (Lancet 2 didn't make the civilian/insurgent distinction), etc... Assuming independent probabilities is unwarranted but you keep doing it because it gives you the numbers you want.
Death to wounded ratios also vary, as dsquared pointed out. I read IBC's two year analysis and for explosions (citing from memory) they found the usual 3 to 1 wounded to dead ratio for bombs and so forth. For gunshots I think it was more like 1 to 1, presumably because many gunshot victims were killed execution-style by death squads.
If you assumed 1.5 casualties per "harmed" household in the ABC poll because of some within-household correlation, and 40 percent deaths out of that total (because enough are from execution-style killings rather than explosions to increase the dead to wounded ratio significantly), you have around 300,000 deaths. Too high for IBC, too low for Lancet. Where'd I get the numbers? Pulled them out of my ass, like you're doing.
Bishop Hill said:
JB
The fact that Roberts has released the data to Kane doesn't prove that others won't refuse to do so on spurious grounds in future"
It's pretty clear to anyone who reads what I actually wrote above that that's not what I said.
You claimed that "picking and choosing" who one released data to "will end up with people only releasing data to those they think will not look at it too hard" and I gave the case of Roberts' release of his data to Kane as a counterexample to show that yor implication is not true (all I needed to provide was a single counterexample, since your claim was essentially all encompassing)
Yo made this claim:
"you can't pick and choose who you release it to either. This will end up with people only releasing data to those they think will not look at it too hard - which will not help advance the search for scientific truth."
I then showed the untruth of that claim with a specific example:
I replied that your above statement is "demonstrably false, since David Kane was given the Roberts data in this case -- and Kane has been critical of the Lancet study, including speculating about the possibility of fraud."
JB
You misunderstand me. When I said "people", I meant "some people". You surely can't have thought that I meant that every single scientist in the world would withhold their data from those who disagreed with them?
This being so, your single counterexample doesn't prove your case.
On the ABC News poll: The more important number, since it is population-based, was that 53% of respondents said they knew someone (or more) close to them who had been physically harmed by the violence of the war. Discounting the number by half for superficial wounds or some double counting, you then have about 6 million people. a one-to-four ratio of dead-to-wounded would then yield 1.5 million deaths----Lancet 2 country.
For the household numbers, you're missing two important data: number of people per household (the old numbers are not reliable), and numbers in each household wounded, as is argued above. Both cannot be estimated with any confidence. Hence the 53% number in the population is a far better guide. None, however, is decisive, since they are derived from a long survey where the question of mortality is never asked.
Also see Juan Cole, April 17, about a report in Arabic that there are 900,000 orphans in Baghdad. Orders of magnitude is what to keep your eye on.
Bishop Hill said: "JB, You misunderstand me. When I said "people", I meant "some people".
OK, I admit that I interpreted your statement as meaning "most people", which is what (most) people usually mean when they say "people".
But if you meant "some people", then your whole argument becomes a bit puzzling -- pointless, really.
Some people have always refused to share their data -- so if this happened in the future, it would hardly be anything out of the ordinary.
Some of the greatest scientists in history did so (Isaac Newton, Tycho Brahe) - and not because they were afraid that others would prove them wrong.
It's really hard to see how the Roberts' case is going to change that one way or the other -- and if that is what you are arguing, the behavior of Roberts lends little if any support to your case.
Because Roberts actually gave his data to someone who is clearly looking to find fault with his results (David Kane), it is a bit of stretch to think that Roberts' behavior could be used as an acceptable excuse for others in the future to keep their data from the "opposition".
One could make a pretty good argument that those who would use the Roberts case as a precedent for not sharing their data with the "opposition" would do so regardless of anything Roberts had done or said.
John Tirman - good comment, but the problem with using the number about "people known to be injured" as a guide is that you don't know how many people forms each respondent's social circle - 10 or 1000 ? The per household number is known - the Lancet gives 7 which is pretty much the same as World Bank I quoted. Guthrie - I did address the methodology point but I don't think you got the argument - let me explain. If only 17% of Iraqi households have at least one person in them physically harmed, then all deaths and injuries must be in those households (ie they are strongly correlated as several people have argued). If deaths and injuries are a minimum of 7.5% of Iraqi population and probably more as per Lancet 2 then each of those households has suffered 50% casualties (7.5% casualties in 17% of population). Adult males make up only 25% of the Iraqi population, therefore half the casualties must be women or children as a minimum (may be more - random would be 75%). But IBC gives only 16% - and that is heavily understated as they do not record military or terrorist deaths (always adult male). You just cannot squeeze the Lancet 2 numbers into both the ABC/BBC poll numbers and the IBC gender/age split data. Even if you assume lowest possible ratio of wounded to killed, perfect correlation of deaths by household and that adult men always get killed or injured first with 100% certainty to torture the data into a completely unrealistic probability distribution it is at least two times too high. Donald - none of the data I am using is an assumption, it is all third party data - either ABC/BBC, Lancet 2 or IBC. For deaths to wounded in conflicts I suggest you reference Coupland and Meddings in the British Medical Journal August 1999 who give 2 wounded to 1 killed as a minimum, stress that many wars see higher ratios (having noted the issues regarding eg deliberate sectarian murder giving higher ratios) and include not only WWII data but also terrorist war cata from Lebanon and Northern Ireland in particular. Conclusion - Lancet 2 cannot be reconciled with other reliable poll data, hence already disproved.
JB
I'm not particularly commenting on the Roberts case, which, as I've pointed out above, I'm pretty new to. I'm more arguing the general principle that scientists should make their data available to anyone who is interested because this is usually the only way it is possible to verify their work. I'm sure you are right that it's always been the case that not everyone has done so, but these people were still in the wrong, weren't they?
We should discourage this sort of behaviour as much as possible, and I think that Roberts' decision to pick and choose who he releases data to was a poor one which sets a bad example. While I take your point about his having given the numbers to an opponent, the very fact of his having refused someone else (why did he do this?) will make it easier for others to refuse data to any opponent in future.
I don't agree that Roberts is beholden to release his data to those who have either belittled his study or (in some cases) actually insinuated, based on little more than speculation, that the study involved fraud.
Spagat has said that the Lancet paper contained "misrepresentations" of mortality figures suggested by other organizations.
Dr Madelyn Hsaio-Rei Hicks (Institute of Psychiatry in London) said in a letter to the Lancet that "I have started to suspect that they [Roberts et al] don't actually know what the interviewing team did." What did she base that conclusion on? Speculation, based on about how much time she believes the interviews would have taken.
If Roberts' refusal to give a few of these people data puts them on notice that there are consequences for making unwarranted claims and engaging in innuendo, then maybe they (or some other researcher) will think twice in the future before they run their mouth.
If it had been me, I would not even have given David Kane the data after the thoroughly unprofessional way that he behaved. Even his own superiors at Harvard gave him a tongue lashing for it. I suspect Roberts was in a bit of a bind in the case of Kane, however, because Roberts is presenting a paper at a conference this summer that Kane is also presenting at (on the Lancet study). So, I'm sure Roberts does not wish to listen to Kane whining about how he was not given the data. Better to give Kane the shovel and let him dig his own hole.
There are lots of qualified scientists who have behaved professionally in this case. My advice to Roberts at al would be to give them the data and to hell with the rest. Then again, Roberts does not need my advice. That is precisely what he has done.
By the way, if you want to read about David Kane's "unprofessional behavior" that I referred to above, you may read this
Kane's original post is no longer online, but the title of removed Kane Post pretty much says it all.
JB claims:
Untrue. For a few hours, there was a tongue lashing from a graduate student, not one of my "superiors." It was removed. Those interested can see the original post here. Linked from there are two tongue lashings, from Tim Lambert and Kieran Healy. Both are my intellectual superiors but not, the last time I checked, my superiors at Harvard.
Whether or not my behavior in that post (or since) has been "unprofessional" is a matter of some dispute. Although I regret the tone of that post, the substance still seems (to me) largely correct. I hope to document that claim at a later date and will pass on a link to Tim once I do.
To reiterate the main point: I have yet to find a single country-wide survey in any country on any topic at any time which, like Lancet II, featured a) a single contact attempt and b) a response rate > 98%. See my blog for endless details (with more to come).
If you know of counter-examples, please provide them.
Well, the digging is proceeding. As to whining, having the data will not prevent me from whining in the slightest. The principle is not: Give me (David Kane) the data. The principle is: Give all your critics (and fans and everyone else) the data. My main "superior" at Harvard, Gary King, has been fighting that battle for years. I am pleased to fight alongside him.
Give Spagat the data.
The tongue lashing that I referred to was from Amy Perfors, Author's Committee Chair for the Harvard IQSS (but I stand corrected, I now see she is actually at MIT).
Here's Perfors on the reason for removing your post (which I linked to above):
"The tone is unacceptable, the facts are shoddy, and the ideas are not endorsed by myself, the other authors on the sidebar, or the Harvard IQSS."
It would certainly appear that Ms. Perfors found your post just a wee bit unprofessional -- not up to the professional standards of the Harvard IQSS, at least.
By the way, your "theory" (expounded above) about why Roberts did not give the data to Spagat ("Spagat is viewed as a threat") is pure, unadulterated speculation.
John Tirman-I don't think the claim rejecting the use of the household number makes much sense since there is a much higher chance of double counting with the 53 percent figure AND we have a decent idea from other studies of how big an average Iraqi household is compared to what the average number of "relatives" and "friends" is. I think John Ridell's comments seem spot on here and they show why the 17 percent household number cannot be reconciled with L2, but here are some additional thoughts on the 53 percent number.
How we interpret the question that produced the 53 percent number depends crucially on what Iraqis mean by "relative" or "close friend." In Iraq, a very narrow reading of "immediate family" implies about 6-7 people and, given the tribal social structure, the number of people living outside the household considered to be close kin can easily number in the dozens. Throw friends in the mix, and we are talking about A LOT OF PEOPLE (20, 30, 50?). Even if the total number of relatives plus friends is, say, ONLY 10 (and we can be confident it is MUCH higher than that), 600,000 Lancet deaths (plus 1.8 million implied injuries--based on standard ratios--plus the many thousands of people who have been "harmed" in other ways, such as being physically harrassed, kidnapped, etc.), suggests that 100 percent of respondents should know a relative or close friend who is harmed. However, if there were 150,000 deaths (including combatants) (plus 450,000 injuries plus thousands of others "harmed"), and we assumed that 20 people classified as family and friends (personally, I know that my family and friends, even close friends, totals more than 20 and I have no one living on my father's side; my significant other, with a large Catholic family, has 20 aunts/unkles and 50 first cousins plus many friends), then we would expect a figure around 53 percent. (One tangent: I've looked at the original poll in pdf form and it actually says 47 percent know somebody and 53 percent don't -- but I assume this is a typo and that ABC/USA Today reported their own poll correctly.)
Also, remember, as a point of comparison, 17 percent of Americans in a recent poll say they know a fellow American who has been killed or injured in Iraq. And this is in the context of 3,200 American deaths and 23,000 injuries (about 5 percent of the lowest Iraqi IBC-based total), out of a total U.S. population of around 300 million (more than eleven times the Iraqi population). This means that physical injuries to 0.0087 percent of the U.S. population fighting a war 8,000 miles away has been sufficient to translate to 17 percent of Americans knowing someone who has been hurt. Yet we're supposed to believe that the total deaths (plus implied injuries) from L2, which together would represent 9 percent of the entire Iraqi population (not including those "harmed" in other ways), ONLY translates to 53 percent? In a society with social networks as big as they are in Iraq? John, do you seriously (truthfully) believe this?
David Kane said: "Whether or not my behavior in that post (or since) has been "unprofessional" is a matter of some dispute. Although I regret the tone of that post, the substance still seems (to me) largely correct. I hope to document that claim at a later date"
You really don't get it, do you?
When it comes to an issue as serious as scientific fraud, it is simply not acceptable to make a claim now and "hope to document that claim at a later date".
Regardless of what you may think, that is unprofessional (at least in the science and engineering world that I am familiar with. I don't pretend to speak for political "science").
I find this argument that scientists should make all data available to every person to be patently ridiculous.
Should I get the data? I am a professional magician with a love of science but no great love of stats. If I got the data what could I possibly do with it? I am unsure I could even display it properly and I doubt very seriously that any of it would make sense. So should working scientist have an obligation to provide me material of which I have no use for? Even worse, does giving me the material then obligate said scientists to respond to any stupid question I might have about material I don't understand?
Anyone want to defend my right to scientific data?
Careful, Colin--you're about to prove that the IBC number is too high if you don't watch out.
John Riddell--permit me to be doubtful about war statistics in general. I've seen estimates for the death toll in the Lebanon war that range all over the place. The same for the French/Algerian conflict. Claims for the estimated civilian death toll in Vietnam go from low hundreds of thousands up to 3 million (or maybe 4 million). The high end would suggest, by your claim, at least 6-8 million wounded. Where were the The Khmer Rouge killed 1.7 million people--I've never heard that they injured 3.4 million. How many were wounded in the Rwandan genocide? 1 million dead, I've read. If people are abducted, whether they are killed or merely wounded is entirely up to their captors and so the wounded to dead ratio in Iraq is going to depend on to what extent you have fighting (where there might be a high wound to death ratio) vs. outright massacres. There's no basic physical law governing what the ratio has to be.
One thing this Iraq debate has reinforced for me is that a great many numbers we read in the press or in books aren't based on very much, but I knew this to some extent from reading about other wars. There might be some wars where there are good data. But I'm suspicious of any claim that there has to be, for instance, at least 2 injuries for every death when death squads are involved. Again, there's no law of physics here--when a group of armed men goes into a village, for instance, they can wipe out nearly everyone and the dead to wounded ratio could be very high and there's no sociological law that determines the fraction of deaths caused by this sort of thing. We had someone (forget who) claiming that there was a refugee to dead ratio we could rely on--it turned out we could rely on it except when we couldn't. I hope nobody driven by misguided physics envy is writing textbooks with these sorts of "laws" of violent conflict being promulgated.
Supposing there were, however, 2 wounds per death then you still get a number from the ABC poll that is probably too high for IBC and too low for Lancet 2. It's been my guess all along that the death toll was in the low hundreds of thousands by 2006 (extrapolating from Lancet1's violent death toll ), and I was stunned and a bit skeptical when L2 came out, but I don't pretend to know this. It's just hard for me personally to swallow that we're missing 90 percent of the deaths. But we won't know, until there's another mortality survey (fat chance)conducted by a different group of experts.
And there probably won't be another mortality survey, because it's been the obvious thing for Lancet skeptics and governments and news organizations to do for the past 3 years and nobody has bothered to do it. Because they don't want to know or don't care.
Dominion
It costs a scientist nothing to post data on the internet or in an online archive. For many journals it's a (sometimes ignored) condition of publication anyway. In this way it can be made available to anyone who is interested, including professional magicians. If you couldn't do anything with it, why would you ask for it in the first place? Also worth pointing out that amateurs can do important science too. Albert Einstein, anyone?
The point about answering idiot questions is a reasonable one. It's not fair to expect "hand-holding". But the combination of the data, the published paper and any other SI ought to be sufficient to allow someone working in the field to reproduce the results. If they can't then I would say it's incumbent on the author to rectify what is a shortcoming in his output, by explaining in more detail what he did. If, however, someone outside the field can't reproduce the work, then refusing requests for help might be reasonable.
But John- who says other poll data is perfectly correct?
As for the wounded issue- as far as I understand what you are saying, it is that too many men are being killled or injured for your liking? Do we have any data on what age boys are considered men, and given we are talking about 3 years of conflict at what age they would be counted differently if injured or killed?
I'll agree it is interesting, and possibly a sign that there is some difficulty gauging the deaths properly, but it is hardly enough to sink the entire study, which also counted excess deaths due to reasons other than violence. And, as well as Donald says, nobody seems interested in doing another study to see what results people get. Would you be so interested?
Actually, Donald, I think the IBC number is too low (so do they!). Moreover, as my post on the ABC poll suggested, with the assumption that the average Iraqi considers 20 other people to be immediate family or close friends (the specifications of the poll question), 150,000 violent deaths (combatants and civilians) and 450,000 implied wounded gets you to the 53 percent number. The IBC total at the time of the poll stood at around 65,000 civilians killed by violence. Since this obviously misses deaths (although how many is an open question since it includes data reported in the media from morgues and hospitals -- you know, where those tricky death certificates in the L2 study come from) and includes only civilians, and since we can be confident that tens of thousands of insurgents, militia members, Iraqi army and police have been killed, 150,000 seems plausible. No one has yet come up with a set of assumptions that squares the L2 findings with the much more expansive ABC poll and, Donald as you point out, the L2 number is probably too high given everything else we know about the war.
Dominion:
Suffice it to say that there is no law that scientists have to provide their data (on the internet or anywhere else) to every Tom Dick and harry who wants it. Science has thrived in the absence of any such law and there is no reason to believe that the Roberts case is going to change that in the future.
You are perfectly correct (astute, actually) in bringing up the issue of time spent explaining the data to those a scientist has provided it to.
Putting the data online with a disclaimer saying "Take it as is" -- while great in principle -- is simply not going to satisfy everyone. In particular, it will not satisfy those who were not qualified to review the data in the first place (because they have no clue).
As to why someone who is not qualified to review the data might ask for it (or download it), there are any number of reasons, and not all of them are benign.
The way to throw doubt on a particular study is to provide enough information to make the critique look legit -- and it helps to have the raw data to concoct such bogus critiques.
The Burnham et al numbers are convincing yes, within their CI, and the other surveys and bits of data we have tend to confirm rather than undermine their numbers. The jujitsu of some of the posts above is just that---neither data nor statistical analysis derived from relevant data. One or two missing variables and it's dust. I mention the ABC News poll only because it indicates orders of magnitude in league with Burnham.
The numbers of refugees, the London Times story about the graveyard overflows, the statement about orphans, the MNF reports showing more violence by 3x outside Baghdad, the collapse of the health system, the polls indicating how Iraqis view the war's violence, among other such tidbits, point to exceptionally high numbers---but exceptional only in relation to the typical news reports that rely on lowest estimates, i.e., IBC's. They are not exceptional in view of what we know about other wars.
What we do know is that Burnham et al have conducted the only surveys where the question has been asked, and that no method critique has been put forward that undermines their conclusions. (Many lame attempts have been made, like the main street bias absurdity, but none stick.) They have released the data to competent---yes, competent---scientists, skeptics among them. I'm not sure what else can be asked of them.
If we want to understand the violence, we need to know its scale. Clearly, the violence is not understood by the US military---perhaps because it is the principal cause---nor by the news media that sees only 2% of Iraq. Burnham et al have provided us with the most reliable answers we have on this, and, yes, I believe the numbers are in the ballpark, based on everything else we know. A pity we don't know more, but that failing cannot be laid at the door of the Johns Hopkins School of Public Health.
It is interesting that after citing the ABC poll to support L2 in an earlier post, John Tirman then ignores everything that has been said in other posts with regard to the poll. Instead, John simply reasserts that the poll, which was much more expansive in its coverage than L2, supports the general magnitude of L2. As my posts and those by John Riddell point out, it does no such thing.
John is correct that available evidence suggests that the IBC count is too low, which IBC itself readily admits. Its coverage is partial and it does not include combatants or non-violent deaths associated with the invasion. However, in the latest period of the L2 study, nearly 1,000 people were dying PER DAY, with around 90 percent apparently going unnoticed by anybody. None of the other bits of evidence John mentions suggest that the scale of the violence is in Lancet-land.
The problems with the health system and the grave overflows suggest that many people are dying -- they tell us nothing about the magnitude. And the report of 900,000 orphans in Baghdd mentioned by John was based on a comment by a single municipal council member without any documentary support added. Not exactly good science John.
Violence is high outside Baghdad (especially in Anbar and Diyala), but EVERY report by MNF and other non-Lancet sources suggests that violence is higher on a per capita basis in Baghdad (polling about fears of violence bears this out too). L2's finding that violence is spread evenly across the country is not supported by anything else we know about the war.
The refugee crisis is aweful and worsening, but it doesn't support L2 either. At the L2 rate of 30,000 a month, we would expect far MORE refugees than the 50,000 a month estimated by the UNHCR. In civil wars involving extensive ethnic cleansing, you typically get ratios of refugees to deaths above 10 to 1 (often as high as 100 to 1). There are a handful of cases that have 2 to 1 ratios, but they are not countries (like Iraq) that have many neighboring countries to seek refuge in, and/or they are not countries (like Iraq) with extensive road and other transportation networks facilitating exit.
So, John, you can assert that L2 is confirmed by other tid-bits of evidence we have, but that assertion does not make it so. L2 is the outlier. Nothing else gets you close to their magnitude.
The calculations in this thread show no convincing evidence that L2 is an outlier. The sketpics are using speculative arithmetic--as I was--but for different purposes. We don't know enough, that is clear, and L2 is the best we have, with no serious method critiques holding up, certainly none that knowledgeable statisticians are putting forward. But we can learn a lot from other wars, it's true.
On refugees for example, let me demonstrate:
Guatemala 110,000 dead; 250,000 displaced. Ratio: 2.5-1
Burundi 200,000 dead; 300,000 total displaced. Ratio: 1.5:1
Angola 1.5 million dead; 4 million displaced. Ratio: 2.7:1
Congo: 1.5-3 million estimated dead since 1994; 2.7 million displaced. Ratio: 2:1 at the high end
East Timor: 200,000 dead; 450,000 displaced. Ratio: 2.25:1
Hence, the 2-3 million refugees from Iraq or internally displaced (not really counted) could easily demonstrate a mortality total of 500,000 or more. Ethnic cleansing is a recent feature of the violence in Iraq. Until a year ago, it was---and still is, in the eyes of the witnesses, Iraqis themselves---instigated by coalition forces, and I think we can agree that was not ethnic cleansing.
The Times' graveyard story did include magnitude, from 60 burials per day before the war to 200 last summer---a neat, if grisly, match to L2---but it was just one graveyard.
Each tidbit adds up to a massive amount of killing, and I see no reason to doubt it's in the Lancet range. More importantly, it tells us important lessons about the war--it expalins the insugrnecy as no other factor does--and it is that which should interest us all.
Sez Colin: "assertion does not make it so."
Followed, without a hint of irony, by: "L2 is the outlier."
Tell us Colin, does the ABC poll include deaths which took place prior to 2006? How do you know?
Kevin, if all I'd done is assert the opposite of what John Tirman said then I'd be guilty of the hypocrisy you acuse me of. Actually, my posts are substantive. If we look at all the other available evidence about what conditions are like in Iraq (studies on casualties by IBC, Brookings, UNDP, WHO, UNAMI, LA Times, official Iraqi and MNF data, and indirect evidence by the ABC poll, what we know about historical death to injury and death to refugee refugees, reporting about the scale, trends, and distribution of military operations, etc.), L2 cannot be reconciled with them in part or in whole. That makes it an outlier.
And your point on the ABC poll is silly. The Iraqis surveyed were asked if they knew a family member or relative "harmed" by violence during the WHOLE period of the war, NOT harmed this year. How do I know? I've read the poll (in original form and via media reports). The ABC poll implies a ceiling for plausible deaths that is much lower than L2 even though the poll included 6 additional months of the war.
John Tirman, the graveyard mentioned in the London Times story is the Wadi al-Salaam cemetery, the main cemetery in Najaf (and one of the largest burial grounds in Iraq, where many Shia from all around the country are sent for burial because the cemetery is near the Imam Ali Shrine (one of the holiest sites in Shia Islam). The story is from 2005 when a large number of civilian deaths were the result of mass casualty bombings of Shia civilians by Sunni insurgents. The story says that, on a bad day, the cemetery received as many as 200 bodies for burial. It is difficult to know how to interpret this number because it says "as many as 200" not an average of 200, and it is not a typical graveyard. As violence has gone up significantly, Shia identity has "hardened" dramatically from pre-invasion days (when Saddam repressed expressions of Shia identity), which means we'd expect a higher percentage of Shia deaths to result in burial in Najaf than before the war. This is obviously correlated to violence, but the relationship is likely nonlinear. Moreover, with an L2 rate of 1,000 deaths per day (many of them Shia) in the 2005-2006 period, we'd actually expect many more than 200 burials in Najaf in an average day.
Colin: The Iraqis surveyed were asked if they knew a family member or relative "harmed" by violence during the WHOLE period of the war, NOT harmed this year. How do I know? I've read the poll (in original form and via media reports).
Really? The version I saw read:
Question 35: "Have you or an immediate family member - by which I mean someone living in this household - been physically harmed by the violence that is occurring in the country at this time?"
What's your version and where can I find it?
What you say about other reports is untrue. For example, the UNDP figure is for war-related deaths only, not all violent deaths, still less excess mortality. It simply isn't comparable to either of the Lancet reports, since neither of them used a war-related deaths category. Most of the other reports you mention depend directly or indirectly on official Iraqi sources - they are passive counts, essentially.
Kevin, the civil violence started after the invasion. The question did not say "this year." It is certainly more reasonable to conclude that the question was inclusive of the war.
The UNDP study did measure excess deaths from war-related violence. Although it did not include crime, it is otherwise a good measure of excess violent deaths, which according to L2 is the vast majority of deaths. During the comparable time period, the UNDP numbers are about 4-5 times lower than L2. And, when you subtract likely combatant casualties from the major combat phase (estimated by the Project on Defense Alternatives to be around 9,000 for the invasion), the residual UNDP numbers are very close to IBC and other estimates. Again, L2 is the outlier.
Colin: It is certainly more reasonable to conclude that the question was inclusive of the war.
You think so? I don't. The phrase "at this time" doesn't sound at all the same to me as "from March 2003 to the present".
The UNDP study did measure excess deaths from war-related violence.
The UNDP didn't even attempt to measure excess deaths: the question asked related to deaths over the entire period covered by the survey, with no distinction made between pre-invasion and post-invasion deaths. It was assumed that deaths which were described as "war related" took place after the invasion. Perhaps you mean that, given that assumption, the estimated 24,000 war related deaths were a component of Lancet excess deaths. Not so; the Lancet definition of households would have excluded most soldiers. Also, there is no reason to suppose that all war related deaths were violent.
As for "comparable time period" we don't know the timing of the deaths reported to the UNDP. The interviewers didn't even ask. So obviously we can't say what the comparable time period is. For all we know they may all have taken place during the immediate post-invasion phase, though that's unlikely.
In short, we have far less information than you suppose and we have virtually nothing that can be used to test the two Lancet studies. Comparing them to the UNDP figures is pointless for the reasons I've just given. Comparing them to passive counts is obviously pointless: any estimate will be an "outlier", as you put it, when compared to a lower bound - a lower bound is not an estimate, however many hacks may choose to describe it as such.
As John Tirman says, it's a pity we don't know more, but it's not the fault of JHU.
Kevin, it isn't clear to me why Lancet excludes soldiers, but let's assume it does. If so, UNDP will count more deaths (because it has more categories of victimes), not fewer, relative to the Lancet.
In terms of excess deaths, you are right (I stand corrected), but again this just means that UNDP overestimates deaths relative to Lancet because it simply counts the total number without adjusting downward to account for the "normal" deathrate.
The UNDP question was posed in the summer of 2004 and asked about the previous 24 months (exact question: "Has any person(s) who was a regular household member died or gone missing during the past 24 months?"), meaning it will include some pre-war deaths. Again, this is a reason why it should overestimate deaths relative to Lancet. Moreover, once a respondent said there was a death, the questionaire asked them to clarify its source (exact choices: "1. Disease; 2. Traffic accident; 3. War related death; 4. During pregnancy, childbirth, or within 40 days after; 5. Other (specify)"). The number for "war-related" deaths was 24,000 which, given the title "war related" suggests these deaths occurred from March 2003-mid-2004.
Compare this number to L2's first time period (through mid-2004), which found about 90,000 excess violent deaths (and a mysterious drop in non-violent deaths).
Colin, those other sources you cite for death tolls are essentially the same set of sources--the LA Times and the UN and IBC all basically rely on either official sources or whatever numbers hospitals and morgues tell them, but official sources in wartime are often worthless and we don't know how well the hospitals and morgues keep track of numbers. If the health care system is falling apart keeping track of the numbers might be low on the priority list. And there might be political pressures (to put it mildly) to lie. I'd rather rely on surveys and if people are reluctant to accept one surprisingly large result from one survey, then have a different group of professionals conduct another.
As for the ABC poll, Kevin D's point about the time period might be valid--you don't know how the individuals responding to the question are going to interpret it if the pollster doesn't specify exactly what he means. So it might be missing some of the deaths in the earlier part of the war. Plus you can make varying assumptions about the average number of "harmed" persons per harmed household and about the fraction which are dead and when you take all that into account, about all you can say for sure is that the violent death toll is likely to be at least in the low hundred thousand range and possibly much higher. Even the 150,000 figure would mean the official sources are missing the majority of violent deaths. You could argue without any evidence that most of those missed are insurgents, not civilians, but that would still mean it is possible for coalition and Iraqi government sources to kill nearly 100,000 people without the press being able to count them or even have a sense of the scale except via polling. So why take the official numbers seriously?
There was an interview with Pedersen about the UNDP survey at the address below. Some of the most interesting material is in the comment section, where the blogger Stephen Soldz adds further Pedersen responses.
Notice that Pedersen (speaking off the top of his head) thinks the UNDP survey teams probably avoided the areas of heavy fighting in the spring of 2004, which if true means the war dead count was biased downward for that reason alone, apart from whatever other problems there might be with the survey question.
http://psychoanalystsopposewar.org/blog/2006/11/26/conversation-with-jo…
Donald, thanks for passing along the Soldz piece on Pedersen. Note a few things from Soldz's conversation with Pedersen:
"[Pedersen] struck me as extremely thoughtful and as having no ax to grind. . . [H]e believes the Lancet rates are too high."
"[Pedersen] may know more about middle east demography and survey methodology than anyone, so his judgments and intuitions cannot be simply dismissed. I would also remind people that Les Roberts refers to Pedersen as 'he highly revered Jon Pederson.'"
"Pedersen acknowledges . . . that the ILCS figures are low. But he thinks they are in the ballpark."
And, Donald, in response to your comment prompting the admission that the survey missed some violent areas/periods: "Even more relevant is that the ILCS fieldwork in both Fallujah and the Shia south was completed before the fighting. that is, by early April. In Lancet 2004, something like half the non-Fallujah violent deaths were from April on. . . . Further, Pederesen agreed with me that the ILCS could not be taken as a gold standard, since he admits an undercount. That said, he does think the undercount was not that high."
What Pedersen thinks isn't particularly important. The point is that the ILCS survey can't be used (as IBC did) to discredit Lancet1. It didn't count all the deaths.
I was sorta hoping for a response to my other point. Even a conservative reading of the ABC poll, one that would discredit Lancet2 if correct, still indicates that the majority of violent deaths in Iraq go unreported. And there's just no reason to think that most of those unreported deaths really are insurgents, though tens of thousands might be. But if the US is killing tens of thousands of insurgents, it's probably also killing a very large number of civilians as collateral damage. All without making it into the press. One would expect this--the US made a decision not to do body counts and if they aren't going to report the number of insurgents they kill, they surely aren't going to report the civilians.
We've also heard that if there are really 600,000 dead, then there'd an even larger number of wounded. Well, now we know there are many hundreds of thousands of casualties--are we supposed to think that the press could tell the difference between a country with 450,000 wounded and 1 million wounded?
Moral of the story--don't trust the official statistics. Surveys are the way to go if you want to know the real numbers and if you think Lancet2 is somehow flawed, then call for another mortality survey.
Well, Donald, what Pedersen thinks does matter given his expertise and his lack of any apparent political bias. He is certainly more qualified than most people posting here (including myself and you) to comment on the plausibility of the L2 findings, and he finds them problematic.
On some of your specific concerns, ILCS did count all the deaths that were recorded in their sample and they didn't avoid Fallujah, they simply did the survey in certain parts of the country (including Fallujah) before violence escalated there. But this shouldn't have a huge effect. Neither ILCS nor L2 count ALL the deaths that exist in Iraq -- they only count those in their sample and then extrapolate them across the country. ILCS picked a random sample and they sampled those places that
were selected randomly. Not every spot of violence will be included in a random selection. Plus, if L2 is right, violence is distributed fairly evenly across Iraq, in which case missing a number of "hot spots" during a particular point in the survey shouldn't affect the outcome too much. And, in terms of coverage, ILCS surveyed 21,688 households compared to 1,849 for L2. Given its larger sample size, better geographic distribution of samples, higher number of deaths recorded, and a far smaller confidence interval, the ILCS study probably superior than Lancet for the comparable time period.
In terms of your point about the ABC poll showing that the majority of deaths go unreported, this is both true and false as it relates to IBC. Yes, there are civilian deaths that don't make it into the media, although given that IBC uses Arabic and Iraqi sources in translation (in addition to Western press) and includes morgue and hospital data reported in the media, they may be missing fewer deaths than you assume. But keep in mind that these are only civilian deaths. I've seen estimates of 20,000-50,000 combatant deaths (which would include about 9,000 from the invasion). There have also been tens of thousands of Iraqi Security Forces (army, police, facility protection service members) killed. If, for the sake of argument, IBC is missing 30,000 out of 100,000 civilian deaths, that would not mean they were missing the "majority" of CIVILIAN deaths (which is the only fatalities they track). And if the remaining deaths approach 50,000, then we get the 150,000 figure I used in my earlier posts on the ABC poll.
We don't know how many civilians have been killed by U.S. forces. My own estimate of Iraqi civilians killed by U.S. forces, based on media reports and escalation of force (EOF) data, put the total at 8,500-15,000 through the end of 2006. There is a reason to believe that many (although probably not all) of the civilian deaths resulting from U.S. raids and combat operations actually do get reported since anti-coalition forces have incentives to play up these numbers and they have ready access to media. This is less true of EOF incidents at checkpoints and alongside convoys, but that data is tracked and we have some sense for the numbers here.
Your assumption that more civilians must be killed relative to insurgents may not be true. During "major combat" (March 19-April 30, 2003), most available data suggests that about twice as many combatants were killed relative to civilians. And, it is important to remember, most of the combat during the invasion was actually not against uniformed Republican Guard troops. Rather, it was against irregular Saddam Fedayeen that were largely indistinguishable from the wider civilian population. Even in Fallujah II, which was a devastating offensive in which perhaps 1,200 insurgents were killed, most civilians (estimates suggest 90-95 percent) left the city before the assault. In the next biggest assault after Fallujah, in Tal Afar in September 2005, U.S. forces killed or captured hundreds of insurgents but because the Serai District where the fighting occurred was evacuated first, only 3 Iraqi civilians were caught in the crossfire.
You're missing the point regarding the ILCS survey. It wasn't a question of randomly missing hotspots--Pedersen said they deliberately avoided the hotspots. That's not a random sampling error, that's a bias in the downward direction. The fighting in the spring of 2004 was reportedly very intense, so they might have missed a significant number of deaths. Neither you nor Pedersen is in a position to say otherwise. All this aside from the vagueness of the question regarding "war-related deaths". If someone in a family was shot by unknown persons, would the respondent count it as "war-related"?
I went back and looked at the IBC criticism of Lancet 2 released in October 2006. One of their main arguments was that there were only records of 60,000 wounded treated in Iraqi hospitals from mid 2004 to mid 2006. Well, your reading of the ABC poll is that there have been 450,000 wounded from 2003 until early 2007 (and we don't really know if the poll respondents were thinking of the entire occupation period either.) If it's going to be hard to reconcile Lancet2 with Iraqi hospital records, it's not going to be much easier to reconcile the ABC poll with those same records.
The ABC poll probably didn't catch military deaths in the invasion phase--those people would have lived in barracks, not in households. Suggesting 50,000 combat deaths in the post-invasion phase (a figure I've also seen cited once or twice) really is a pretty damning illustration of how little we really know about the level of violence in Iraq. It's also more than a little trusting to imagine that if the US military really has killed 50,000 people in the post-invasion phase then all these people are insurgents. On those rare occasions when the press can investigate a US military claim regarding dead insurgents they've found (from what I recall) that the locals give quite a different story. If we had actual investigations into each of those estimated 50,000 insurgent deaths I suspect there'd be a very wide range of claims regarding both how many really died and how many of those dead were civilians.
Also, think through the implication of what you are saying if the 50,000 dead insurgent figure is correct. You like the 3 to 1 wounded to dead ratio , so that's 200,000 dead or wounded people living in Iraqi households who were injured or killed by coalition forces. I don't think it's so easy to make a distinction between insurgent and civilian sympathizers or even bystanders in guerrilla wars, but suppose all those people had weapons and were using them to fire at the US forces. Then that's 200,000 people taking potshots at US forces. John Riddell thinks there's only about 1 casualty per "harmed household", so that's 1.5 million people in households where someone takes potshots at coalition forces.
In hindsight, I think it was a mistake for IBC to try to separate out civilians from others, because in a guerilla war that's notoriously difficult to do. And what it has done is allow people to think the level of violence is much lower than it actually is, even if we disregard the Lancet . By your conservative estimate (and my own based on the ABC poll would be higher) is that the true death toll is 2.5 times greater than IBC's claim, and the proportion killed by the US is much larger than is realized, but that's all right, because most of those killed by the US were "insurgents".
You mentioned incentives to report deaths. The Iraqi "resistance" seems hostile to reporters--it's why you don't have very many reporters of a leftist bent traveling with the insurgents giving their view of the war. I think there was a little of that in 2003 and early 2004, but not so much afterwards. And if Arab media outlets tried doing this the US and Iraqi government forces would consider them fair game. Also, incentives aren't always what you'd think. The Vietnamese government now claims it deliberately understated the number of its own dead, both civilian and military, so as not to lower morale during the war.
Donald, you are wrong about ILCS. They did not "avoid" Fallujah and the Shia areas where violence spiked in April 2004, they simply completed their fieldwork in these areas before the spike happened. So there is no evidence of systematic bias (remember: it is L2 that says violence is the same everywhere, and the ILCS included many more clusters).
It is difficult to know how to judge the gap between the wounded records and the number of implied wounded in various estimates. Some wounded will not seek medical care because their wounds are not severe enough or it is too dangerous to travel to (or get care from) hospitals. So, yes, the records from hospitals are likely to be BIGGER undercounts than hospital, morgue, and cemetery records of deaths (you know, the places that hand out those death certificates everybody surveyed by L2 had). But I think you'd have to admit that the shear scale of the 1-1.5 million implied wounded in L2 (IBC was being VERY conservative by suggesting only 800,000 implied wounded from L2), most of which never (apparently) sought medical care, is very strange and implausible.
Your adduced implications from my estimate of 50,000 combatant casualties are also problematic. SOME of these deaths ARE uniformed (Iraqi Army during major combat, Iraqi security forces now), and therefore easier to distinguish from the civilian population. Many other combatant casualties (especially the ISF) have been killed by anti-coaltion forces not the U.S. So how many civilian-appearing "insurgents" and "militia members" has the U.S. killed? No idea, but if we subtract 9,000 Iraqi army deaths and 15,000-20,0000 ISF deaths (I've seen estimates of 12,000 Iraqi police alone killed since the war began), then the residual number of hard-to-distinguish anti-coalition forces would be 21,000-26,000 (again, these are all guesses based on bad data).
The notion that these 21,000-26,000 implies 3 times more people "taking potshots" at coalition forces (based on the death to wounded ratio) reveals a misunderstanding for how the war is fought. Insurgents usually operate in small cells and the cell members may be spread out. There is no reason to believe that insurgents killed in ones and twos necessarily produce 3 wounded insurgents for every dead one. But let's say that they do, then this would imply perhaps 100,000 people living in households that were killed or wounded during engagements between U.S. forces and individuals assumed to be combatants. Many of these wounded will include noncombatants caught in the crossfire. But these are not necessarily all "extra" casualties to add to my estimate of 8,500-15,000 civilians killed by U.S. forces (and the implied 25,500-45,000 civilians wounded), since this estimate includes many "crossfire" deaths.
Are the ratios of combatant to civilian deaths attributable to U.S. forces based on my numbers plausible? Perhaps. You actually completely ignored my substantive comments and examples from various large-scale engagements where the number of civilian deaths may be much lower than one-to-one.
Now you are correct that there are certainly deaths that coalition forces think (or say) are insurgents that may actually be noncombatants that never demonstrated hostile act (by firing a weapon) or hostile intent (by threatening imminent attack, planting an IED, etc.). How many? We don't know. There are cases where locals have said those all or most of those killed were civilians not insurgents (the Ishaqi incident, where the U.S. claimed that it was responding to hostile fire and locals assert a civilian massacre, is a good example). Further investigation sometimes bears these accusations out -- but at other times it does not. The insurgents are not stupid. They understand the power of accusations of excessive civilian casualties in de-legitimizing U.S. actions, so they adopt tactics that invite civilian casualties and have huge incentives to advertise deaths that do (and do not) occur.
Which brings me to your media argument. Again, you seem to not understand the way insurgents do business. EVERYTHING they do is aimed at creating impressions (what the military calls "information operations"). They cannot win on the battlefield in a traditional sense -- so they have to create "impressions" (what some people call "propaganda of the deed"). They want to create the impression that the occupiers cannot win and they want to create the impression that the occupiers engage in widespread atrocity (to remove any residual legitimacy created by the notion that coalition troops fight "justly" even if the war itself is unjust). They do not, as a rule, embed large numbers of Western journalists with their cells (with some exceptions, e.g., Nir Rosen), but you are wrong about their access to, and use of, the media. The estimate of 600 civilians killed by U.S. forces in Fallujah I, for example, was reported by an Al Jazeera cameraman accompanying insurgents. This number was then picked up by the AP and echoed throughout other Western media outlets without any attempt to corroborate it (it may be true or false, but that is not the point here). Insurgents operate their own newspapers (through fronts), TV stations, and regularly film attacks and broadcast images of coalition-caused civilian deaths on the internet. They have no problem getting their message out to Iraqi and regional Arab media, both newspapers and satellite-TV channels -- many of which are broadly sympathetic to Sunni Arab resistance in the face of Western occupation.
So, I return to my main point. If the coalition-caused civilian deaths are as high as some suggest (L2 suggests 186,000), many if not most of these should show up in the media and, either through diffusion to Western outlets or through translation, appear in IBC. The exception is deaths that occur at checkpoints and alongside convoys (EOF incidents that occur outside of combat with insurgents and are likely to be under the radar of the media). That is why my estimate of 8,500-15,000 is derived by ADDING EOF estimates to ones from IBC and Brookings.
John Tirman: 'East Timor: 200,000 dead; 450,000 displaced. Ratio: 2.25:1
Hence, the 2-3 million refugees from Iraq or internally displaced (not really counted) could easily demonstrate a mortality total of 500,000 or more. Ethnic cleansing is a recent feature of the violence in Iraq. Until a year ago, it was---and still is, in the eyes of the witnesses, Iraqis themselves---instigated by coalition forces, and I think we can agree that was not ethnic cleansing.
Coming a bit late to this thread, but, you know, that seems to be a very sketchy method of predicting mortality. For instance, if we take the example of Afghanistan, in the first few years after the overthrow of the Taleban, it's well-known that millions of Afghans returned to their country, but that doesn't mean that noone died in that conflict. It's possible, although I don't know, that this was the case too in the first few months after the invasion of Iraq, with exiles going home. Whether and in what numbers people flee is so conflict specific it's hard to draw such a conclusion.