My post on the Lancet article has attracted a fair amount of comment, both in the comments here and on other blogs. On the whole, those who have addressed my criticisms have disagreed with them. I’ve read the criticisms and re-read both the new and the 2004 Iraqi death toll studies a couple of more times. Between the two, I’ve become convinced that some (but not all) of my earlier concerns were unjustified.
In this post, I’m going to try to respond to most of the substantive criticisms (and a few of the other comments). I’ll let you know where my views have changed, and I’ll try to clarify or restate my position where they haven’t. Further discussion is, as always, more than welcome. I’d always rather have someone let me know that I’m saying something stupid than walk around ignorant of that fact. Well, almost always, anyway. Just try not to bruise my ego too badly, OK?
1: Extraordinary results:
I’ll start with a comment that doesn’t actually address anything I said, but which does illustrate something important. Wolfwalker wrote (in the comments here):
The study claimed that 650,000 Iraqis have died as a direct or indirect result of the invasion, right?
Has it occurred to any of the study’s defenders that 650,000 amounts to 2.5% of Iraq’s total prewar population? That the study claims that most of the deaths are among males of military age — which would make the death toll almost nine percent of that demographic group? And yet no other source anywhere gives a figure that’s within an order of magnitude of the study’s figure. No one else seems to have even noticed the disappearance of one out of every eleven adult Iraqi men.
This latest Lancet study does make a claim that is clearly at odds with almost all of the limited information that was previously available on the death toll in Iraq. The paper does point out that the number of excess deaths that they calculated represents about 2.5% of the population of the country. The death toll that they calculated is in fact an order of magnitude greater than any other estimate. Those are extraordinary claims.
I think it is reasonable to subject extraordinary claims to an increased level of scrutiny – and that’s why I’ve put so much time into examining this paper. What’s not reasonable is the approach that Wolfwalker, President Bush, and many others have been taking. An extraordinary claim that is supported by evidence-based study needs to be addressed on its merits. It should not be dismissed out of hand simply because the conclusions don’t fit someone’s pre-existing view of reality.
2: Non-violent excess deaths:
In the earlier post, I argued that including the increase in the non-violent death rate since the start of the war was unjustified, since that increase was shown to be unjustified. Josh, over at Thoughts from Kansas, thinks that this is a valid criticism. Several of my commenters disagree, pointing out that if you try hard enough, you could probably subdivide the cause of death finely enough that no individual cause of death would show a significant increase. Another commenter points out the difficulties involved in identifying causes of death.
Those criticisms are legitimate and compelling, but there is an important point that they fail to address. I am not the one who separated out non-violent deaths and tested the increase for significance. The authors of the study did. They were confident enough in their ability to identify non-violent deaths to list them separately, and they decided to test that particular increase for significance. Looking at the companion (non-peer-reviewed) article, it appears that they did so in an attempt to detect evidence for deaths resulting from the deterioration of the health care system, and had they found a significant increase, I’m sure that they would have talked about health care deterioration in the Lancet article.
If they had not tested the increase in non-violent deaths, I would have no complaint. However, I think that including those deaths after they tested for, and found no, significance is highly questionable. I think this is especially true given that the non-violent death rate was actually lower than the prewar estimate in two of the three periods they covered in the study.
3: Selection of cluster sites.
I was concerned that the way in which the clusters were selected resulted in a heavy bias toward urban areas. I got hammered on this one about eighteen different ways, and my critics were right. The site selection process was reasonable, and probably resulted in a sample that was representative of the population of the country.
That having been said, I still would have been happier if the authors had used a stratified sampling approach, and examined urban and rural areas separately. One of the commenters here did not think that there was any reason to think that the death rate would vary between urban and rural regions, but I think there might be. Sectarian violence is going to be more likely to occur, I think, where there are groups from the rival sects living in close proximity to one another. That is probably going to happen more often in cities. I do realize, however, that logistical and (more importantly) safety concerns would make it difficult to use that type of sampling design.
4: Choice of population figures
I was concerned by the study authors’ decision to use one estimate for the population of Iraq without mentioning two other ones. Opinion seems to be split on this one. Josh’s criticism was the clearest:
Choosing a higher population estimate does inflate the mortality estimate to some extent, but unless there’s some basis for thinking the UNDP estimate is worse than other estimates, there’s no basis for claiming that this is a flaw, or even a bias (except in comparisons between this study and the earlier one).
The thing here is that I do think that there’s some basis for at least questioning whether the UNDP estimate was better than the others. As far as I can tell, three independent studies of the population were conducted. I don’t know what techniques were used in any of them, but I’m reasonably certain that none was a census. Two of the three studies produced estimates that were similar to each other – within a few hundred thousand individuals. The third produced an estimate that was 1.6 million individuals greater than the higher of the other two studies. With two similar estimates and one dissimilar one, I think it is quite reasonable to ask why the dissimilar one was favored over one of the other two – particularly when selecting one of the others would have resulted in an estimate of about 75,000 fewer deaths. One reader did point out that better censuses usually give higher numbers. However, it does not appear that any of those estimates was derived from a census, and I’m not sure that better estimates always are larger.
5: Responsibility for the deaths
I was a bit upset when I was writing the earlier part, with the result that my writing probably wasn’t as clear as it should have been. A number of people have pointed out that we are responsible for the deaths that have resulted from sectarian violence, and they are right. The United States bears the bulk of the responsibility for what has happened and is still happening in Iraq. The reason that I was upset that the actions of coalition troops were singled out for criticism in the paper while sectarian violence was ignored is because there is a difference between what we, as a nation, are responsible for and what the troops on the ground are responsible for. As a nation, we’re responsible for pretty much the whole bloody mess. We managed to conduct a well-orchistrated military operation, but as a direct result of unbelievable bungling totally screwed up everything after that. The disasterous occupation set up the conditions for all of the ongoing violence. That does not, however, mean that the troops on the ground are responsible for all of the deaths. That might not seem like an important distinction to everyone, but it is to me. I’ll have a separate post talking about some of the reasons for that tomorrow.
6: Me as a PR hack
One of the comments today was a bit harsher than most:
I am a demographer and a statistician, and I offer one major criticism of his attack on this paper. He fails to show how any of the points he lists would result in any substantial changes in the final estimate.
Actually, I pointed out that using the lower population figures would lower the number of excess deaths by about 10%, and that excluding the non-violent excess deaths would knock another few percent off. At the time, I wasn’t sure what effect the methods used to select the clusters would have, and said so. (And as I mentioned already, I’ve since figured out that I was wrong on that one anyway.) You might not think that those changes are substantial. If so, I’d honestly be interested in hearing why.
In general Dunford is using a standard PR technique to create the impression that there a lot of problems with the study and it should not believed without identifying a single real problem that, if corrected, would change the results. The more the uninformed reader reads sentences that notes apparent problems while discussing the article, the less the uninformed reader is going to trust the findings reported in the article. Remember the goal of science is to discover new knowledge while the goal of public relations is to create a false image to manipulate the public. It is clear that John Hopkins study is science and Dunford’s tract pure PR.
That is absolutely not what I was attempting to do. In fact, I explicitly said at the start of the article that I thought the new study represents the best available estimate. At the end of the study, I said that I was convinced that the real death toll is much higher than the official estimates. I wrote the post for two reasons, and two reasons only. The first is that I was genuinely interested in finding out if the things that I thought were problems with the study really were – and I don’t know of many better ways of finding that out than going ahead and throwing the questions out there for others to read and evaluate. The second reason is that I think that there is a great deal to be learned through critical examination, and little to be gained from not evaluating things critically.
A final note:
The more I read the Lancet paper, and the more I examine my own concerns with that article, the more reasonable their conclusions appear. I still think that their number is likely to be on the high side, but I don’t think it’s going to be high by as much as I would have liked to believe. In fact, I think that the actual number is quite likely to fall within their stated 95% confidence interval. It’s painful to think that we have so much blood on our hands as a nation, but the concerns that I still have with the paper’s methodology are almost certainly not severe enough to account for even half of their estimate.
For a while I’ve been of the opinion that we could not morally withdraw from Iraq, because withdrawing would result in more harm to the Iraqi people than staying. It looks like I’ve got some hard, hard thinking to do about that now.