Slate has published a response from Burnham and Roberts to Kaplan's botched criticism of Lancet 2.
Kaplan's latest article focused on two baseless criticisms of our 2006 study. First, he claimed that our measured base line rate, the rate of natural deaths for the year before the invasion, was too low. We had estimated the rate to be 5.5 deaths per thousand per year. Kaplan claims that the rate was really 10, according to U.N. figures. He wrote, "[I]f Iraq's pre-invasion rate really was 5.5 per 1,000, it was lower than almost every country in the Middle East, and many countries in Europe." This is just wrong! If Kaplan had checked the U.N. death-rate figures, most Middle Eastern nations really do have lower death rates than most European countries, and in fact have lower death rates than 5.5. Jordan's death rate is 4.2, Iran's 5.3, and Syria's 3.5. The reason for the lower rate is simple: Most Middle Eastern nations have much younger populations compared to most Western nations. Obviously, the elderly die at a greater rate than young people and account for a disproportionate number of deaths. ...
Kaplan's second fault with our study was his claim that our sampling suffered from "main street bias." That is to say, we systematically missed the back streets and focused on main streets, where he assumes more deaths have occurred. This claim is not credible for a number of reasons. Our study team worked very hard to ensure that our sample households were selected at random. We set up rigorous guidelines and methods so that any street block within our chosen village had an equal chance of being selected. Once we started, we went to the next nearest 39 doorways in a chain that typically spanned two to three blocks. Thus, the first-picked block usually did not provide most of the houses within a given cluster. It is also important that most violent deaths probably happened outside of the home, making the location of the house on the street irrelevant.
Kaplan's response to this is intellectually dishonest. There is absolutely no question that he was wrong to claim that 5.5 is a lower rate than most countries in the Middle East, but Kaplan refuses to admit that he was wrong.
On "main street bias" Kaplan offers:
The second part of their rebuttal on the matter of "main street bias" -- that people might have been killed away from home, so it doesn't matter if the houses surveyed were disproportionately close to violence -- might be valid for many countries. But in Iraq, unemployment is high, transportation is limited (in part due to scarcity of gasoline), and in many cities, Sunnis don't go into Shiite neighborhoods, and vice versa. This controversy would be easier to settle if Burnham and Roberts were clearer -- and less contradictory -- about just how their main streets were chosen.
Does Kaplan think that Sunni neighbourhoods don't have main streets? Or is it that Shiite neighbourhoods don't have main streets? People who live in back streets will have a main street within walking distance where they would go to buy stuff.
First, the issues at hand concern statistical methods, for which my sources have impeccable credentials.
I wonder which one of these experts provided him with the dartboard analogy.
What an überwanker.
I see that Jon Pederson of the UNDC thinks that Lancet 2's recall period of over 4 years is much too long. Of course it's not surprising that Pederson would agree with the wide consensus among experts in crisis-mortality studies.
Also, since "main street bias" has become so controversial, you'd think that Burnham & Roberts would release a more detailed description of their sampling methods. Perhaps they could a page or two to their on-line report. I'll believe that they "set up rigorous guidelines and methods so that any street block within our chosen village had an equal chance of being selected," when they actually describe these guidelines.
Except that the consensus seems to be against him here. The SMART protocols are irrelevant since they don't take into account the use of death certificates to control for recall bias. And epidemiologists seem to be solidly behind the methodology used, as of course the Lancet peer reviewers must have been.
Furthermore, as mentioned in a previous thread, DHS mortality surveys have routinely used recall periods of several years (up to ten in some cases).
Probably not necessary unless the controversy is actually reflected within the epidemiological community. I don't see that even Pederson argued for "main street bias" being a real problem.
Burnham & Roberts said: Kaplan's response to this is intellectually dishonest. There is absolutely no question that he was wrong to claim that 5.5 is a lower rate than most countries in the Middle East, but Kaplan refuses to admit that he was wrong.
Alas, B&R are utterly econonomical with the truth. Normal academic etiquette REQUIRES that they refer to the WHO life tables for the region. These show entirely plausibly, given that Iraq was subject to UN sanctions severely limiting Iraq's oil exports (c.90% of the total?), that Iraq's crude mortality rate in 2001 was nearly double that anywhere else in the region.
Anton,
As I've pointed out in previous threads, the DHS questionnaire is very different from the Lancet questionnaire.
The epidemiologists I've seen quoted in the press have mostly been supporting the general idea of random sampling. I doubt that most have reviewed the study in any detail. Certainly some epidemiologists and demographers are sceptical: Daponte, Pederson, Apfelroth.
Apelroth is a pathology professor, not an epidemiologist, which is why the media reports his opinions, along with those of economists, physicists, guitarists etc rather than those of epidemiologists.
Daponte and Pederson have been filtered through Kaplan, who I don't trust.
Article republished from Nature here:
http://tango.gnn.tv/blogs/19286/Iraqi_death_toll_withstands_scrutiny
The discrepancy does not invalidate the new result, and if the researchers underestimated the pre-war death rate, it's possible that they may have also underestimated the post-war rate. But some researchers say the paper should have addressed the issue. "There should have been more introspection," says Beth Osborne Daponte, a demographer at Carnegie Mellon University in Pittsburgh, Pennsylvania. "That increased my discomfort."
Other researchers share that discomfort. Debarati Guha-Sapir is director of the Centre for Research on the Epidemiology of Disasters in Brussels. She has some methodological concerns about the paper, including the use of local people -- who might have opposed the occupation -- as interviewers. She also points out that the result does not fit with any she has recorded in 15 years of studying conflict zones. Even in Darfur, where armed groups have wiped out whole villages, she says that researchers have not recorded the 500 predominately violent deaths per day that the Johns Hopkins team estimates are occurring in Iraq.
An independent account of Pedersen's misgivings:
http://blog.washingtonpost.com/worldopinionroundup/2006/10/is_iraqs_civ…
Pedersen said he thinks the Lancet numbers are "high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much."
Ragout wrote:
It is certain that some are, because there are a lot of epidemiologists and demographers out there and there ain't much that we all agree on. However, I personally have witnessed roughly about as much scepticism about the basic approach taken by Roberts and Burnham as there appears to be among climate scientists on AGW.
"Other researchers share that discomfort. Debarati Guha-Sapir is director of the Centre for Research on the Epidemiology of Disasters in Brussels. She has some methodological concerns about the paper, including the use of local people -- who might have opposed the occupation -- as interviewers. She also points out that the result does not fit with any she has recorded in 15 years of studying conflict zones. Even in Darfur, where armed groups have wiped out whole villages, she says that researchers have not recorded the 500 predominately violent deaths per day that the Johns Hopkins team estimates are occurring in Iraq."
Darfur has an estimated population of around 7.3 million people.
In approximately 3 years of conflict 400-450,000 people are estimated to have died. (Violent deaths would be a subset of these since many Darfuris have died of hunger, thirst, exposure and lack of medical care.)
450,000 excess deaths out of 7.3 million over three years is an excess death rate of roughly 20,000 per million per year.
600,000 excess deaths in an Iraqi population of 25 million over three years is around 8,000 per million per year.
If we accept that there have been relatively few (if any) excess deaths in Iraqi Kurdistan and that the Kurds represent roughly 20% of the Iraqi population we're talking about 200,000 excess deaths in a non-Kurdish population of around 20,000,000 implying an excess death rate of around 10,000 per million per year.
So, the Lancet study does not imply that the rate of excess deaths in Iraq is greater than or even equal to that in Darfur.
Ian Gould writes:
"If we accept that there have been relatively few (if any) excess deaths in Iraqi Kurdistan and that the Kurds represent roughly 20% of the Iraqi population we're talking about 200,000 excess deaths in a non-Kurdish population of around 20,000,000 implying an excess death rate of around 10,000 per million per year.
So, the Lancet study does not imply that the rate of excess deaths in Iraq is greater than or even equal to that in Darfur."
What doesn't fit is the claimed number of *violent* deaths in Iraq in relation to what else is going on. Burnham et al. think their best guess is 600,000 violent deaths. Per capita, that's a level of violence on about the scale of the Bosnian War, or what happened in the Lebanese Civil War over more than fifteen years. It might be on the scale of violent killing in Darfur, depending on which estimate you believe.
The puzzling thing is that you don't see the level of displacement that level of violence has produced elsewhere. Or the breakdown in basic health care.
Nor do the sectarian killers in Iraq seem to have anything like the freedom to operate available to the Bosnian Serbs or the killers in Darfur, where "armed groups have wiped out whole villages". The killing in Iraq seems to be carried out retail rather than wholesale.
Half a million Iraqis have been displaced to Syria alone, and it doesn't look like the flow of refugees is going to slow down anytime soon. It's true that the displacement took a while to get going; I suspect the average Iraqi, unlike targeted groups in Rwanda or the Balkans, wasn't initially expecting to have to flee. Not that I have any particular evidence on Iraqi psychology in that area.
I would say that when injured Iraqis are avoiding hospitals because death squads periodically visit them, basic health care has broken down.
As is Pederson's survey.
Perhaps, but the peer reviewers would necessarily have done so.
Not that that makes Roberts et al. methodologically impeccable, of course. But virtually all the epidemiologists and demographers who have pointed out flaws have gone on to say that, so far, it's by far the best mortality estimate we've got. It can be improved upon, but it can't be rejected.
While we're comparing Iraq and Darfur:
- what proportion of the deaths in Darfur have been reported in the English-language media?
- How come the Sudanese Ministry of Health hasn't recorded these deaths?
Seriously though, the comparitively low reported level of additional nonviolent deaths in the Lancet is one of the few points that does raise legitimate questions about the study.
It's possible that the improvements in health and nutrition in the relatively peaceful northern and southern areas has offset a war-related increase in mortality in Baghdad and the Sunni regions.
Then too, there were surprisingly few civilian deaths amongst civilians reported during the second Battle of Fallujah. Normally if several hundred thousand people are forced from their homes, you'd express a big increase in deaths from disease, exposure and malnutrition. That appears not to have happened there and has been attributed to the Iraqi extended family and tribal systems which meant most families were taken in by their relatives.
Is it possible that because they blame the US for the war, or because they fear retribution by insurgents if interview information is leaked, interviewees have attributed nonviolent deaths or insurgent-caused deaths to American forces?
Of course, if there are legitimate problems with the Lancet study the proper response is to seek to address them and to want a better survey done not to say "See, those commie-fag-liar-traitors want Saddam back. Praise the lord and pass the ammo."
It's interesting too that the critics of the first Lancet study were basically supporters of the war effort who also insisted that the war was going wonderfully well. (Remember Seixon's diatribes about how only a total lunatic would suggest there was, or could be, a civil war in Iraq?)
It's now pretty generally accepted that the war has been in Tony Blair's words "a catastrophe".
So what drives the current wave of critics?
Many possible most are genuinely concerned by what they see as problems with the study. Others just seem to be so emotionally invested in their position that they can't shift from it.
I think the owrd for the latter group is "bitter-enders".
Yeah, if people have some doubts about this study I don't blame them, but what makes me suspicious of the motives of many critics is that they don't all join with the Lancet team's call for more investigations to confirm or deny their conclusions. It's to the great credit of Burnham and Roberts that they keep pointing that if 2.5 percent of the population has died it ought to be easy to confirm this level of violence. There's little point in continuing to argue that the study might be flawed for this or that hypothetical reason, or to argue that it can't be right because the official statistics are much lower. Settle the issue with more studies. It's apparently possible to take polls in Iraq on very dangerous subjects, such as what people think about attacks on US troops, so it's not like a poll on mortality is just out of the question.
I think the word for the latter group is "bitter-enders".
Hmmm...hmmm...
I personally think these folks were attracted to the seductions of war (because they didn't have to serve), thought that their wargasm was love, and now they've awakened to a coyote ugly plumper and don't know how to say goodbye.
So, I guess that, yes, we're back to Ian's 'bitter-enders'.
Alrighty then. Prize to Ian.
Best,
D
Iraqis are being displaced, but apparently not at the level you might expect from a conflict as violent as the civil wars in Bosnia, Lebanon, or Darfur.
Darfur in particular has resulted in a radical increases in nonviolent deaths among those affected by the fighting. Burnham et al. seem to have expected a similar increase in nonviolent deaths in Iraq post invasion. The failure of the 2006 survey to detect a statistically significant increase in nonviolent deaths raises questions.
If the Burnham et al. estimate of violent deaths is correct, why isn't there more collateral damage? And conversely, if their estimate of nonviolent deaths is correct, how is Iraq managing to carry on a such a violent conflict with so few refugees and without a significant increase in other deaths?
If the Burnham et al. estimate of violent deaths is correct, why isn't there more collateral damage? And conversely, if their estimate of nonviolent deaths is correct, how is Iraq managing to carry on a such a violent conflict with so few refugees and without a significant increase in other deaths?
That's a lotta FUD packed in a small space. Congratulations.
Best,
D
Will McLean:
If the Burnham et al. estimate of violent deaths is correct, why isn't there more collateral damage? And conversely, if their estimate of nonviolent deaths is correct, how is Iraq managing to carry on a such a violent conflict with so few refugees and without a significant increase in other deaths?
Will, we know for a fact that there were a very significant number of people displaced from Fallujah before the US retook the city from the insurgents. We know this because the US forces were at great pains to evacuate noncombatants before the battle.
Deaths among that group appear to have been much lower than one would expect from a comparison with Darfur or other conflicts.
I've suggested that one reason for this might be the extended family system in Iraq. Alternately it might be due to the work of the Iraq Health ministry (which back before sanctions at least was apparently very well-run and highly competent); the international aid groups and the US military.
If we know for a fact that 100,000+ people can be displaced for several weeks with minimal loss of life it becomes less
remarkable that even larger groups of people can be displaced with low loss of life.
Recent reports say that 100,000 Iraqis a month after leaving the country but these people aren't ending up in refugee camps for the most part. They're the middle classes and those lucky enough to have family in either the west or other Arab countries who can assist their departure and help them provide for their families.
"I suspect the average Iraqi, unlike targeted groups in Rwanda or the Balkans, wasn't initially expecting to have to flee." I hadn't though of that, but it sounds reasonable. After all, in the Balkans, Sudan and Rwanda specific groups were targeted, in large part to deliberately spread terror and force them to flee.
Ian,
You may be right that the rate of excess deaths is about the same in Iraq and Darfur. But Guha-Sapir is also right when he says that Burnham et al are reporting a rate of *violent* deaths that is much higher in Iraq than in Darfur.
Last year, Debarati Guha-Sapir and Olivier Degomme evaluated 24 (!) mortality surveys from Darfur, and came up with the following figures for the period September 2003 to January 2005.
Total number of deaths 134,000
Excess number of deaths 120,000
Deaths due to violence 35,000
(alternative estimates of total deaths: 98,000 and 181,000)
Guha-Sapir and Degomme write: "Mortality in Darfur, like in most other complex emergencies today, is largely caused by infectious diseases and malnutrition, not direct violence." So, in Darfur, the vast majority of deaths were not from violence, while in Iraq, Burnham et al report the opposite.
BTW, I think your 400-450,000 figure is from Eric Reeves, who is an advocate for Darfur and a Professor of English. Nothing wrong with either of those things, of course, but I'll go with the figures reported by epidemiologists.
Donald says: It's to the great credit of Burnham and Roberts that they keep pointing that if 2.5 percent of the population has died it ought to be easy to confirm this level of violence.
Unfortunately, I think this has been another point to their discredit. This has been Roberts more than Burnham, though, as far as I can tell. Here's one case of him making this argument:
"this is the easiest one for the press to either debunk or verify, because if Iraqi body count is correct, if you go out to the graveyards and morgues across Iraq, only one in ten bodies coming in must be from violence, and 90% must be from natural causes. If we're right, the majority of bodies over the last three years that have come in, must have been from violence. And it would be pretty darn easy to go to four or five or six places across Iraq and get a quick pulse on whether or not it's a tenth of all deaths that are from violence, or most of all deaths that are from violence."
But this claim is absurd. As Fred Kaplan has correctly pointed out recently, if it were so easy to get "verification" like this, why didn't the Lancet team do it?
One reason I can surmise is that if they had claimed to have "verified" their study by visiting a handful of graveyards in arbitrary locations in August 2006 and extrapolating whatever death rates they found at that time and in those few arbitrary locations across the whole country, they'd have been laughed at.
First of all, why would you assume that any rate you found for violent deaths in graveyards *right now*, at the most violent period since the invasion could be assumed as the average rate since 2003?
And this is before the issue of the crudity of the method suggested. It's not just the tiny number of locations suggested that's ridiculous either. For Roberts, apparently the geographic locations of these "four or five or six" graveyards are irrelevant as well. Any old arbitrary locations will do (That is, unless the "verification" team happened to be in Kurdistan or some other low violence area, as opposed to say Baghdad, then, after his study had been "debunked" using the methods he suggested, we can presume the locations of these graveyards would suddenly become relevant, and we would discover that it's not so "easy" after all).
This is yet another in the long list of ways in which Roberts has been very misleading on these issues. Also interesting is that this claim is not unlike the problem with the recent 'estimate' given by the Iraqi Health Minister. He took the recent 100-per-day death rate from a recent month in 2006 and just extrapolated it across the whole period back to 2003. But the ministry rates were nothing like that in 2003 or 2004, for example. Doing this inflates the whole period in proportion to the current highest rate.
Donald recently described the Health Minister's estimate as "pure crap", yet when Roberts suggests it's "easy" to verify his results if people would only go use methods even more "crappy", then it's to his "great credit".
None of this makes much sense, until we start considering that some people around here may be using quite a few double standards when they are required for Lancet or its authors.
The bottom line is that, yes, there have been 100s of thousands of excess deaths in Iraq, and yes, the invasion has been a huge disaster.
But Burnham et al's estimates aren't actually good estimates. They're not as good as the estimates we have for Darfur and other humanitarian emergencies.
If you want to argue for collecting more data, I think you should start by acknowledging that in most humanitarian emergencies we have much more data. We typically have dozens of mortality surveys, not just 2.
I forgot to include the link to Guha-Sapir and Degomme's study of Darfur.
"The second part of their rebuttal on the matter of "main street bias" -- that people might have been killed away from home, so it doesn't matter if the houses surveyed were disproportionately close to violence"
So much for the need for randomization in choosing households, which they claim to have done with "rigorous guidelines" (see Kaplan on this). It's irrelevant anyway. People are often killed away from home so it's still random no matter how you picked the houses!
Then Roberts/Burnham employ the logical fallacy that you need to be an epidemiologist in order to offer a credible critique of cluster sampling methodology, as if this were some exclusive area of knowledge reserved for epidemiologists.
And Roberts et al have successfully danced around yet another criticism....
joshd, the point of the graveyard thing is to provide an independant confirmation (or disconfirmation) of the number. Of course it will be a very rough estimate but should give an idea whether is closer to 500,000 or 50,000.
Tim, that's exactly what it can't provide. There's variation in the death rates over time and space. This should be easy to discern. Lancet provides a graph of its trends along with IBC's and some other one. And it changes over the period, peaking now.
Iraq morgue figures over various stages of the war have ranged from the low 20's to over 100 per day. That's alone allows for a factor of five disparity depending solely on which point you chose to do the "independent confirmation". If you did it during late 2003 for example you'd get one rate, while if you did it now you'd get a completely different, and much higher, rate. And that's still assuming both samples were representative, which is not even being required of this "verification", or "debunking".
But even if you take away the issue of variation over time, what they suggest couldn't even provide "confirmation" of current death rates either, even in a ball park, unless the areas you sampled could be assumed representative of the nation currently. Picking "four or five or six" arbitrarily is not adequate.
It seems little more than a way to parry criticisms. Tell the critic that they could go "verify" the results "easily", and turn it around to why they aren't doing that. The reason for this is they know that Fred Kaplan is not going to go searching through graveyards in Iraq and it's a way to shut him up. But there's no reason for him to do that anyway, not for the reasons they say at least, because their claims about the "verification" are, to borrow a phrase, "pure crap".
It may not be as bad as assuming you can produce a nationwide estimate of deaths by asking soldiers about their feelings, but it's pretty close.
As Fred Kaplan has correctly pointed out recently, if it were so easy to get "verification" like this, why didn't the Lancet team do it?
Right, and if they do it and get confirmation of their results, Fred Kaplan will say: "Oh Les, what a blind fool I've been!" Like hell he will. He will claim they picked unrepresentative graveyards. But the results will be accepted by most people if some fairly reputable reporter goes there with a camera crew and comes back with pictures of numerous freshly-dug graves in several different towns and a reasonably convincing account of how the sites were selected.
Just how useful such an exercise would be depends on whether Iraqis follow customs such as putting the date of death on tombstones; or whether local caretakers actually keep records of burials, as Roberts believes they do. But with a bit of luck it should certainly be possible to tell whether funerals are four times as numerous as they were in 2002.
It's fascinating to see how the critics treat the very idea of doing a bit of fieldwork as something quite preposterous.
If Burnham and Roberts are to satisfy their critics, they had better give up hope of doing anything else for a long time, because those Lancet critics are one demanding bunch.
Fred Kaplan wants a narrower confidence interval, then when he gets it he wants a different randomization scheme.
David Kane wants the raw data, and he wants it without any unusual events.
Ragout wants to see the interview script, so he can be sure there's no household overlap or recall bias.
And Josh wants ... well, it's not clear what Josh wants, unless it's for the study authors to announce that they don't believe their own numbers.
Come to think of it, there is absolutely no way Burnham and Roberts will ever satisfy their critics, and they have far exceeded any obligation of professionalism, ethics or courtesy in going as far as they have. Burnham et al. is a pretty damn good study, the best anyone has done on a very important and difficult subject. The only real problem with it is that people are appalled by its results. As they should be.
JoshD: "The second part of their rebuttal on the matter of "main street bias" -- that people might have been killed away from home, so it doesn't matter if the houses surveyed were disproportionately close to violence.
So much for the need for randomization in choosing households, which they claim to have done with "rigorous guidelines" (see Kaplan on this). It's irrelevant anyway. People are often killed away from home so it's still random no matter how you picked the houses! "
Josh the point of randomising the selection of households has nothing to do with whether violence occurs there or elsehwere it's to try and ensure that the random sample is representative of the population as a whole."
People who live on main roads or in single as opposed to multiple dwellings, may differ from the rest of the population in a variety of ways. For starters, they're likely to be richer which in Iraq probably means they're more likely to have been members of the Baath Party and therefore potential targets for reprisals.
Equally, if the middle class live in more modern areas with western-style road grids while the poor live in areas with lots of back-alleys differences in access to medical services would be a factor.
I have a great deal of respect for the work of IBC but I think that, for reasons that are unclear to me, you've become emotionally committed to disproving the Lancet figures and are now making spurious arguments against it.
jre:
Lancet critics.
Best,
D
"David Kane wants the raw data, and he wants it without any unusual events."
No, I want the raw data and a detailed description of the methodology for collecting it, along with a copy of the computer code used to create the estimates. This is a standard request in the social sciences. Failure to provide it would have prevented publication in many journals, albeit not the Lancet.
"This is a standard request in the social sciences."
Funny, it seemed to present John Lott with quite a lot of difficulty.
Is joshd the designated Deltoid contact on all matters Burnham and Roberts or are there others for other places?
"for reasons that are unclear to me, you've become emotionally committed to disproving the Lancet figures and are now making spurious arguments against it."
As I've repeatedly observed, people defending the Lancet study (including its authors) seem far too interested in theorizing about the supposed motives of its critics, rather than in addressing the criticisms.
And when they do address the criticisms, as here, it's usually evasive or contradictory, if not downright deceptive. You can think whatever you want about my emotions, but I made no "spurious arguments". GB & LR here appear to be trying to say that the location of the houses isn't too relevant because people will often die away from their homes, from which follows my comment.
If this is not what they were trying to imply, go ahead and explain what they supposedly meant.
My emotions in this affair, since you're so interested, range from frustration to contempt. Frustration with their evasiveness and contradictory claims about their methodology (it looks to me like they don't really know exactly how the sampling was done and are just saying whatever - often contradictory - things are necessary to parry the criticisms about it), and contempt for their persistently deceptive and self-serving abuse and distortion of facts and source material in their report and in subsequent interviews and such, all to the effect of providing *spurious* support or "corroboration" for their study.
The emotion of surprise at this behavior has long since past, as I've observed it for so long now.
"Per capita, that's a level of violence on about the scale of the Bosnian War"
Funny, I've been using that as a point in favor of the Roberts estimate.
It's nonsensical to call that evasive, or contradictory to their claims of proper randomization. "Randomization isn't as important as you might think" and "We went ahead and properly randomized anyway" are mutually reinforcing arguments, not contradictory ones.
Josh, I said the 150,000 figure was pure crap--the same figure you also dismissed, though probably with some other phrase. I don't really care about the inconsistency--just pointing it out.
I'll repeat that it ought to be easy to check up on the order of magnitude of deaths in Iraq.
I've often seen references to Iraqi polls. I'd be curious to know how many households are visited when they do these polls. If there are typically about 6 people in a household and if 1/40 of the Iraqi population has died from post-invasion violence, you'd expect maybe 1 household in 10 to have suffered one or more such losses. (Possibly as many as 1 out of 7, if losses are scattered evenly throughout households). If IBC's mortality rate of about 1 in 500 is correct, you'd expect no more than 1 household in 80 to have suffered a death and probably less than that.
Someone else can decide on the minimum number of households you'd need to survey to see if the death toll is in the several hundred thousand range. I'd be surprised if the pollsters working in Iraq couldn't settle the question if they or the people who hire them wanted it settled.
I have low standards myself, or put another way, there's so little interest in determining the Iraq death toll I'd be happy to see someone trying anything. So let someone survey 100 households randomly scattered around Iraq, with a total of 600 people. If they find 10-20 violent post-invasion deaths, that hits Lancet 2 on the nose. If they find 0 or 1 or 2 or 3 , that's IBC's territory.
There's a chance of getting 10 or 20 deaths even if IBC is right, or 1 or 2 if Lancet2 is right, but like I said, my standards are low. One result or the other should make those who favored the losing side reconsider their position. Also, according to this nifty little thing called the Markov inequality, the chance of getting a number greater than or equal to k is less than or equal to (Expectation value/k). So if we take IBC's mortality and multiply by 1.5 to get a number they're willing to live with, you'd have a mortality of 1 in 300 or so and the expectation number would be 2 in my hypothetical survey. So there's guaranteed to be less than a 20 percent chance that there would be 10 violent deaths out of 600 people and if that number turned up, I'd expect IBC members to scratch their heads and wonder whether the toll might not be a lot higher than they realized. If only a couple of deaths showed up, I'm not sure what it would show quantitatively, except that IBC's position would be strengthened.
But anyway, I'm guessing the actual pollsters visit many more than 100 households.
Donald,
This exchange with Jon Pedersen (led author of the ILCS study that IBC has referd to as the "Gold Standard" of Iraqi mortality studies) will interest you.
[Link](http://www.medialens.org/forum/viewtopic.php?t=2059&sid=bd973461bd558ac…)
He echoed the Iraqi Health Minister's estimate of 150,000 dead. He also says the pre-inascion death rate was "higher than 5".
Hopefully he'll elaborate on his criticism of the study rather than limit himself to brief remarks to journalists. If he can back these statements up he should do so.
I also asked him what field work he recommends as a way to check the Lancet study's findings.
Hopefully he'll keep replying.
Contradictory statements:
published Lancet methodology:
"The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street."
reply to Kaplan:
"We set up rigorous guidelines and methods so that any street block within our chosen village had an equal chance of being selected."
How do they rectify these two contradictory statements? They have subsequently asserted that the methodology they published is wrong ("oversimplified") but that still doesn't explain how "any street block" could have "equal chance of being selected" if they used anything like their published methodology, even if the published version is "oversimplified". How can a methodology dependent on "main streets" or "cross streets" at all provide "equal chance" that the streets and blocks which aren't "main streets" or "cross streets" get selected?
That makes no sense. And who knows what they mean by a "main street" in the first place, as they don't explain how they made this distinction in Iraqi neighborhoods. They don't even seem to know actually. But they assert that "any street block within our chosen village had an equal chance of being selected."
Moreover, how is the published methodology "oversimplified" exactly? What was the actual methodology then? What percentage of the time did it depart from the published methodology? How often did it sample streets that were not "main streets" (however that's defined) or "cross streets" to them? If we knew this, maybe we could evaluate the reliability of their "equal chance" assertion.
Any claim that "randomization isn't too important" seems to contradict their determined efforts to claim (contradicting themselves again) that all streets had an "equal chance" of selection, or the repeated references to "random" this and that in their published paper. All the assumptions for all the huge extrapolations needed to get to their estimate hinge on "randomness".
Perhaps they should have published an explanation for why randomization isn't too important in their study, along with the methodology they actually used, that is, if they know what it was.
I love how the home scientists are treating this study as if it were interviews of a bunch of undergrads about cola preference. I wonder if "Freshman brand identification in the soft drink arena" studies might have different standards or reasons for data management than a study in a FRACKING WAR ZONE!@@@@@@
AAAARGHH#J$GLKJ$HGOI
BRAINSSSSSSSSSSSSS THEY ARE EATING MY BRAINSSSSSSSSSS
Well said joshd.
But what I want to know is: why are the Sunnis and Shiites slacking? They are nearly as hopeless as the England batsmen. Burnham and Roberts set a target for them of c.850 violent deaths a day in 2006, and in the last few days (and months) they have only managed a paltry 150 or so. Where is Paul McGeogh when we need him as scorer? Or are SBS deliberately under-reporting? Perhaps SBS in fact only reports the minor incidents and ignores the much bigger slaughters that would achieve the Burnham-Roberts target.
"why are the Sunnis and Shiites slacking?"
Tim C: if you've really reached the level of senescence and irrelevance where the only way you can attract the attention you so desperately crave is to engage in grotesque and revolting stunts can I suggest you just pop down to the nearest preschool and show the toddlers your wee-wee?
This just in from the New York Times:
Iraq is in a civil war.
(http://www.nytimes.com/2006/11/26/world/middleeast/26war.html)
A few selected quotes.
..."I think that at this time, and for some time now, the level of violence in Iraq meets the definition of civil war that any reasonable person would have," said James Fearon, a political scientist at Stanford...
... many scholars say the bloodshed here already puts Iraq in the top ranks of the civil wars of the last half-century. The carnage of recent days -- beginning with bombings on Thursday in a Shiite district of Baghdad that killed more than 200 people -- reinforces their assertion.
Mr. Fearon and a colleague at Stanford, David D. Laitin, say the deaths per year in Iraq, with at least 50,000 reportedly killed since March 2003, place this conflict on par with wars in Burundi and Bosnia.....
That is, even using the IBC underestimates, this is a very bloody civil war. I imagine that doing good epidemiological work in the midst of a diffuse civil war could cause some crimps in the research protocol that aren't readily appreciated in more fortunate areas.
Peer review and replication are two of the hallmarks of rational (as compared to pathological) science, but replication gets a bit more of the weight in my book. Someone needs to replicate the B & R study, using similar or other methods (the grave count sounds good), not this endless sniping by those without the background. Peer review is review by professional peers, after all, not just people who can type. I understand that George Mason U, in the US, has a desire to be seen in opposition to left-leaning academia. Perhaps some of their researchers (Wegman?) could design and carry out a improved study to provide more accurate estimates?
I really hope Pietersen spends the entire next day in the crease in response to Tim Curtin's sad commentary.
If you think I was mocking your tragically silly post in which you said that a single report of an air strike death before the war's official start was "evidence that something is wrong with the underlying data" you are right.
Thank you for making it clear that what you really want is
It is also accepted in the sciences generally that one does not accuse a researcher of fraud without a damn good reason for doing so. You, sir, had no such reason when you accused Burnham and Roberts of fraud -- which is why your post on the subject was justifiably pulled.
Mr. Kane, show some class by apologizing to Burnham and Roberts for your inexcusable post. Until you do so, at least spare us your relentless whining about why they won't give you their data.
at least spare us your relentless whining about why they won't give you their data.
Yes, the new mendacicizer technique.
I think, personally, the appropriate response is a more succinct STFU. Just a thought.
Best,
D
Joe, thanks very much for that link. Pederson and Soldz both made good points. I wish Pederson would get more involved in the debate.
One of the most interesting points Pederson made was on the violent/nonviolent claim, though I don't see how it could be possible for the mixup to have occurred accidentally since the paper goes into detail about the various causes of violent death. It'd be good to know what's on the death certificates.
The relatively low rate of nonviolent deaths was a surprise--pre-Lancet 2 I'd have guessed violent deaths in the 200,000 range and non-violent deaths comparable, based on the notion that Lancet 1's midrange estimate (which wasn't badly damaged by any IBC critique, IMO) could just be extrapolated up to the present, possibly with a bit of an increase since violence is increasing by all accounts.
jre,
You list 3 things that Burnham et al could do to satisfy their critics, and say there is "absolutely no way" they could ever do these things. But they've already done the hardest one! Lancet 1 was much-criticized for having too small a sample size. To their credit, Burnham et al addressed these criticisms by redoing their "power" calculations and increasing the sample size in Lancet 2.
Of course, they could also easily release the raw data and a questionnaire.
Anton,
You say that the peer reviewers must have signed off on the Lancet papers. But this isn't so: it is quite possible to publish a paper even when some of the peer reviewers think it is crap. The editor has the final say.
In my experience, it is pretty common for the editor to overrule some of the peer reviewers. I'd guess that it's even more common in medical journals, where the editor seems to play a very prominent role.
One point I forgot to make regarding the discussion with Pedersen that Joe E linked to--
Pedersen himself apparently agrees that his study undercounted violent deaths, though he thinks their figure is in the ballpark. So it's not the gold standard for Iraq mortality studies. His admission also means that there is no meaningful difference between the midrange violent death toll in Lancet 1 and the one in ILCS. Lancet 1 gave about 60,000 violent deaths (with a huge error bar) for the first 18 months and somewhere in the high 30s for the first 13 months. That's all violent deaths. ILCS gave 18-29,000 "war-related" deaths for that period and Pedersen agrees that it is low, though in the ballpark. (What "war-related" means is a big problem problem, but I'll ignore it.)
IBC made a big point about the alleged incompatibility between the ILCS estimate and the corresponding number one could use from Lancet 1, but that argument looks pretty flimsy now (and the difference wasn't that big to begin with). This also means that IBC can't use the ILCS number to justify its claim that its methodology probably picked up more than 50 percent of the deaths.
Of course, there's still the gap between ILCS with its now unknown error bar and Lancet 2, where the 95 percent CI for the violent death toll in the first 13 months is roughly 50,000 to 130,000 or so. Pedersen clearly doesn't believe the high end of that range, but I wonder if he would accept the low end as a possibility?