The editors at Slate really don't like epidemiology. Not content with Christopher Hitchens' clueless attack on the Lancet study they've published another attack on the study. And this one is by Fred Kaplan, the man who made such a dreadful hash of it when he tried to criticize the first Lancet study. Kaplan writes:
The [first] study's sample was too small, the data-gathering too slipshod, the range of uncertainty so wide as to render the estimate useless.
So he's learned nothing about statistics since his botched criticism of the first study. Kaplan concedes that the new study has a smaller confidence interval, but you just know he's going to find some reason to dismiss it, and sure enough:
But the study has two major flaws -- the upshot of which is that it's impossible to infer anything meaningful from it, except that a lot of Iraqis have died and the number is getting higher.
So what's the first flaw that Kaplan claims to have found?
Based on the household surveys, the report estimates that, just before the war, Iraq's mortality rate was 5.5 per 1,000. (That is, for every 1,000 people, 5.5 die each year.) The results also show that, in the three and a half years since the war began, this rate has shot up to 13.3 per 1,000. So, the "excess deaths" amount to 7.8 (13.3 minus 5.5) per 1,000. They extrapolate from this figure to reach their estimate of 655,000 deaths.
However, according to data from the United Nations, based on surveys taken at the time, Iraq's preinvasion mortality rate was 10 per 1,000. The difference between 13.3 and 10.0 is only 3.3, less than half of 7.8. ...
(If the Hopkins researchers want to claim that their estimate is more reliable than the United Nations', they will have to prove the point. It is also noteworthy that, if Iraq's preinvasion mortality rate really was 5.5 per 1,000, it was lower than that of almost every country in the Middle East, and many countries in Western Europe.)
Kaplan gives a broken link for the UN data, but you can see it for yourself if you go here and select Iraq from the list. The UN's table does indeed gives a mortality rate of 10 per 1,000 for Iraq for 1995-2000. We see straight away the mistake that Burnham et al made. They should have presented their estimate of Iraqi deaths in a big table and not provided any details of the source other than "surveys". Presumably, Kaplan would then have accepted the number uncritically.
What are the "surveys taken at the time" that Kaplan reckons contradict the Lancet study? The 2004 Lancet study provides the answer:
No surveys or census based estimates of crude mortality have been
undertaken in Iraq in more than a decade, and the last estimate of
under-five mortality was from a UNICEF sponsored demographic survey
from 1999.
That's right, there weren't any. The UN number is just a guess. The Lancet number is more reliable than the UN number because it is based on a survey rather than being just a guess. Kaplan even admitted this in his critique of the first Lancet study.
According to quite comprehensive data collected by the United Nations, Iraq's mortality rate from 1980-85 was 8.1 per 1,000. From 1985-90, the years leading up to the 1991 Gulf War, the rate declined to 6.8 per 1,000. After '91, the numbers are murkier, but clearly they went up.
Did he forget writing this or something?
Kaplan also claimed that a death rate of 5.5 "was lower than that of almost every country in the Middle East". I went to the UN population page and looked up the death rates for every country neighbouring Iraq for 1995-2000. Here are the numbers: Iran 5.5 Jordan 4.6 Kuwait 1.8 Saudi Arabia 4.1 Syria 3.9 Turkey 6.6. All but one are less than or equal to 5.5. Remember, this is the same source that Kaplan used for the death rate for Iraq. Kaplan's claim seems to have been made with a reckless disregard for the truth.
The second flaw Kaplan claims is the so-called "main-street bias" which I dealt with here.
He does have a comment from Burnham protesting the Science reporters misrepresentation:
I did not ever tell the writer from Science that the raw data have been destroyed. Absolutely NOT! It is sitting right here! What I did say is that our Iraqi colleagues are very concerned about security, not just theirs but the neighborhoods they surveyed. They have asked us for the moment not to release the data to others as there might be some identifiers there. I am sure that we can remove any unique identifiers, but I am bound to honor their requests, as they have staked so much in collecting the data. We will be discussing this over time with our Iraqi colleagues, and I would imagine that in due course we can make it available to those interested. ...
Under human subjects regulations we could not keep unique identifiers, so we limited the information collected -- such as street and house numbers. The team did not write down information on the forms on the specific decision making process for each location.
From this, Kaplan promptly contrives another rationale for him to reject the study:
It sounds as if he's saying he didn't destroy the data because they never existed in the first place. If that's the case, how does Burnham know whether his instructions on methodology were followed at all? How can anyone verify the findings? And this is a peer-reviewed article. Who were these peers? And what did they review?
Why does Slate hate epidemiology? Why?
Kaplan concedes that the new study has a smaller confidence interval....
It doesn't, unless you include Fallujah in the 2004 CI, which he didn't when he was waffling about a dartboard. The dartboard is now even wider, but apparently that doesn't bother Kaplan because it's farther from zero. He does seem to be getting worse.
But the study has two major flaws -- the upshot of which is that it's impossible to infer anything meaningful from it, except that a lot of Iraqis have died and the number is getting higher.
In other words, you can't infer anything from the survey except its conclusion?
I'm not competent to judge statistical claims. I read at Good Math/Bad Math that the methodology used is standard, that it offers the best estimate available, and that the estimates tend to be larger than the actual events. I'm curious as to whether any of you have any sense of how much larger the estimates tend to be. Also, how surprised would you be if the excess deaths were below the lower boundary given?
the more I read the study and the more I talked with statisticians
Perhaps Kaplan would like to tell readers of this blog the names of the statisticians he consulted. Or perhaps readers of this blog would like to ask Kaplan about it.
It's a relief hearing the data wasn't destroyed. That didn't sound good, though I couldn't understand why the Lancet authors couldn't simply ask the Iraqi surveyors where they went. There were only 47 clusters scattered around various towns.
I had the same questions SomecallmeTim had when I read that other blog. He (forgot the name) was defending the validity of the Lancet, but said that cluster surveys have a history of overestimating the death toll, but also turn out to be closer than any other estimate one could get from passive counting.
"...cluster surveys have a history of overestimating the death toll..."
I gotta ask, what is the gold standard against which cluster sampling is compared? It seems to me any body count methodology will likely be a lower bound of the actual death toll, and may significantly undercount the dead.
"Also, how surprised would you be if the excess deaths were below the lower boundary given?"
I would be quite suprised, as this would be below 2 standard deviations, which only has a one in 40 chance of occurring (roughly the same chance as rolling a double 6 on two dice, which is 1 in 36). But it's certainly not impossible. I wouldn't bet my life savings on that chance, but I'm a conservative gambler.
The problem with narrowing error bounds, if I remember my stats correctly, is that error only decreases as the square root of the number of samples. So if the study had wanted to be twice as accurate (ie halve the sd from ~120,000 to 60,000 - I might be wrong about those numbers, I haven't re-looked them up) they would have had to interview 4 times as many households, in 188 clusters.
off topic, on blog: Anyone care to tackle this catalogue of lies on global warming?
If one assumes, as Kaplan does, that the pre-war death rate given by the UN (10 per 1000) is correct, and if one further assumes that the Lancet's post invasion death rate of 13.3 per 1000 per year is correct, one gets 275,000 excess deaths in Iraq since the war began (as opposed to the Lancet's estimate of 650,000).
But, acording to Kaplan:
"it's impossible to infer anything meaningful from it, except that a lot of Iraqis have died and the number is getting higher."
Hundreds of thousuands dead and that's not meaningful?
Perhaps not to people like Kaplan.
I can't figure out whether such people are just stupid or just callous (or perhaps both).
For all the smart people they have at Microsoft, one would think that their standards for contributors to Slate would be just a wee bit higher.
One would think -- and if one did, one would be doing more than many of the contributors to Slate.
Let's assume for kicks that the UN pre-war death rate of 10 per 1000 is correct.
Let's also assume (for more kicks) that the number of post invasion deaths given by Iraq Body Count is correct: 50,000.
Now, let's consider what these two together mean (other than a lot of kicks).
Taken together, these two assumptions fix the post war death rate at 10.6 per 1000 per year, which means, of course, that the death rate after the invasion is only 0.6 per 1000 higher than the pre-invasion rate given by the UN (10 per 1000).
In other words, if we assume that the UN pre-invasion number is correct and that the IBC post-invasion death total is correct, we come to the conclusion that the death rate has only gone up 6% since the war began.
I see.
All that violence that we have seen in Iraq over the past 3 years must have really been there all along. (As the Talking Heads once put it, everything is "Same as it ever was".)
Yes, I know. I know. "Saddam was a bad man" -- the Neoneocon** (or is it Neoconcon?) answer to every criticism.)
A Neoneocon is someone who was once a Neocon and broke off when the appellation acquired a less than stellar reputation (you know, started being equated with "Liars and nitwits").
I couldn't understand why the Lancet authors couldn't simply ask the Iraqi surveyors where they went. There were only 47 clusters scattered around various towns
I could. What about an easy to identify cluster that, say, it's so small that it would be easy to ask around who was interviewed? What if the iraqi army pays a visit to "find" (maybe under pressure?) that some of the responders "lied" to the Lancet interviewers? It might sound paranoid, but I doubt thoughts like that scape the minds of iraqis that live with violence as a daily routine.
I lived in totalitarian Argentina in the 70s. As a boy, I was instructed by my parents never to share with other kids that at home we liked Che Guevara. I can easily understan paranoia.
The reported death rate for Sweden is 10.3. The reported death rate for Hungary is 13.1. While I don't believe a lot of the lowest death rates at the link below, I would assume that the European ones are at least fairly plausible.
https://www.cia.gov/cia/publications/factbook/rankorder/2066rank.html
I'll leave it to you to decide on whether that makes any of the Lancet's estimates credible.
RD, the European ones aren't in the slightest bit relevant, since the age structure is different.I debunked this almost two years ago.
That had occurred to me (lest you wonder -- I have never voted for GWB, and am really just looking to understand the widely disparate estimates). I still find it hard to fathom that Iraqis were going to half as many funerals as Swedes were, assuming they know the same number of people. That doesn't sound like the Iraq I've heard about, particularly under sanctions (which supposedly were killing children at a high rate) and a repressive regime.
Since you've clearly thought about it a lot, how about the estimates of the number of people who died in American air strikes (13% of 55% -- call it 7%) or car bombings (27% of 55%, or roughly 14%)? These are both seemingly fairly noticeable and verifiable events, and until recently when the violence really spiraled, car bombings that only injured people were being reported on CNN, and not just the ones in Baghdad. 14% implies that roughly 84,000 people have died in car bombings and other bombings, right? Does any evidence back that up at all? And in an era where US air strikes are often documented (both when on and off target), do we really believe that around 42,000 have died as a result of air strikes?
Just trying to understand it all -- not trying to be a troll.
That's a good point, Palo, but I was thinking of the Iraqi survey team maybe passing the info to the Lancet authors and no one else. If even that can't be done out of legitimate fears of people being killed, then it's hard to see how one could examine this alleged "main street bias" effect to see if it could have been a problem. Maybe they could just say how far it was from the main highways, or which neighborhoods were in back alleys or--well, I'm not sure.
Remember that most of these reports of cause of death are probably from people who did not observe the actual event. Even if they did, the cause of an explosion (car bomb? air strike? unexploded ordnance? mortar attack? IED? suicide bomber?) is not necessarily going to be evident to an observer. So while the death reports, mostly backed up by death certificates should be fairly reliable, the cause of death reports probably tell us more about who Iraqis are inclined to blame than about the actual causes of death.
From "Iraq Death Rate Estimates Defended by Researchers
by Deena Beasley
"A controversial estimate by public health experts that hundreds of thousands of Iraqis have died because of the March 2003 U.S.-led invasion of Iraq is likely an accurate assessment, researchers said on Saturday.
"Over the last 25 years, this sort of methodology has been used more and more often, especially by relief agencies in times of emergency," said Dr. David Rush, a professor and epidemiologist at Tufts University in Boston."
"Rush, speaking at a meeting in Los Angeles on the medical consequences of the Iraq war, said that the relatively small size of the sample -- 1,849 households -- doesn't change the findings, although it does widen the "confidence limits," hence the large range of the estimated additional deaths.
"In addition, the biases inherent in cluster sampling, such as wording of questionnaires, would tend to undercount, rather than inflate, the number of deaths, Rush said."
http://www.commondreams.org/headlines06/1022-03.htm
Hey, but what does Dr. Rush know about such stuff. He's just a professor and epidemiologist at Tufts University.
Compared to columnists writing for Slate Magazine (Hitchens, Kaplan), physicist's-turned-cluster-sampling experts (Neil Johnson), economists (Michael Spagat) and psychic dogs, Rush is clearly no match.
He and others like him should just stand aside and let the real experts do what they do best: engage in handwaving and pure speculation -- unadulterated by reality.
And all those people fleeing Iraq? They are undoubtedly doing it for fun.
"Out of the population of 26 million, 1.6 million Iraqis have fled the country and a further 1.5 million are displaced within Iraq, according to the UN High Commissioner for Refugees. In Jordan alone there are 500,000 Iraqi refugees and a further 450,000 in Syria. In Syria alone they are arriving at the rate of 40,000 a month."
http://news.independent.co.uk/world/middle_east/article1919327.ece
I still find it hard to fathom that Iraqis were going to half as many funerals as Swedes were, assuming they know the same number of people
It ain't rocket science. Young populations die less. If an entire population was 12 years old then clearly there wouldn't be much dying going on for 40 or 50 years.
The mortality rate of the West Bank is 3.8 not because of their excellent health care but because half the population is below the age of 18.
"The UN's table does indeed gives a mortality rate of 10 per 1,000 for Iraq for 1995-2000."
So the AVERAGE death rate for the period was estimated at 10 per thousand.
There were some pretty critical events during that period - most pertinent was the relaxation of the sanctions regime in 1998 which led to more food and medical supplies reaching Iraq.
So if the AVERAGE death rate was 10 per thousand, the death rate in 2000 - and 2003 - would be lower.
"...cluster surveys have a history of overestimating the death toll..."
Where is the proof for this blanket statement? How would one confirm this anyway? Presumably one must have an independent method and how does one know that that method is more accurate than cluster sampling?
I am dubious of such general statements, especially when they come with no reference.
The above statement conflicts with what someone who knows about such things -- Dr. David Rush, a professor and epidemiologist at Tufts University in Boston -- has said about cluster sampling in refernce to the Lancet study:
"the biases inherent in cluster sampling, such as wording of questionnaires, would tend to undercount, rather than inflate, the number of deaths, Rush said."
http://www.commondreams.org/headlines06/1022-03.htm
I am heartened to see people discussing the reason for the low death rates in Iraq (before the war, that is) and neighboring countries, namely the age structure.
This, in turn, is due to high birth rates. You will find such high birth rates in all the Muslim countries, from Morocco, across north Africa and all of the Middle East to Indonesia. (Our census bureau has terrific international data by country at:
www.census.gov/ipc/www/idbsum.html ). You can look it up: Each Muslim country has roughly quintupled since 1950.
But no one talks about the consequences of this continuous rapid growth. Children always outnumber adults. Young people outnumber adults, and they are largely unemployed. The strain on education, environment, economy, and society grows larger with each generation. Coping gets tougher for everyone. No wonder so many youth rebel and join jihad.
Meanwhile, countries with stable populations, like China and Ireland, thrive. Population growth rates don't determine destiny, but quintupling your population in 50 years? That's clearly catastrophic.
I might be conservative in my definitions, but doubling your population, as China has done in that time period, is not what I'd call having a stable population.
Also, three of the countries in the Middle East that have had the biggest increase in their population, are Kuwait, the United Arab Emirates, and Saudi Arabia. Yet, these are also three of the most stable countries in the region, and all have to import foreign workers. And what's more, Israel's population has also quindoubled in the same time frame.
What I am trying to indicate is that the increase of the population is clearly not the only factor at play. I would assume that economic and social factors have to be taken into consideration as well. Wars, have a big influence on such things, as have the political structure in teh country.