Daniel Davies was on the radio talking about the Lancet study.
Richard Miniter interviews Gilbert Burnham.
Deena Beasley reports what experts in cluster sampling think of the study:
"Over the last 25 years, this sort of methodology has been used more and more often, especially by relief agencies in times of emergency," said Dr. David Rush, a professor and epidemiologist at Tufts University in Boston. ...
Rush, speaking at a meeting in Los Angeles on the medical consequences of the Iraq war, said that the relatively small size of the sample -- 1,849 households -- doesn't change the findings, although it does widen the "confidence limits," hence the large range of the estimated additional deaths.
In addition, the biases inherent in cluster sampling, such as wording of questionnaires, would tend to undercount, rather than inflate, the number of deaths, Rush said.
"I think this is an extremely credible study," said Michael Intriligator, professor of economics at the University of California at Los Angeles.
Intriligator, who said he commonly uses cluster sampling in his own work, noted that the study's most remarkable finding was the death rates in the country have risen from 5.5 per thousand Iraqis per year before the invasion to 13.2 per thousand per year as of the study's July cutoff.
Hey, these guys are in apropos fields -- they're not even physicists -- what're they doing commenting on the study?
I thought the passage below from the Miniter interview was extremely revealing. First, Minter's criticism is exactly right and important. Second, once again we see a Lancet author giving an answer that contradicts what they wrote in their study (the study makes no mention of sleeping as part of the definition of a household).
Second, once again we see a Lancet author giving an answer that contradicts what they wrote in their study (the study makes no mention of sleeping as part of the definition of a household).
The study doesn't mention sleeping as not part of the definition either, so there's no contradiction. It says: "Deaths were recorded only if the decedent had lived in the household continuously for 3 months before the event." Above, Burnham explains the criteria for being classified as "living in a household"--eating and sleeping there.
Burnham's "There are many old people in Hungary" line was hilarious.
The Lancet study says, "For this study, a household was defined as a unit that ate together, and had a separate entrance from the street or a separate apartment entrance." There is no mention of sleeping.
Even if this were a trivial point, I still think this kind of sloppiness undermines the Lancet authors' credibility.
But I don't think it is a trivial point. As far as I can tell from the ILCS, a large proportion of Iraqis -- probably more than 1/3 -- do not eat where they sleep.
Even if this were a trivial point, I still think this kind of sloppiness undermines the Lancet authors' credibility.
It is a trivial point, almost absurdly so, and the attempt to build it up into something more substantial - particularly now that the author has confirmed that a very precise definition was actually used in the survey - is to be honest not doing wonders for your own credibility.
D^2,
When did you become such an expert on Iraqi living arrangements? Perhaps you can enlighten us as to what a *housh* is, or a *mushtamal*? Perhaps you can explain why 38% of Iraqis do not own a stove?
According to the ILCS, the *vast* majority of Iraqis live in houshes and mushtamals, rather than in "villas" or "apartments. I'm not totally sure what a housh is, but it often seems to mean a group of houses built around a central courtyard, typically occupied by a group of related families. A mushtamal would be called an "in-law" unit in the US. In both cases, I expect that it's common for people to regularly eat with their (related) neighbors. For example, an unmarried adult son will often live in a mushtamal next to his parent's house and may usually eat with his parents. Is he a household member or not?
As long as the same criteria has been applied to all the samples, it's actually irrelevant if they count all people who eat there or all people who sleep there.
It can even be argued, it's not even important if the same critiria is applied to all samples or not. It is possible fairly safe to presume that people eat somewhere close to where they sleep, and thus would be in the same area, and still be part of the region sampled.
Remember what was counted was number of deaths, and a large percentage was documented by death certificates.
The numbers were then used to figure out the total number - not the number per household, or any other household related statistics.
Kristjan,
If the vagueness of their definition of a household led them to, for example, double-count all deaths, both before and after the war, then there were 300,000 excess violent deaths instead of 600,000.
You are mistaken to say that the survey measures total deaths rather than deaths per household. Instead, the survey measures deaths per household, and calculating total deaths requires outside information (the size of the population).
And I'm having a hard time reconciling Burnham and Roberts' claim that their data review eliminated all double counting, with their statement that "Under human subjects regulations we could not keep unique identifiers, so we limited the information collected." Or with the fact that they didn't even collect the age of 10% of the deceased.
Why is it that critics always persume that errors work towards a too high number? It would be just as likely to not count someone as to count somone twice. Also, this would also lead to the pre-war fatality to be overcounted.
As a matter of fact, the survey measures number of deaths per people counted. This is then worked into a total.
The Lancet study says, "For this study, a household was defined as a unit that ate together, and had a separate entrance from the street or a separate apartment entrance." There is no mention of sleeping.
Again, there is no contradiction here. A household is defined in the manner above. A household member, for the purposes of counting the death, is defined in terms of eating and sleeping at that location for three months. Surely you can see that a household is not the same thing as a person living in that household, so they require separate criteria?
If the vagueness of their definition of a household led them to, for example, double-count all deaths, both before and after the war, then there were 300,000 excess violent deaths instead of 600,000.
Critics of the study have frequently objected that the pre-war crude death rate seems too low. Are you suggesting that it should actually be even lower by half?
How, furthermore, could they have double-counted all deaths? If, as you say, 1/3 of Iraqis don't eat where they sleep, then no deaths in that group would have been counted at all by the criteria Burnham gave; they wouldn't be counted as belonging to any household. Your objection implies an undercount of deaths, not an overcount.
Any comment on Burnham et al.'s methodology deserves to be taken seriously if and only if it evinces some desire to get closer to the truth.
Right now, I'm struggling to see how Ragout's objection can qualify -- but I'm willing to be convinced
Ragout -- under what scenario do you imagine a confusion over the definition of "household" could result in an overcounted death? Do you figure that cousin Omar's death was reported by one household, because he always ate breakfast there, and his other cousins, in another household, also reported him because he always ate dinner there? And both households had copies of the death certificate? And the survey team failed to notice the same name and date on the certificate? I fail to see how such a concatenation of unlikely events could possibly happen often enough to make any difference to the study.
If you had some other scenario in mind, please describe it, and explain why you think it is likely enough to make a difference.
And if it really just comes down to "Burnham said sleeping in the interview, and the paper didn't say anything about sleeping, and that seems fishy to me", aren't you just looking around for any old stick to beat Burnham with?
I'm not totally sure what a housh is
Oh, okay then. Come back when you've asked your friends at LGF. And look up 'nitpicker' while you're there.
In any case, the study's authors have said that there was cross-checking for double-counts.
If the vagueness of their definition of a household led them to, for example, double-count all deaths, both before and after the war, then there were 300,000 excess violent deaths instead of 600,000.
This would rather tend to require that the people checking the death certificates didn't notice that they were being shown the same certificate twice. You don't have to know what a housh is to work out that one.
D^2,
No, the double-counting problem can arise even if they don't literally count someone twice. It also happens whenever they attribute a death to an interviewed household that should have been attributed to some non-interviewed household. That's another reason why the Lancet authors' claim that they checked for double-counting is laughable.
And I don't know why you think they couldn't have been shown the same death certificate twice. The interviewers worked in teams of four, and didn't necessarily travel in one big pack.
jre & ahem,
When you're finished abusing me, you might want to go over to Andrew Gelman's blog and abuse him too. He says, "The definition of 'household' is another matter, since I could imagine porous definitions of whether someone lived in the household." Or perhaps you'd prefer to mock Prof. Apfelroth, who said pretty much the same thing in a letter published in the Lancet, criticizing the first study. Hell, Lambert has endorsed Gelman's expertise, so get started on him.
Kristjan,
I'm not "presuming" that this error "works towards a higher number." Double-counting can only lead to an overcount.
Anton,
I have no idea what you mean when you distinguish between the definition of a household and a household member, though I think the fault is Roberts'. They say a household is "a unit that ate together." That's a group of people. Then -- in the same sentence -- they say a household has "a separate entrance," so now household means a structure. You're doing the same thing (confusing a household with a housing unit) so it's very hard to understand what you're saying.
jre,
I've already answered your question, but let me say it again. They don't have to literally count "cousin Omar" twice by visiting the place where he eats breakfast and the place he eats dinner. If their definition of a household is vague enough that Omar will be counted whether they go to his breakfast household or his dinner household, the double-counting problem is the same. If Omar has two households, he's twice as likely to be counted as he should be.
Oh, and if I'm right about what a housh is, Omar could easily have 10 households.
Oh stop pussyfooting around and just call the surveyors liars and the whole thing a fraud, Ragout. What was the number of death certificates the study says they were shown? Come out and call them liars, the only implication of your line of "reasoning", or find some new game to play because this one's tired.
Sorry ragout, but double counting is more likely to lead to an undercount. If someone is counted as part of two households then that increases the denominator. Whereas they checked deaths to avoid double counting them, so it doesn't increase the numerator. Hence the death rate is biased downwards by double couunting.
It's weird how you managed to get this backwards isn't it?
No, the double-counting problem can arise even if they don't literally count someone twice. It also happens whenever they attribute a death to an interviewed household that should have been attributed to some non-interviewed household
Well no; this "double counting" would also tend to make people "overestimate" the size of their households (scare quotes because I don't think it is double counting and I don't think it would overestimate the size of the household) and this would disappear ("cancel out" in the literal sense) in the calculation of the death rate. For every dead "Omar" who you're claiming should have been in another household, you would add 20 living Omars, Saddams Mohammeds and Maliks.
I suppose it could in principle be the case that people who ate in one house but slept in another had a different risk of death compared to people who didn't, but surely to Allah, ragout, if the word "nitpicking" is to have any meaning at all ...
D^2 & Lambert,
Someone whose membership in the household is uncertain is a lot more likely to be reported if they have been killed. It's a mortality survey that asks only a few questions, mostly releated to death, remember?
As (and others) have been saying over and over for at least a year now: conscientious interviewers & respondents are likely to err on the side of reporting too many deaths when there is ambiguity as to whether a death should be reported.
Indeed this argument is getting tired, and I dispair of ever being able to explain it to you when I can't even convince you that sleeping and eating aren't the same thing.
I think Ragout's argument makes sense, but if I recall correctly the death certificate cross-check rate strongly limits the possible effect.
It seems fairly clear to me:
So it is clear that a structure can contain multiple households, if they have separate entrances within the structure.
Ragout, I'll accept your argument as having been offered in good faith. But it's mistaken. Here's why:
For the study's purposes, the effect of cousin Omar's belonging to more than one household is exactly what would result from duplicating him (along with his history and fate) in those extra places. Counting him several times would (as noted above) be the same as expanding the population by those extra Omars. The Omar effect would be nil unless all the Omars are systematically different from the rest of the population.[1] It is true, as you say, that the probability of counting Omar's death is higher, whether or not one of his doppelgangers was actually added to the rolls. But, as also noted above, death-certificate checking can only reduce that number. So, in order to have an effect on the study's results, the effect you postulate has to involve:
1) Significant numbers of Iraqis belonging to more than one household each,
2) who are more likely to die than the average, and
3) who are enough more likely to die that they tip the scales even after death-certificate checking.
In the absence of any evidence that households really did overlap, or a plausible mechanism by which that overlap might make a difference, this whole line of argument seems, to put it bluntly, hand-waving. But I'll go the extra mile with you, Ragout. If there is anything wrong with the argument I've laid out above, what is it? And let's get quantitative here: The study reported 547 post-invasion deaths. How many of those do you think could have been the result of household overlap? To get the 13.3 rate down to 5.5, our hypothetical denialist[2] has got to assume that 321 of those reported deaths were the result of poor Omar and his overcounted brethren, each of whom eats supper at a different table every night, belongs to every family in Baghdad, and has a life expectancy measured in minutes. It ain't happening in this universe.
[1] At this point, you can almost hear the wheels start turning. A caution, though: post-hoc theorizing about why this population is different will be stopped at the door as bringing down the quality of the establishment.
[2] To be fair, Ragout did not go so far as to say that the overcounting effect would wipe out all the excess deaths. But let's keep our eyes on the doughnut: if it does not make any difference to the study's conclusions, why even argue about it?
jre,
Multiple-household members like Omar don't have to be more likely to die. They just have to be much more likely to be reported when they die, compared to multiple-household members that didn't die.
My guess would be that the majority of Iraqis live very near to their extended families, and often eat with them. Many live near several households where they might be considered household members if a vague question about death were posed. So, I'm going to guess that, on average, each Iraqi might be claimed by 2 neighboring households, were he or she to die. But only a small percentage of the time (say, < 10%) would he or she be claimed as a member of their neighbor's household, unless that neighbor were pressed. So, we have 2 times too many deaths being reported.
Let's role-play!
Imagine the Lancet guys come to interview an elderly couple who live in their own house next. Their three adult sons used to live with their own families, in three houses next door. All four houses surround the extended family's courtyard. Most mornings, the four families breakfast at the picnic table in the courtyard. Unfortunately, one son was killed last year.
Lancet interviewer: "We're doing a survey to count deaths. For the good of our country we must estimate the number that have been killed in these terrible times."
Elderly man: "How can I help?"
Lancet: "Who lives in your household?"
Man: My wife and I.
Lancet: "Did any household members die from January 2002 until the war began?
Man: "No"
Lancet: "Did any household member die since the war began?"
Man: "Yes, my beloved son was killed by a thief 6 months ago."
Lancet: "Had he lived in your household for more than 3 months?"
Man: "Of course, my son has lived here all his life!"
Lancet: "Thank you for your time."
And the Lancent interviewer moves on, having recorded 3 people and 1 death, but never having counted his son's widow, nor the other two sons and their family.
trrll,
I think I understand what the Lancet authors were trying to say, when they wrote:
But I think they've confused Anton with their sloppy writing. I think Anton is claiming that this sentence only refers to a structure, when it clearly refers to both a structure and a group of people.
Ragout -
That's not how the survey was conducted. To quote the study:
Throughout the survey description, it is evident that the team took great pains to avoid just the kind of household miscount as takes place in your fictional example. The teams consisted of medical doctors with previous experience in doing surveys. Each team spent two days in training specific to this survey before starting. Experts in epidemiology have reviewed the study design and deemed it sound. Against that, we have a number of people who remain unconvinced because they can imagine circumstances that might result in an overcount of deaths.
OK, that's your privilege -- but you need to at least try to imagine circumstances that have different effects if you are to have a credible claim to being interested in the truth.
Let me supply the rest of your interview:
Johns Hopkins interviewer: And did you and your wife have any other children, sir?
Man: Yes; my other two sons were in the Republican Guard.
Johns Hopkins interviewer: And where are they now?
Man: I do not know. They had lived in the Army barracks outside Baghdad since 2001. They called me after the Army was disbanded, but I have not heard from them since.
You see, household overlap is not the only sampling anomaly that can be imagined. Household gaps can also exist. In other words, Iraqis who are essentially homeless for purposes of the survey will not, by definition, be counted. If I wanted to offer a post-hoc criticism of the survey, I could theorize that hundreds of thousands of Iraqis could not be counted by the survey's methods, and that since the homeless are more likely to die than the homeful, that this would result in an undercount of deaths.
Do I think that actually happened? No; I just wanted to demonstrate how easy it is to spin a "Just-So" story without any actual evidence to back it up. On its face, the study seems to be very well designed and conducted in such a way as to minimize miscounts. If you have evidence that the scenario you set forth actually happened in any significant number of cases, what is it? How many of the 547 reported deaths would you guess resulted from such miscounting? And if it is extremely unlikely that such miscounts had any effect on the study's results, why are we discussing it?
Huh? No, I'm not claiming that. For one thing, I don't think structures eat. I'm merely pointing out that a household is an entirely different kind of thing--partly a spatial location, partly a structure, partly a group of people--than a household member, who is a single person. Therefore it's perfectly consistent to define a household by various criteria including "people eat here together," but to assign members to that household via the stricter criterion "This person eats and sleeps here." It simply means that some people would be counted as belonging to no household, if they didn't sleep where they ate. And if only the dead folks are checked that way, that implies an undercount of deaths.
Lancet: "So he slept and ate right here for 3 months?"
Man: "Well, he slept across the courtyard there, with his wife and family, but we usually had breakfast."
Or--
Lancet: "So no one but you and your son regularly ate and slept here during this period?"
Man: "Oh no, I have two other sons who do the same--they live in those houses there--and all have families."
Or--
Lancet: "What births and deaths have occurred in your family during this period?"
Man: "Well, my poor dead son had one child..."
Lancet: "Does the child live here?"
Man: "Yes, along with his mother."
Really, unless there's an outright conspiracy of silence among respondents, it's hard to see how they're somehow only going to hear about the dead not-quite-members of the household, without realizing those not-quite-members shouldn't be counted.