John Allen Paulos writes about Iraqi war deaths:
Another figure in the news recently has been the number of Iraqis killed in the war. President Bush mentioned last month that in addition to the more than 2,100 American soldiers killed so far in Iraq, that there were approximately 30,000 Iraqis killed. He was likely referring to the approximate figure put out by Iraq Body Count, a group of primarily British researchers who use online Western media reports to compile an extensive list of Iraqi civilians killed. The organization checks the names and associated details of those killed. It necessarily misses all those whose names don't make it into the reports, and it makes no attempt to estimate the number it misses. The group's list contains almost 30,000 names at present.
A study that appeared in the prestigious British medical journal, the Lancet, in October 2004, used statistical sampling techniques to estimate all Iraqis killed because of the war and its myriad direct and indirect effects. The figure researchers came up with at that time — 15 months ago — was approximately 100,000 dead, albeit with a large margin of error. The Lancet study used the same techniques that Les F. Roberts, a researcher at Johns Hopkins and lead author of the study, used to investigate mortality caused by the wars in Bosnia, the Congo and Rwanda. Although Roberts' work in those locations was unquestioned and widely cited by many, including British Prime Minister Tony Blair and former Secretary of State Colin Powell, the Lancet estimates on Iraq were unfortunately dismissed or ignored in 2004.
These last 15 months have considerably raised the American death toll, the IBC numbers, and any update that may be in the works for the Lancet's staggering 100,000 figure. In fact, if the Lancet estimates rose at a rate proportional to the IBC's numbers since October 2004 — from about 17,000 then to about 30,000 — the updated figure would be approximately 175,000 Iraqis dead since the war began.
Ack, not Paulos too.
The Lancet study used the same techniques that Les F. Roberts, a researcher at Johns Hopkins and lead author of the study, used to investigate mortality caused by the wars in Bosnia, the Congo and Rwanda.
Really? I surely hope Paulos hasn't read the techniques Roberts used in those other places. If he has, then this is a large whopper, with extra cheese. This is something I have already pointed out and debunked before.
Here is the mortality studies done by Roberts in the DRC:
http://intranet.theirc.org/docs/mortalityIreport.pdf
http://intranet.theirc.org/docs/mortII_report.pdf
Can anyone seriously claim that these techniques are the "same" as the one done in Iraq?
The obfuscation continues.
The techniques were the same only in the sense that all the studies used some form of random sampling. But the Congo, Bosnia, and Rwanda studies used different sampling techniques, had different sample sizes, and were reported in more detail. From what I've seen, these other studies were generally better in all these respects: larger sample sizes, closer to the ideal of sampling proportional to population size, etc. The IRC has done over a dozen mortality surveys in the Congo, for example.
As an aside, I don't think that any of them surveyed the whole country, so I don't see why Seixon likes them.
And it's a sure bet that none of these earlier studies focussed on results that excluded the 3% of the data with the most violent deaths. In fairness to Roberts, I should say that he doesn't do this: he emphasizes results that use all of the data. It's only Lambert and the popular press that push the ad hoc estimate of 100,000 deaths.
I forgot another way in which the earlier studies were better: they had shorter recall periods, typically one year, rather than almost 4 years in the Roberts Iraq study.
Oh, I forgot to mention one minor, insignificant, completely irrelevant difference: the other studies did not reflect badly on the American government and the American public.
The recall period for the Lancet study was 32 months, or "almost four years" by Ragout's reckoning. Whether it's 32 months or 12 months doesn't seem to be that big a deal -- surely people are likely to remember for a couple of years that a family member died?
Oops, looks like you're right that the recall period was only 32 months, but it's still very long for a study of this type.
The death rate is basically deaths/(household members). The concern that has been raised by Prof Apfelroth and others is that the surveyed Iraqis will misreport the number of household members, and report the deaths of non-household members. This misreporting is likely to be greater the longer is the recall period.
Suppose I lived with my cousin's family for six months last year, but now live with my wife. Do my cousin's family count as my household members or not? What if my cousin was killed? We have no way of knowing how this would be reported to the survey, or even what the researchers hoped to have reported.
Did the study adjust for houisehold size effects?
You'd expect the size of post-war families to be higher as people left more violent areas to take refuge with family members or had their houses destroyed in the fighting or the looting.
Ian Gould wondered:
Yup. They counted up person-months of exposure.
Not necessarily. It depends on how much displacement occurred, and whether displacement resulted in dispersal or aggregation. The mean number of persons per household over the 14.6 month pre-invasion period was 7.7. Over a 17.8 month post-invasion period it was about 3% higher at 7.9 pph, i.e., essentially unchanged.
Ragout claimed:
Hmmm. I don't know what you've been reading, but it's common for retrospective family event histories to extend back over 30 to 35 years. In general you do exact month calculations when the partial periods are largish fractions of the total period, so less than maybe three or four years; much more than that, you switch to exact year calculations, but that has nothing much to do with recall bias. In this biz, the bigger worry is that the longer the retrospective period (like, 30 years) the more likely it is that recall bias will tend to understate the event rate, not overstate it.
The death certificate verification was intended to ensure that deaths of non-household members weren't included.
Did you live in their household, or they in yours? If you lived in their household, you'd add six person-months to their household total. If they lived in yours, they'd add (6 months)*(number of persons) to your household total.
Not necessarily. It depends on how much displacement occurred, and whether displacement resulted in dispersal or aggregation.
According to the ILCS, 5% of Iraq's population were forced to change residence due to war.
Did the study adjust for houisehold size effects?
In the way Ragout describes, yes, in another way, no. They assumed a general household size for all of Iraq (7 I believe) and used that to figure out how many households were to be in each governate. Of course, this would actually vary by governate, but I'm not sure what kind of effects this had at all. Doesn't really matter since I don't think any of the surveys account for this type of thing, although the ILCS does give a picture of the differences.
The ILCS tells us that the average household size for Iraq is 6.4. The lowest is in Basra, with 5.4, the highest is Muthanna with 8.2. From a brief look, there seems to be at least a slight correlation between household size and whether or not the place is urban.
That was all just some additional information for the hell of it, since I think everyone has already come to grips with that the Iraq survey by Roberts does not use the "same techniques" as previous ones. Well, unless by that Paulos meant sampling... lol.
Then you think wrongly. Quelle surprise, eh?
Then you think wrongly. Quelle surprise, eh?
Oh sorry, I just haven't seen anyone actually provide a rebuttal to Ragout's and my assessment that there is little in common between the surveys except for that fact that they are, you know, surveys.
I started out with: "Can anyone seriously claim that these techniques are the "same" as the one done in Iraq?"
Ragout added: "The techniques were the same only in the sense that all the studies used some form of random sampling."
So we're both wrong Robert? When are you going to get around to telling us why? I haven't seen a single argument provided as far showing us to be in error. Not even from Lambert.
The obfuscation continues. Quelle surprise.
"The techniques were the same only in the sense that all the studies used some form of random sampling."
Well, for one thing they both don't rely on body counts.
Ragout sees the elephant. I see the elephant.
Anyone else want to join us on our safari? Or are we trying to find a big enough cupboard to hide the elephant<.phpa> in?
The other studies are similar in all important ways. Certainly the differences do not justify the differing treatments. Read Greg has, I think, stated the real reason for the different treatment.
Robert,
32 months is a very long recall period for a mortality study of a crisis situation, as in the Lancet Iraq study. True, if one were extremely careful with the questionnaire design (as in "event history" studies), one could ask about a longer time period (though you need to follow persons, not families). But the Lancet Iraq study does not seem to have been at all careful to specify just who is included in the household.
I notice that you weren't able to answer my question, which pretty much demonstrates that the Lancet household definition is ambiguous. You ask "Did you live in their household, or they in yours?" But this is exactly the point at issue! The Lancet researchers didn't give their respondents clear directions about how to answer!
Maybe my example wasn't clear enough? Suppose that when interviewed, I live with my wife and no one else. Suppose further that last year, I lived for six months in my cousin's house, along with his wife and children. Do my cousin, his wife, and his children, count as my household members or not?
What if my cousin died? Do I report his death to the Lancet researchers? Does it matter if he died while I was living with him or not?
Personally, I think the IBC figures have gotten more attention than the Lancet figures because people are interested in violent deaths directly attributable to the war. And the Lancet's 100,000 deaths are mostly not violent deaths.
Also, I looked at the Washington Post's first article about the Lancet study. It says that the small sample size is a reason to doubt the study. And indeed, the Iraq study had a smaller sample size than the earlier genocide studies.
Further, progress marches on, and it is becoming more and more clear that larger sample sizes are needed for this kind of mortality studies. In another thread I noted that MSF's recent Darfur study discussed the need for larger sample sizes when studying violent death (as opposed to pandemics). The IRC's latest Congo study makes exactly the same point, and uses a sample size of 19,500 -- 20 times the Iraq sample size.
Check out the paragraph at the bottom of page 2 and top of page 3 of the report. Your situation is covered, which pretty much demonstrates that the Lancet household definition is (in your particular situation) unambiguous.
Hmmm. So, are you saying the difference between 32 months and, say, 24 months is going to have a big effect on accuracy? Cuz we know 32 months isn't at all long in non-crisis situations and those extra 8 months you might be worried about? They're going to be in the pre-invasion period.
Complaining about sample size is a cheap-ass complaint. Sample sizes can always be bigger. Sample size (mostly) affects precision rather than the location of the estimate.
Robert,
You claim that my example is covered and the household definition is unambiguous. So it should be trivial to answer my question. In my example, Do my cousin, his wife, and his children, count as my household members or not?
As to the sample size: the full-sample estimate is so imprecise as to be worthless. So many emphasize instead the 100,000 figure, which excludes Falluja. But people want to know about violent deaths -- civilians killed by soldiers -- and most of the 100,000 deaths are not from violence.
The other studies are similar in all important ways. Certainly the differences do not justify the differing treatments.
The Iraq survey used cluster clumping, aka the "grouping process", which none of the earlier surveys used. Not only that, the cluster sizes were smaller, and there were more clusters. Some of the Congo surveys used spatial sampling, which was not done in the Iraq study. With the sheer amount of studies done in for example Congo, there was more weight behind various assumptions that they made, which was not the case in Iraq.
In addition to that, the language in the Congo surveys is much more cautious and candid than the Iraq study. You will never find a statement such as this one in the Iraq study, but it appears in the latest Congo study: "The areas visited may not have been typical of eastern DRC. In particular, local officials and Congolese protested repeatedly that interviewers were going to the safest areas and their findings would understate the severity of
the situation. Given that adequate security was one of the major criteria for selecting sites to visit, this criticism is probably valid."
No, in the Iraq study, they say things such as: "Third, interviewers might, by chance, have gone to an atypical area for the Falluja cluster. We do not believe this to be the case."
Hell, they even said that the Fallujah outlier wasn't atypical. How's that for a change.
With that said, spare me your selective outrage. There's been no spilt milk over the "Burying of the ILCS" at this blog, or on blogs supportive of this one, now has there? Nope, even though the ILCS is gigantic in comparison, and far more exhaustive and informative.
As for what Read Greg said: "Oh, I forgot to mention one minor, insignificant, completely irrelevant difference: the other studies did not reflect badly on the American government and the American public."
Yeah, I mean, the Abu Ghraib story, the "torture" stories, and everything else, I am sure that they did not reflect badly on the American government and the American public. Yeah, the American media is definitely not a fan of anything that does that......... Whatever.
Complaining about sample size is a cheap-ass complaint.
Heh. Yeah, sample sizes don't really mean anything, do they? Why, there's probably no reason why major media pollsters usually use about 1,000 Americans in their opinion polls in the USA. They might as well use 500 or 200. Doesn't really matter right? Sample sizes are hype. ;) ;)
Personally, I think the IBC figures have gotten more attention than the Lancet figures because people are interested in violent deaths directly attributable to the war. And the Lancet's 100,000 deaths are mostly not violent deaths.
Not to mention that the IBC data is fully publically available, downloadable, and viewable in many different ways directly on their website. People enjoy interactivity and transparency, something that the Roberts survey of Iraq doesn't provide at all. Here we are, digging up the data from Roberts a full year later, while the IBC has had running raw data available on their site nearly from the start.
Don't tell me that hasn't played a role in media coverage and public interest. You don't have to go to marketing school to understand that.
Ragout insisted:
I'm puzzled by your puzzlement. What's unclear about the paragraph?
Oh man, getting you guys to answer questions is like pulling teeth.
From Roberts' Congo report:
"In the first stage, clusters were assigned to clinic areas systematically, proportional to population (the WHO method). In the second stage, specific locations were selected using [GPS]. [...] Within each clinic area, a crude map was made. An imaginary grid was superimposed on the map. [Two numbers would be randomly drawn,] the first number would correspond to the distnace that interviewers would travel north from the reference point, and the second number correspond to the distance they would move east to the starting point. A GPS unit was used to guide investigators to the targeted location. The 10 households closest to each [GPS] location were visited."
From Roberts' Iraq report:
"We obtained population estimates for each of Iraq's 18 Governates [...w]e assigned 33 clusters to governates via systematic equal-step sampling from a randomly selected start. Once a town, village, or urban neighborhood was selected, the team drove to the edges of the area and stored the site coordinates in a GPS unit. The area was drawn as a map subdivided by increments of 100m. A pair of random numbers was selected between zero and the number of 100m increments on each axis, corresponding to some point in the village. The GPS unit was used to guide interviewers to the selected point. Once at that point, the nearest 30 households were visited."
Hmmm. Sounds pretty similar to me.
Hmmm. Sounds pretty similar to me.
Yeah, taking things out of context sure does help make your case.
How about the parts where the Congo was done using spatial sampling, with different cluster sizes, numbers of clusters, and so on? How about where the Iraq survey clumped clusters and grouped governates together and eliminated 26% of the population areas of Iraq from being in the sample?
Apples and oranges are "similar" too, when you wrap them in paper and give them to a blind person.
The sampling approach in the 2004 Iraq survey was almost exactly like the sampling approach in the 2001 Congo survey, and the 2001 Congo survey was a refinement of the 2000 Congo surveys. Cluster number and size affect estimate precision, not estimate location.
Robert,
The Lancet Iraq study claims "Respondents were also asked to describe the composition of their household on Jan 1, 2002." I maintain that this has no meaning, because over time households split and merge; are formed and disappear.
Consider the simplest example. When interviewed in 2004, I am living with my wife. 32 months ago, on 1/1/2002, I was not yet married and was living with my parents, brothers, and sisters. Similarly, my wife was living with her parents and siblings on 1/1/2002.
Who is in the sample? Both sets of parents and siblings? Only mine? Neither?
I look forward to hearing once again how obvious the answer is, without actually being told the answer.
Robert,
Regarding the spatial sampling. The Congo studies appeared to have used GPS sampling within much smaller areas ("clinic areas") than in Iraq, where spatial sampling was used within large cities. See Roberts' discussion of how "spatial sampling is [subject] to a rural bias." This rural bias is going to be much more of a concern when spatial sampling is used in larger areas, because population density can vary more.
Also, I note that by the time the IRC got to the 11th Congo study, the sample size was over 18 times larger than the Iraq study. Perhaps we should take the Washington Post at its word, and accept that this was one reason they put more credence in the Congo studies?
Ragout:
Ah, now I understand your question, and confusion.
First, the interviewer doesn't need to make the decision on the spot. The important thing is to collect all the data about all household members and their comings and goings; when the person-months are being counted up later on, close calls can be adjudicated in a consistent way.
However, your case is actually pretty simple. It sounds to me like you're saying the household at your given address wasn't in existence over the entire period? No problemo. In general, what's important is the person-months of exposure and the number of events, so if your household has only been at that address for 30 (or whatever) months out of the 32, you just count up person months and events over the period you were there. Your parents and siblings, her parents and siblings, don't count. Your wife (and you yourself) count for the period of time you were there. If you were there for 15 months and your wife came in a month later, you have 29 total person-months between the two of you.
Actually, to be fair, you've hit on a potential design bias in the GPS thing: there is a potential bias if deaths are related to density, for example, if infectious disease, accidents, and collateral deaths (from aerial bombing) were higher in areas with higher density. If so, sampling by equal-areas would tend to result in slight underestimates of mortality.
Sample size affects estimate precision, not estimate location.
Robert,
You've given one possible interpretation of the Lancet survey definition of a household. But you've given no reason at all why yours is the right one -- and why the other plausible interpretations are wrong. The problem is that conscientious interviewers and respondents are likely to interpret ambiguity in favor of reporting more deaths.
Your claim that biases from GPS sampling are "slight" -- without any argument at all -- is just absurd.
Finally, you are indeed right that sample size affects precision, not bias. But let me repeat what I said about sample size: the full-sample estimate is so imprecise as to be worthless. So many emphasize instead the 100,000 figure, which excludes Falluja. But people want to know about violent deaths — civilians killed by soldiers — and most of the 100,000 deaths are not from violence.
Today's (jan 9)online issue of counterpunch
http://www.counterpunch.org/
has an article on the Lancet data and its reanalysis by another statistician. He uses a nonparametric analysis (I'm only vaguely aware of what I'm talking about here, so you'll have to go read the article yourself) and comes to the conclusion that the death toll is somewhat higher than the Lancet analysis suggests.
The article is by Andrew Cockburn, but I've forgotten the name of the statistician.
Thanks for an interesting link, Donald. I fear that Cockburn is in line for one of Andrew Sullivan's prizes with this line:
Googling Pierre Sprey suggests he is a former Pentagon expert and something of a character. Can anybody offer a guess as to how he is getting the numbers quoted?
Ragout wrote:
Well, it's not my definition, it's the standard one (and it does introduce what's called survivor respondent bias). When you select households, as was done here, it's the housing unit that identifies the sample. You were confused about your example because you were thinking it was a you (or your family) that was the unit, not the house. That's why I described (in some other thread) exactly what a typical collection form looks like. These forms are created with the express purpose to cut down on misstatement.
Ragout, I'm going to venture to guess you've never done any large scale primary data collection; further, I'm going to guess you haven't done much analysis of household event histories. Forgive me if I'm wrong, and then perhaps you can correct me by describing your experiences--but it strikes me you think these things are pretty ad hoc. These are extraordinarily standard definitions. As for interviewers, you don't pick any guy off the street. You spend a fair amount of time selecting them and training them ahead of time on exactly what to ask, exactly what order to ask it, exactly what tone of voice to use. If there are any doubts whatsoever, they write down everything and you sort it out when they get back to the office. If the interviewee reports on someone who died you mark it down along with the number of months he or she spent in the household, without arguing. If the household spell ended before the person died, it doesn't get counted. You typically have debriefings each day to spot problems and to make sure every interviewer knows what to do if they run into the same thing. I get the feeling you think the PI grabs the first warm bodies he finds as interviewers, sends them into the field, and sits back in an air-conditioned office drinking mint juleps until a data file magically appears in one's inbox.
About sample sizes:
From the 2001 Congo study:
"A minimum sample size was selected so as to detect a doubling of mortality above the baseline rate over a one-year period. Where time and logistics permitted, a larger sample was taken. Baseline mortality was assumed to be 1.5 per 1000 population per month [18 per thousand per year]. This calculation assumes that visiting clusters of 10 households each would reduce the statistical power of the sample by 50 percent (the "design effect = 2)."
From the Iraq study:
"Assuming a crude mortality rate of 10 per 1000 perople per year, 95% confidence, and 80% power to detect a 65% increase in mortality, we derived a target sample size of 4300 individuals. We assumed that every household had seven individuals, and a sample of 30 clusters of 30 households each (n=6300) was chosen to provide a safety margin. We selected 33 clusters in anticipation that 10% of selected clusters would be too insecure to visit."
Seems to me the sample size considerations were pretty similar. The 2001 Congo sample consisted of 1500 households, with an average household size of about 7. The Iraq sample was 1000 households, with household sizes in the same ballpark.
Robert,
Regarding sample sizes: the Congo study you mention was only one of a dozen. And the problem with assuming a design effect of 2 when deciding on sample size, is that the design effect is much larger than that when studying mortality. It's typically greater than 10. In the Lancet Iraq study it was 29 for the full sample.
The assumption of DE=2 is the norm for studies of immunization and disease, but it is far off when studying mortality due to violence, giving a sample size that is much too small.
Robert,
I am plenty familiar with large scale data collection, and you are confused when you suggest that housing units and households are identical. It is certainly not at all "standard" to use these terms interchangeably.
In fact, in the mortality surveys we're discussing here, the point is typically to determine the experience, since fleeing their villages, of refugees currently living in a camp. So there is one household but two housing units at issue. Surely you don't claim that deaths before coming to the camp do not count?
Similarly, you think that multi-decade household event histories do not ask about people's experiences at earlier addresses? I am astonished and incredulous.
Like I said, it's perfectly possible that you're right that household=housing unit is the definition Roberts used. But it's hardly the only "standard" one. For example, in this recent mortality survey, "a householder was asked to provide information on all family members as of 25 December 2004 (the day before the tsunami)." So, for the tsunami study, the respondents in my example would report on me, my parents, and my siblings at my previous residence, and not on my wife at my current house.
Also, if you're right, survivor bias is a bigger problem than Roberts' acknowledges, since it can occur not only in vacant housing units but also in occupied housing units (that were occupied by a different household 32 months ago).
Ragout claimed:
and
and
Ahh, I'm beginning to understand why our views differ so. We must be reading different studies.
I agree 29 would be a very big design effect, particularly since there were only 30 households per cluster -- at least, there were 30 households per cluster in the study I've been reading. A design effect that big would mean that each cluster is well-modeled by one household and the other 29 households were identical. Funny they didn't mention that.
From the study you linked to "A householder was asked to provide information on all family members as of [the day before the tsunami]. Information on the family was verified by referring to the household registration card, issued by the local government for food rations." Hmmm. Sounds to me like you'd need a registration card for you and your wife at your new residence, and in order to get your new card you'd have to be dis-enrolled at your parents' place.
Excellent. I'm glad to hear it. What sorts of functions did you perform? Um, was calculating intracluster correlations part of your job?
I agree. Empirically, the magnitude of the bias would depend on how many households weren't there 32 months before the interview date, which would be clear from the raw data forms. Roberts noted that "no households were dead or gone away, except in Falluja, where there were 23," which makes it sound like the survivor bias isn't big in the other clusters.
Would the survivor effect bias the death toll up or down, or does it depend on circumstances? I suppose the families that moved away from the Fallujah cluster and left the houses empty might have had lower death tolls because they were smart enough to move early. (Though according to the paper the Fallujah residents said there had been many deaths in the empty houses.) On the other hand the empty houses might be empty because everyone died, or most people died and the survivors fled. I'm probably answering my own question--the effect could go either way and the surveyors wouldn't know which way it went.
Robert,
On the design effect, it's you who are misremembering the Lancet study. It says "The crude mortality rate during the period of war and occupation was 12.3 per 1000 people per year (95% CI 1.4-23.2; design
effect=29.3)" So basically, you're claiming that Roberts miscalculated the DE.
On "the mortality surveys we're discussing here." It remains true that the Lancet Iraq survey is an adaption of methods typically used to survey refugee camps. I can't imagine what your objection to this statement is.
On the 4 year recall period. You got me there. I did misremember.
On the registration card. Presumably the pre-tsunami info wasn't thrown away. Anyway the description of the population sampled is very clear ("a householder was asked to provide information on all family members as of [the day before the tsunami]") in sharp contrast with the Iraq study. And it's clearly different than what you claim is "the standard" definition.
On survivor bias. The Lancet Iraq study makes no mention at all of inquiries about previous occupants of a housing unit, when the current occupant moved in after Jan 1 2002. It seems pretty clear that the authors hadn't considered this point. They only discuss survivor bias in the context of vacant housing units.
Robert,
Two more points.
First, one reason I find the tsunami study's definition of the sampling unit so much more credible is that they use technical terms ("householder," "household," "family") in precise ways. In contrast the Iraq study seems to use the term "household" to stand for at least 3 separate concepts (household, housing unit, and respondent).
Second, you seem to be claiming that the DE can't be greater than 30, because there are only 30 households per cluster. I think you're wrong on this point. There are several hundred people per cluster, and it seems to me that it is the number of people, not households, that is the relevant factor in calculating the DE. It is people who die, not households.
Ragout wrote:
Oh dear. You're right. Mea culpa. Not as a defense, but offered merely as an explanation, I consider the Falluja cluster to be aberrant, so I mentally skip over it. The fact that adding one cluster changes the design effect from 2.0 to 29.3 is just another reason why one of these things is not like the others, one of these things doesn't belong. BTW, speaking of aberrancy, I realize that I made a typo in my code in bootstrapping the excess mortality estimate including Falluja (I indexed incorrectly, and the error didn't affect the bootstrap for the clusters excluding Falluja). I've updated the bootstrap graphic, and marked the means and BCa CIs.
The Iraq sampling frame is wildly different from what's done in refugee camps. In refugee camps the population is, by definition, displaced and are much more likely to have experienced mortality events. Here, the point is to get an overall national estimate of the pre- and post-invasion mortality in situ. And I know you've insisted that what people want to know is the number of violent deaths; however, the point of this study is to estimate deaths from all causes. In most post-war situations, disease and malnutrition cause more deaths than direct violence.
I find Roberts' household definition pretty clear. Let's recap, adding my emphasis: "We defined households as a group of people living together and sleeping under the same roof[s]. If multiple families were living in the same building they were regarded as one household unless they had separate entrances onto the street. If the household agreed to be interviewed, the interviewees were asked for the age and sex of every current household member. Respondents were also asked to describe the composition of their household on Jan. 1, 2002, and asked about any births, deaths, or visitors who stayed in the household for more than 2 months. Periods of visitation, and individual periods of residence since a birth or before a death were recorded to the nearest month. Interviewers asked about any discrepancies between the 2002 and 2004 household compositions not accounted for by reported births and deaths. [...] The deceased had to be living in the household at the time of death and for more than 2 months before to be considered a household death."
So: it excludes your parents, your siblings, and the time when you lived in your cousin's house. It excludes your wife's parents, but it does include your wife from the time when she entered the household. If your cousin's wife's brother's nephew visited in your house for less than 2 months, he doesn't count, even if he died during that time. If he stayed for 3 months, he did. If he moved out after 3 months and then died a month later, he wasn't a part of the household and didn't get counted as a household death -- his 3 months of household tenure still added 3 person months to your household total.
Donald, on survivor bias, the event they were tracking was mortality, which was relatively low -- in their sample, the pre-invasion CDR was 5 per thousand per year. What this means is that one missed death in the numerator carries a lot of leverage and you'd need about 200 missed person-years in the denominator in order to have a neutral effect on your estimate of the mortality rate. Although the survivor bias can go in either direction, its generally thought that survivor bias leads to an underestimate of the death rate.
You describe a definition of a household that's appropriate for a point in time (a group of people living in a housing unit) but not a longitudinal study. I agree that his definition of the current household is fairly clear. You ignore the ambiguous part: "respondents were also asked to describe the composition of their household on Jan. 1, 2002." As I understand it, you think this really means "the composition of their present housing unit." You could be right (although if so, why didn't Roberts write "housing unit" ?). However, another natural reading is that it refers to the household of the respondent on 1/1/2002, which may have been a different housing unit. This is just like the tsunami study. In my example, the household of the respondent (me) on 1/1/2002 was me, my parents, and my siblings; I've since moved and my household now consists of me and my wife.
Also, I note that your interpretation is still ambiguous. Probably the biggest ambiguity is when the housing unit was occupied by a completely different set of people on 1/1/2002. You've said it's obvious that the different set of people wouldn't count, and perhaps you're right, so let's consider an intermediate case. Suppose that on 1/1/2002, I'm was living with my wife and children. A year later, my children were killed. Following their deaths, my wife and I move somewhere else, and we're later interviewed at the new address. As I understand your interpretation, this is a case where "the household at your given address wasn't in existence over the entire period," So my children's deaths shouldn't be counted. Even if this strange result was what Roberts intended, I find it hard to believe that the interviewers followed this procedure.
You write, "The Iraq sampling frame is wildly different from what's done in refugee camps." But it's only the setting that's very different. Roberts used exactly the same sampling frame and sampling methods that are usually used in mortality studies of refugee camps and such. Roberts' whole career consists of these kinds of studies. If I looked at the footnotes of the Iraq study, I'm sure I would see many references to these crisis-mortality studies. So when you say Roberts used the "standard" definition of a household, my natural assumption is that you were talking about this crisis-mortality literature. If not, what are you talking about?
The problem is that refugee-study methods just don't adapt very well to the Iraq setting. I've mentioned that scaling up from a single refugee camp to an entire country means that GPS spatial sampling will be a much bigger problem. In other threads, I've pointed out that a household-based survey may be appropriate for a refugee camp, but not for an urbanized country where many people live in institutional settings -- prisons, hospitals, military barracks, school dorms, etc. Imagine if you conducted a mortality survey of the US, and didn't include nursing homes! (Or maybe Roberts did sample these settings; he doesn't say). Finally, it's reasonable to assume that few people get married or divorced while fleeing from genocidal militias, and ignore the issue. But this is a poor assumption for Iraq.
Ragout, why don't you e-mail Roberts and ask him about your household question? His address appears on the Lancet article.
You're saying that in order to get national estimates of excess mortality people have been doing two-stage cluster sampling on refugee camps? And you're saying such an approach would be appropriate for refugee populations but not for national populations? Well, I'd agree it's inappropriate to one of those two situations.
Robert,
The Congo, Darfur, and tsunami studies aren't national estimates, and don't claim to be. What are you talking about?
And I actually am interested in an answer to my question: When you say that there's a "standard" definition of a household in longitudinal studies, where can I find that standard definition? Obviously I don't think there is any such thing, but I'm interested in attempts that have been made.
You claimed that the Iraq survey "used exactly the same sampling frame and sampling methods that are usually used in mortality studies of refugee camps and such." It doesn't.
What we're talking about is retrospective, not longitudinal.
From the WHO Health Surveys
From the UN
From the DHS
You can find two detailed country-specific interviewer manuals for the DHS here and here.
However, if you're interested in longitudinal panel studies (which this ain't) you can check out the EU household panel studies, or the US National Longitudinal Surveys, or (though I wouldn't recommend this particular one) the venerable Panel Study of Income Dynamics. You'll find that in those panel studies, fissioned-off household members create new households, and therefore are not included in the original households.
In none of the surveys above would your cousin or your wife's parents be considered a part of your current household.
The sampling approach in the 2004 Iraq survey was almost exactly like the sampling approach in the 2001 Congo survey, and the 2001 Congo survey was a refinement of the 2000 Congo surveys. Cluster number and size affect estimate precision, not estimate location.
Repeating a fallacy doesn't make it true. You didn't address the most important things I listed, such as spatial sampling and not to mention the "grouping process" which sets the Iraq study apart from all other studies that Roberts has done in the past. Plus the very forceful language of the Iraq study, compared with the careful and nuanced language of the Congo studies.
Ignoring all the differences doesn't make them similar, Robert. The fact that the cluster sizes and numbers are different than the Iraq study, and thus the precision - why isn't that good as any other reason to put more weight behind the Congo studies instead of the Iraq one? I mean, just the difference between number of studies, 11 vs. 1, gives more weight to the Congo ones.
So what you, Lambert, Paulos, now Schwartz, and others are saying is that "they did more studies in Congo, with more precision than the Iraq study, yet for some odd reason, the media stood behind the former and not the latter!!"
Yeah, I guess when you are in denial you can't see what's right in front of your face.
What we're talking about is retrospective, not longitudinal.
Not only is this a trivial, pedantic point but it is also incorrect. Data can be both retrospective and longitudinal, and the Iraq study is both. For your edification, here's a nice discussion.
All your mass of irrelevant links indicates is that you don't understand the issue I've raised one little bit. Still, it's instructive to consider exactly why they're irrelevant.
The PSID and NLSY are prospective studies, in contrast to the retrospective Iraq study. This means that if you wanted to study mortality in these data, you could look at a people at a point in time, follow them, and find out how many died at some later date. The Lancet study doesn't have this luxury. The people they want information about are already dead. So they must ask living people about the deaths of others with whom they have some well-specified relationship.
As it happens, some of the developing country surveys you mention do ask about mortality. Specifically, they ask about the deaths of children and siblings of interviewed people. This is a very common method of obtaining retrospective mortality data. The Iraq study's method of asking about the deaths of former household members is much less common, because "former household member" is not a well-specified relationship.
To summarize: mother/daughter, brother/sister: well-specified relationships widely used in mortality studies. Household member/former household member: not a well-specified relationship, rarely used in mortality studies.
BTW, you're wrong about the PSID sample too. If I, as an original sample member, later moved in with my cousin, my cousin and his family would be part of the sample as long as I lived there.
You claimed that the Iraq survey "used exactly the same sampling frame and sampling methods that are usually used in mortality studies of refugee camps and such." It doesn't.
Did so! Nyah, nyah!
Ragout:
Sigh. I thought my "retrospective vs. longitudinal" thing was clear, but now I see it wasn't, at least to the willfully obstinate. I was contrasting retrospective with longitudinal in the same way that I'd say "we're talking about men, not all humans." Since the Iraq study is a retrospective study, I was trying to restrict the conversation to the relevant comparison and exclude non-retrospective longitudinal studies.
Wait a minute. I call bullshit. That's not the situation you posed way above in post #6. The situation you posed there wasn't that you moved in with your cousin later, it was that you moved into and out of your cousin's place before the interview, and that your interview was in that new household.
And, as for that post #6, you claimed: "We have no way of knowing how this would be reported to the survey, or even what the researchers hoped to have reported."
Yes, there is a way: e-mail Roberts and ask him.