Burnham answers questions about Lancet study

For those of you interested in Iraqi hospitals

"Sunni Arabs are nervous of even going to the central Baghdad morgue to look for their dead because they fear they may be targeted by Shia gunmen. One Sunni who took his brother to the morgue was asked: "Do you know who killed him?" When he answered: "Yes" he was immediately shot dead. Many people with bullet wounds fear entering a hospital on the grounds that they will be accused of being an insurgent."

Burnham said: The number of people or households interviewed and the number of clusters used does NOT depend on the population of the country.

However the extrapolation up from the sample to an estimate for the whole population does depend on the estimate of the total population in 2002, and here he chose to rely on a single estimate disregarding the World Bank and other independent sources, resulting in about 60,000 "extra" "deaths". The "extra" bit was also critically dependent on the pre-invasion mortality rate. Rather than use respectable independent estimates of that, Burnham used the rate generated by his clusters. This by any reckoning is either an unethical and unscientific procedure, or else implies that his clusters were not representative of the mortality rate for the population as a whole. The baseline should be independent of the sample.

Using the World Bank's totally credible pre-invasion mortality rate, Burnham's excess dwindles to around 100,000 if memory serves, consistent with the IBC as I recall.

Tim Curtin: The baseline should be independent of the sample.

In a longitudinal study? You amaze me, Tim.

Tim Curtin wrote:

The baseline should be independent of the sample.

Oh my.

There may finally be a claim of bias that might stand up--medialens has a link to an article in Science where several researchers claim there is a "main street effect". That is, the sampling procedure may have made it much more likely that people living near main streets were sampled, and the argument is that these people would also be the most likely ones to be exposed to high levels of violence--attacks on US convoys, car bombs, etc...

I'm not sure that would apply to death squads. But it would apply, presumably, to many forms of violence. (The people asserting the effect sound hostile to the Lancet authors, btw, but that's not uncommon in science. And it seems IBC has been in contact with them--the vague statements about survey techniques that would apply to studies of disease but not to wars make more sense now.)

I'll look for the link--at medialens the article was copied out, but I didn't know if it would be appropriate to copy it and paste it here.

I don't have time to look--here's the opening lines of the article, so someone else could maybe find a link

Science 20 October 2006:
Vol. 314. no. 5798, pp. 396 - 397
DOI: 10.1126/science.314.5798.396

News of the Week

EPIDEMIOLOGY:

Iraqi Death Estimates Called Too High; Methods Faulted

John Bohannon

It's here for anyone who has a subscription:

http://www.sciencemag.org/cgi/content/summary/314/5798/396

I presume it's okay to quote a paragraph:

Neil Johnson and Sean Gourley, physicists at Oxford University in the U.K. who have been analyzing Iraqi casualty data for a separate study, also question whether the sample is representative. The paper indicates that the survey team avoided small back alleys for safety reasons. But this could bias the data because deaths from car bombs, street-market explosions, and shootings from vehicles should be more likely on larger streets, says Johnson. Burnham counters that such streets were included and that the methods section of the published paper is oversimplified. He also told Science that he does not know exactly how the Iraqi team conducted its survey; the details about neighborhoods surveyed were destroyed "in case they fell into the wrong hands and could increase the risks to residents." These explanations have infuriated the study's critics. Michael Spagat, an economist at Royal Holloway, University of London, who specializes in civil conflicts, says the scientific community should call for an in-depth investigation into the researchers' procedures. "It is almost a crime to let it go unchallenged," adds Johnson.

Of course the paper does not indicate that the survey team avoided small back alleys; where did the author (John Bohannon) get that idea?

The three critics mentioned seem to be working on a critique, but as yet it's a bit shapeless; here's a story about it:

http://www.scoop.co.nz/stories/print.html?path=WO0610/S00436.htm

Kevin Donoghue wondered:

Of course the paper does not indicate that the survey team avoided small back alleys; where did the author (John Bohannon) get that idea?

It's because of the description in the paper about how crossing streets off main streets were selected; it was ambiguous. In the "David Kane" thread, Morgan had the same question when he asked about qualifying crossing streets. Frankly, I had had the same question, but I heard through second-hand channels a couple of days ago that the sampling frame had been extended to include non-crossing streets. I didn't reply to Morgan then because my info was second-hand and I was waiting on direct confirmation.

Fair enough, Bohannon's article seemed like something of a hatchet job, but I don't know the answers he was getting from Burnham. At all events the physicists' critique should be entertaining when it appears: they are devotees of the power laws cult apparently.

I would love to see the entire Science letter (strange that it would not be in Nature), but cannot. As a geographer, I don't see this snippet above as lethal to the findings.

1.) Superficially there may appear to be a problem with spatial correlation between the causal mechanisms and the measurement of death. However, insufficient time has elapsed to where one can demonstrate that the sampled households spatially autocorrelate with reported car bombs, artillery strikes, shootings, and what not. Its hard to argue that a bomb in the Suq or the CBD or at the Mosque will only affect main street residents, but not the alley residents. Only urban warfare within the neighborhood would have likely have this effect.

2.) The primate city, Baghdad appears to have been Europeanized (see national geographic map of the city)in a manner similar to European cities during the mid 19th century. One can see streets laid out in a grid, roundabouts--no mechanism to avoid proximity to the main street. Make a run of forty residences, and likely you have included the furthest point from the main street.

3.) Most critical--violence is likely to have occurred while people are traveling, and would therefore be just as likely to affect those living adjacent to back allies and the like, as those whose residences front the mainstreets.

Mike

Kevin: Can I presume that the data undermines Bohannon et al.'s log tranformation for the conflict in Iraq?

Mike

One thing I found interesting is that Gourley has said:

"we went back through their original Lancet paper and looked at the sampling method there - they used a GPS device to make a more representative sample. Except for the case of Falluja where they employed the 'main street bias' method described in the the second lancet paper."

Recall that Falluja was the extreme outliar in L1. According to all other evidence (even L2) it's fairly clear that it was a wild overestimate.

And it seems the whole of L2 was done Falluja-style.***

Perhaps a lot of loose ends will all start to come together from this. Good work Gourley/Johnson/Spagat.

***But then who really knows how it was done. Another rather stunning quote in the Science article reads: "[Burnham] also told Science that he does not know exactly how the Iraqi team conducted its survey."

There was an email exchange between one of the critics and an outsider that was linked at the medialens site. This guy seemed to be taking the 112,000 excess death toll from Lancet 2 for the first 13 months and comparing it the ILCS survey (28,000 at most) and saying that maybe a factor of 4 correction would be valid. I don't mean to be nasty, but that sounds like something I would say.

If the criticism turns out to be valid, would any of Lancet 2 be salvageable? For instance, they find that Baghdad was only medium-violent by Iraq standards--would that still be true? Or is there just no way to tell, so the study would have to be scrapped? In which case we'd be backing to taking the media reported figures and multiplying by some unknown correction factor, since the arguments people have been making about why it's likely the violence level is considerably higher than the official statistics would still be valid.

Also, is this main street criticism valid at all for Lancet 1? The critics seem to think so for Fallujah, but Fallujah was a special case.

Donald Johnson: please provide links, its exhausting hunting down someone said what on medialens without one.

Mike

Can I presume that the data undermines Bohannon et al.'s log tranformation for the conflict in Iraq?

Mike,

I presume you mean Johnson, Gourley and Spagat? I've no idea. I never heard of these guys before today; the only physicist who ever discussed his research with me was trying to model the internal waves you see in a pint of Guinness when it is settling. He wasn't expecting any help from me, he just didn't like drinking alone. Judging by the Guardian link, these are diaspora physicists who mostly work in fields far from their native quarks. I'm quite curious to see what they come up with.

When I first saw what they were at I thought they had simply misread the paper, which remains a possibility. So far all we have is a promise that a critique will be forthcoming.

This may be the relevant Medialens post:

http://www.medialens.org/forum/viewtopic.php?t=1949

Donald,

As a veteran of the Lancet wars, you may recall these posts at Ragout's blog, where he complains that Tim Lambert has failed to address the Apfelroth critique of the 2004 paper and explains what Apfelroth is trying to say. I think he was crediting Apfelroth with an idea which is in fact his own.

Notice that if Ragout is right, the Lancet 2004 study oversampled the fringes of cities, whereas if the Oxford physicists are right, the Lancet 2006 study oversampled the city centres.

What the critiques have in common is the assumption that the oversampling will bias mortality upward. (Fair play to Ragout, he acknowledged that the bias might actually be downward.) But if oversampling the centres gets you "too many" deaths and so does oversampling the suburbs, where do you go to get "too few" - the desert?

It also occurred to me that maybe there's a main street bias regarding non-violent deaths and I don't know which way that would go. It is peculiar that the Lancet 2 paper shows such a small increase in non-violent mortality. I know breaking the data down into subsets means the analysis is less and less precise, but still, I would have thought there'd be a more even breakdown between violent vs. nonviolent deaths, whatever the actual level is.

I didn't remember Ragout's argument, or not that one. It'll be interesting to see if the dumb arguments against Lancet2 are shelved now that there is a possibly serious criticism on the table.

I'm somewhat sympathetic to the gut level analyses of IBC and the rightwing skeptics, though, because I think the same way on this subject--with my gut. The same gut which made me a little leery of accepting this study's numbers also makes me think the numbers in the press are much too low, particularly regarding the killing done by coalition forces. The few numbers on that subject which do exist make no sense-- 370 civilians identified as killed by the coalition in the third year, according to an IBC press release last spring (no, I don't have the link and don't know if it is still online), vs. a claim by Michael O'Hanlon the harsh Lancet critic who said in late 2005 that the coalition was killing 750-1000 insurgents per month. In no known universe could both those statistics be correct, or so my gut tells me. And I don't think the US has troops in Iraq just to rebuild schools, so I tend to believe that a lot of insurgents are dying. (No link for that O'Hanlon number either, but I found it by googling around earlier this year during the medialens Lancet/IBC war.) And dsquare's Northern Ireland analogy, which I found semi-convincing (or at least the American translation to cities with high murder rates) is an appeal to the gut which made sense to me.

Re: the mainstreet bias

According to posts above.

Lancet I reportedly assigned clusters by by random geographical coordinates which somewhat oversampled less densely populated areas.

Lancet II assigned clusters within 40 houses of a "main street" which supposedly somewhat oversampled more populated areas.

Both Lancet I and Lancet II came up with similar estimates (98000 vs 112000) for excess deaths for the 1st post war period.

So maybe there was no biasing effect.

BTW previous studies have claimed higher child mortality in rural Iraq. Versus hypothesized higher violence mortality in cities and "main streets". Maybe they cancel out. Or are too weak.

Tim Curtin said: "Using the World Bank's totally credible pre-invasion mortality rate, Burnham's excess dwindles to around 100,000 if memory serves, consistent with the IBC as I recall."

You recall incorrectly.

Also, if Lancet had used the World Bank's pre-war death rate estimate of 8.8 per 1000 (2000) instead of the 5.5 per 1000 rate that they came up with, the total excess deaths would have been 375,000 instead of 650,000 -- NOT 100,000, as you indicated.

This is pretty simple math -- basic fractions.

Besides, even the incorrectly caluclated 100,000 numberthat you give is not consistent with the Iraq Body Count estimate, which is roughly half that.

Exercise your fingers a little. It's really not that much effort.

http://www.iraqbodycount.org/

"I think the same way on this subject--with my gut. "

My gut tells me it needs more food; an independent opinion from my MD tells me I am 10 kilos overwight. George Bush was told by his gut to invade Iraq.

Conclusion: guts can't be trusted.

Bush uses his gut to think because, after the operation, that's all he has.

just to observe that "main street bias" is perhaps a curious term to use for a methodology that, by definition, does not sample main streets at all. "Road adjoining main street bias" would rather throw into relief that there is no a priori reason to believe (as Johnson et al keep asserting without evidence) that the roads sampled would have unusually high violence.

Agreed - it seemed to me that there could be a downward bias if anything. But I don't know enough about how the sampling was done. Johnson et al allege that Burnham doesn't either, and in fairness the paper isn't clear on that; but the point you make tells against the critics a bit. It's a familiar pattern: spot a possible source of bias and have everyone think the bias must be upward.

"Rather than use respectable independent estimates of that, Burnham used the rate generated by his clusters"

We compared the estimate of postinvasion deaths given by George Bush in that press conference with an independent estimate of prewar deaths given by the irai government press office in a press release decrying UN sanctions and found that the invasion had actually restored life to 2000 Iraqis.

Correction.

I misrembered. The study randomly selected a street intersecting a "main street", then *numbered* the houses on that intersecting street, and then randomly selected one of those numbered houses to start their interviewing.

How far down the intersecting street the houses were numbered the study doesn't say (presumably to the end of the street), but it was not limited to only 40 as I stated.

My initial take was that the sampling methods in both studies have similar biases: they sample according to area rather than according to population, so they will both oversample less-dense areas. I hadn't realized that in the second study they exclude main streets and streets not intersecting main streets.

First, on the fringes of cities, I think this new scheme will undersample less dense areas, since people in these areas mostly live strung out along the main street. I'm thinking here of US exurbs and semi-rural areas, but see the agricultural areas (in green) on this map of Basrah. In truly rural areas, I find it hard to imagine how this scheme is supposed to operate (what if there are no side streets?).

Second, this scheme will undersample people living in apartment buildings. They sample street addresses with equal probability, so a house is just as likely to be sampled as an apartment building, even though a lot more people living in the apartment building. Also, I assume that apartment buildings are more likely to be located on main streets.

Third, if you look at the map of Basrah, you see that *most* blocks are neither main streets nor streets intersecting main streets. Most blocks are interior blocks, back alleys where the street doesn't run through and intersect with the main streets.

So, in cities, it seems that the Burnham et al sampling scheme systematically excludes *most* of the population. Hence, the bias could be large, though it's hard to guess which direction it is in.

Still, I'll speculate. I would guess that the excluded interior blocks are safer, and more likely to be avoided by military and sectarian patrols. In these back streets, they probably run greater risk of ambush, and would find it harder to run away if attacked. The undersampled apartment buidings might be easier to secure from sectarian violence, by just stationing a few guards in the lobby. Or they could be more dangerous if fighting is more likely on main streets. On the third hand, many people are presumably not killed at home. Then the bias would be driven by socioeconomic differences between people living in, say apartments vs. houses, or quiet streets vs. busy streets.

Richard

In defense of gut analysis--it's fine to use one's gut so long as you don't take its conclusions too seriously, I think. It's perfectly reasonable for people to hear this new estimate of 600,000 dead and think "Wow, that's ten times higher than most claims. Is there a bias in the study? Maybe it should be replicated and until then, all our beliefs about the Iraq death toll are up in the air." In the meantime, there will be plenty of blog posts arguing (from gut-level analysis) why the study's results are or are not plausible.

Being tentative about Lancet 2 seems more reasonable than either rejecting the study out of hand because it disagrees with other claims, or on the other hand, accepting it as almost certainly true because one can't see how any sort of bias could have crept in. But that's just my gut talking.

It occurs to me that an excellent sampling frame for Iraq exists: the one put together for the Iraq Living Conditions Survey. For example, they sent large numbers of people into the field to identify and count housing units, and insure that their sample frame was up to date.

So it isn't actually necessary to rely on the second or third-best sampling techniques. I wonder why Burnham et al didn't use the ILCS frame? It's easy to imagine plenty of legitimate reasons why they didn't (maybe the ILCS folks wouldn't give them the data) but I'm still curious.

Ragout,

Can you describe the ILCS sampling techniques in enough detail to evaluate. Why were they good, besides sampling many more clusters.

Where in the report can they be found?

Also, I've never found in the report what question the ILCS people actually asked regarding civilian deaths. Why isn't there any statistical data set out in the report for that question? Or is it in some supplement.

Does anyone else find this exchange strange?

Minneapolis, Minn.: I have a couple of skeptical questions, not politically motivated but in the interest of good assessment of your work.

First, if I understand correctly, you had a very low refusal rate, which is surprising. How did you get to the households you surveyed? Specifically, did you choose them genuinely and fully randomly, or did you have to go through some set of community leaders?

Second, what is your explanation for the fact that there appears to be such a discrepancy between your numbers and the number of death certificates issued that might reflect your numbers?

Gilbert Burnham: Our concern is for populations caught up in war. To improve protection of civilians we need to have data about who is dying and why. It is hard to collect these data, which is why there are so few reports. Our intent is to help develop better methods to do this. The study was originally intended to be completed by July or August of 2006, but a number of delays happened and we finished it when we did. If we can discuss these findings to discover better ways of protecting innocents from death in wars, then we will feel that the risks of all were worthwhile.

Perhaps the transcript is messed up, but Burnham seems to ignore the critical question of randomness in respondent selection. Has he addressed this elsewhere (besides what is in the report)?

It seemed to me like he ducked the question.

David Kane, the randomness is described in the study (though they say that description was over-simplified).

JB: You are right, my recall was wrong, and I apologise. However there are still some issues: using my fingers at your suggestion, I found the WHO site on Iraq, showing a crude mortality rate in 1998 of 9.029 and from its lifetables for 2001, 9.26, a pre-invasion increase, and well above the Johns Hopkins 5.5. The 2001 lifetables show total population in that year of 23,583,410, and an implied net growth rate of 2.6% p.a. Using the WHO baseline and the Lancet/JH crude mortality rates, we have an initial beneficial impact of the 2003 invasion on the mortality rate in 2003/04(down from 9.26 to 7.5), then a small rise in 2004/5, from 9.26 to 10.9 (producing excess deaths of 43,301), and a much bigger rise in 2005/06, to 19.8 per 1000. This implies excess deaths of 261,000 in that year, for a total of over 304,000 (ignoring the apparent beneficial effect in 2003/04). That is 350,000 fewer than Lancet/JH. Is this really believable, given IBC's patient compilations of daily deaths that appear to match the headline events reported in the media? The Lancet/JH daily death rate in 2005/06 is 716 (using the WHO population projection, many more using theirs), compared with TOTAL deaths in 2001 of 213,000 or 583 per day. Frankly, these figures show that the JH study was indeed no more than political propaganda devoid of all scientific credibility. But in the language of this blog no doubt the IBC people are all liars and the JH team lies in repose above us with the archangels and all the heroes of the death squads responsible for at least two-thirds of the JH deaths as even JH grudgingly half admit.

From Iraq Body Count:

"Casualty figures are derived from a comprehensive survey of online media reports from recognized sources. Where these sources report differing figures, the range (a minimum and a maximum) are given. This method is also used to deal with any residual uncertainty about the civilian or non-combatant status of the dead. All results are independently reviewed and error-checked by at least three members of the Iraq Body Count project team before publication."

There is absolutely no way that this method can produce anything but a huge undercount, given conditions in Iraq. Media reports? Please. Just consider the fact that Islamic custom requires bodies to be buried within a day. Lots of victims are not dying in hospitals and never go to the morgue.

"..But that's just my gut talking. "

Yes, but that's the problem. Please ask your gut to stop talking; try some research instead. Continual references to your gut do not add anything to the discussion.

"Just consider the fact that Islamic custom requires bodies to be buried within a day."

Excellent point. The same is true of Orthodox Jews, and I can assure you that adherents, if faced with the choice of either burying the body within 24 hours or reporting it to authorities, would absolutely follow the former track.

been looking around and found the first version of the 2006 Lancet paper. It is on the MIT site [link].

The key place to look here is in the methods section. From the first version of the 2006 paper

Methods

"In this survey,sites were collected according to the population size and the geographic distribution in Iraq.The survey included 16 of the 18 governates in Iraq, with larger population areas having more sample sites.The sites were selected entirely at random, so all households had an equal chance of being included.The survey used a standard cluster survey method, which is a recommended method for measuring deaths in conflict situations"

which, in the lancet version of the paper strangely changes to this;

Methods

"As a first stage of sampling, 50 clusters were selected
systematically by Governorate with a population
proportional to size approach, on the basis of the 2004
UNDP/Iraqi Ministry of Planning population estimates
(table 1). At the second stage of sampling, the
Governorate's constituent administrative units were
listed by population or estimated population, and
location(s) were selected randomly proportionate to
population size. The third stage consisted of random
selection of a main street within the administrative unit
from a list of all main streets. A residential street was
then randomly selected from a list of residential streets
crossing the main street. On the residential street, houses
were numbered and a start household was randomly
selected."

Suddenly the references to random disappear. In the first version of the paper, the housing selection is random, in the second version of the paper the housing selection is anything but. Why will the authors not admit to main street bias when their sampling algorithm so clearly contains it?

jkbaxter,

That's not a "first version" of the Lancet paper. It's a simplified account intended for the general reader.

fair point Kevin - "a simplified account for the general reader"

So it is okay to tell the general public that this is a random survey when in fact it couldn't be further from the truth?

Have you stopped beating your wife, jkbaxter?

okay - let's put it another way, why state in one version of the paper that the sampling method is 'random' and in another version of the paper that it is not random?

1) the authors were not aware that there was a bias in which case they were simply wrong when describing the sampling method in the MIT version

2) the authors were aware of the bias, but instead kept this feature of the survey hidden in the MIT version.

do you even accept that the main street + cross street algorithm introduces a bias to the sampling?

Ragout, you obviously know a bit about statistics, so I'm curious:

If a study intended to use a random sample actually uses one that may be biased, is standard practice to reject it in its entirety? E.g. are we wrong to place any credence in telephone polls, where a large fraction of people will be unreachable or refuse to respond?

Do you have any guesses how large the biases you identify are likely to be in practice, and in which direction?

Wouldn't a reasonable response to the problems you identify be simply to treat the study as if it had a larger confidence interval than the reported one? And if so, are there any rules of thumb to suggest how much larger?

Basically, my question is, suppose you're right, what then? Presumably even a study with the flaws you describe provides some information about Iraqi mortality, no?

do you even accept that the main street + cross street algorithm introduces a bias to the sampling?

For now my comment upthread stands.

Um, Richard? Since both of us are calling for more research, there is no substance to our disagreement, except a silly semantic one.

"Um, Richard? Since both of us are calling for more research, there is no substance to our disagreement, except a silly semantic one."

I'm not calling for more research. I'm just asking you to stop using your gut as a substitute for thought. I happen to think that the Lancet data are pretty clear and that your objections aren't really based on anything substantive.

I was reading through the posts, and one comment from chew2 caught my eye.

something along the lines of both studies came up with similar estimates for post war excess deaths (98,000 vs 112,000).

if i have read the papers right there was not main street bias in the first study but it might have existed in the second. Then if the numbers are equal the bias in the second paper must be insignificant. Which is the point chew2 is making.

However my understanding of the main street bias criticism is that it seems to only apply to violent deaths. Do the excess violent death rates agree for both surveys?

I've had a quick look and neither paper states what the violent death rates are for each period of time. I guess you could work it out, but I don't trust my maths on this one. Any volunteers?

Richard, I just sent off a reply which went into the void. Here's a short version of it.

1. I'll keep using my gut. Everyone does--they just don't all use the metaphor. If the Lancet2 had said 6 million deaths everyone's gut response would have been "nonsense". Yours would have emitted a few rumbles.

2. Substantive reasons--I don't have any I'd consider strong. There are some I'd consider worthy enough to justify calls for more research. The UNDP/ILCS study showed 19-28,000 war-related deaths for the first 13 months. The comparable numbers from Lancet2 for that time period would be 50-125,000 or so (Table 2 or 3, I think.) The latter numbers included criminal murders and the former did not, so they are not directly comparable, but they do seem to indicate a discrepancy of maybe a factor of 2-4, depending on how criminal murders factor in. The Lancet paper says passive surveillance tends to greatly undercount deaths. Perhaps so in most wars, but I haven't seen anyone claim the Israeli Palestinian conflict is five or ten times more violent than generally reported. So it's not unreasonable to require more than one study to demonstrate that passive surveillance methods are failing by a factor of ten in this war.
There is also this alleged main street bias which needs to be investigated.

3. Gut analyses--Everyone uses them. Dsquared brought up the Northern Ireland/Jamaica/LA in the crack wars analogy to show that IBC's figures seemed too low. Sounded plausible to me. Nobody would take this as conclusive. And again, there's got to be some number the Lancet2 paper might have produced that your gut would have told you was too high, something short of the entire population of Iraq.

4. So my gut tells me that we need more research. Sorry if that metaphor bothers you.

Donald Johnson wrote:

The UNDP/ILCS study showed 19-28,000 war-related deaths for the first 13 months.

How did the UNDP/ILCS study get that number?

chew2,

The ILCS sampling method is described in appendix 2 of the Analytical Report. They go to great lengths to sample small areas (blocks) proportional to population size, and to develop weights to adjust for any errors in the population data found by their field personnel. It's very much like the appendix you'd find in a US government-sponsored multi-million dollar survey.

The questionnaire can be found here.

Ragout wrote:

They go to great lengths to sample small areas (blocks) proportional to population size

I haven't looked at the ILCS sampling for a while, but I thought they had an equal number of households per governate (except for Baghdad)?

Looking at the ILCS, it seems you are right. I should have said they sampled blocks proportional to population size within each governate, and then used weights to get a representative sample. Still, the basic point remains the same: the ILCS used an excellent sampling scheme.

BTW, here's the correct link to the ILCS report.

Right. I didn't mean to imply that their sampling scheme was poor -- merely that it wasn't proportional to population. That means that there's an extra step if you're trying to get from survey findings to national totals.

anyone know the numbers for the violent death rates for the first 17.8 months of both the 2004 and 2006 Lancet papers?

I get 140,000+ for the first 17.8 months of the second paper, not sure if this is right though?

anyone have an estimate for the first paper? From what I can tell it must be less than 98,000 but how much less. 50,000?

If I'm reading the ILCS right, there were 1100 households in 110 clusters interviewed for each governate, except Baghdad which had 3300 and 330.

*This would seem to seriously over sample less populous governates, since the same number was sampled regardless of the population of the governate. Baghdad was also undersampled relative to its population.* Could we call this a highly biased sample?

As to the sampling. They used the 1997 Iraq Census to break down the country into rough census tracts (clusters) consisting of 100-200 households. They selected a sample of these clusters "randomly proportionate to population". Within each cluster they supposedly mapped each household and then selected 10 households by "linear sytematic sampling". I don't see how they could have mapped every household in the cluster without actually going out to each cluster. It's not clear to me whether that is any more random than the Hopkins study.

chew2,

Yes, they oversampled smaller governates so that they could report precise statistics even for small governates. But no, this doesn't cause any bias. They used weights to adjust for the oversampling.

You also say "*I don't see how they could have mapped every household in the cluster without actually going out to each cluster*."

But they did go to every cluster! First they randomly selected blocks in proportion to the population size in the 1997 Census. Then they sent people into the field to draw maps and make lists of addresses in the blocks. Then, back in the home office, the statistician selected a random sample of 10 housing units from the block. They also used the information from the "listing" operations to adjust the weights for any changes in population since 1997. Finally, people were sent into the field a second time to actually conduct the interviews.

All this adds up to a very high quality sampling method. It's much better than the Burnham et al method. It's even much better than most surveys conducted in the US.

Ragout,

my understanding is that they had 500 people working on the UNDP survey vs. the two teams (of 3?) working on the Lancet Survey. As well the Lancet survey only had a budget of $50,000 so hard to fault them for not measuring as many houses and having to use a cluster design.

Ragout,

First I forgot to thank you for posting the links to the ILCS questionaire etc.

The ILCS questionaire asked the household about deaths and people gone missing in the household over the previous 24 months, due to illness and various other causes including "war related deaths".

Why weren't the mortality figures printed in the report and a death rates calculated? I couldn't find them. All I could find was the section in the Analytical report on war related deaths, and child morality figures. We could then compare tthe mortality figures with other estimates for accidents, crime, illness etc. and calibrate the "war related deaths" estimate to see whether it was in the right ball park.

Why do most claim the ILCS warfare deaths figure relate to the 12 months after the end of the war when the questionaire refers to the "prior 24 months"?

Were the original child mortality figures too low (they decided to re-survey because they suspected this), and if so could the war mortality figures have been low also?

Thanks.

The 19-28,000 figure was mentioned somewhere in the ILCS survey. I downloaded part of it and read it myself, but don't have it available where I am now.

TJ11 asked about violent death rates in Lancet 2. I think the closest you can come to estimating the first 18 months is by taking the violent death mortality rate from Table 3. For March 03-April 04 the 95 percent CI is 1.8 to 4.9 per thousand per year, with 3.2 being the center. (This is where I got my 50,000 to 125,000 figure for the first 13 months, though I think I goofed and the latter should be more like 130,000). For the May 04-May 05 period the violent mortality rate is 4.0 to 9.8, but centers on 6.6

I guess you'd take the 3.2 for the first 14 months and the 6.6 for the remaining 4 months. Using 25 million for the population, I get around 148,000. Might have goofed again on the arithmetic--I did it in a hurry.

thanks for that Donald, I had a go at the calculation too and got around 145,000 deaths. It seemed strange to me that this would be higher than the 112,000 quoted in the paper. But then I realised it was for violent deaths.

I also had a go at calculating the violent deaths for the second paper and got about 60,000 deaths. Have you had a go at this?

We keep hearing that "main street bias" may have inflated the Lancet figure.

A couple of quick questions:

1. Where do people think most US and Iraqi government patrolling takes place?

2. Where do people think insurgents or common criminals are mroe liekly to operate with impunity? Main roads or back alleys?

tj11,

In Hopkins I there was a statistically significant increase in the total death rate, but the sample was too small to claim that there was a statistically significant increase in either the violent or the non-violent death rates, even though there was an increase in both violent and non violent deaths in the sample. In Hopkins II there was a statistically significant increase in both the violent death rate and the total death rate, but not in the non-violent death rate.

So you can't really compare the violent deaths in Hopkins II with those in Hopkins I. That is why they only compared the overall estimated deaths in Hopkins II with those in Hopkins I.

Possibly our 18 month Lancet II estimate is a tad high, since the 6.6 mortality for the second year might have been steadily increasing--it possibly was lower during the first 4 months. But with the huge error bars, it doesn't matter much.

Yeah, the usual estimate for Lancet 1 was 60,000 violent deaths--I think the more precise estimate people got somehow (not from the paper, I don't think) was 57,000. Anyway, 21 deaths and each represented about 3000 deaths. I have no idea what the confidence interval would be on that and defer to chew2 that it was so big that comparisons with Lancet 2 aren't really very meaningful.

I've never had a good back-of-the envelope understanding about why the CI would be so big for those violent death estimates in Lancet I. I think it was 21 deaths in 15 different clusters (or maybe it was 14 excluding Fallujah), and my impression (don't remember if it is based on anything someone said in the previous Lancet wars at this blog) was that no cluster outside Fallujah had more than 3 violent deaths. But don't take my word for that, because I don't remember where I got that impression. Anyway, I'm very far from an expert and so defer to the people around here who are.

thanks for the replies, just checking that I understand.

The hopkins 1 survey found an increase in the total number of deaths but couldn't determine if it was from violent or non-violent deaths because the results were not statistically significant. I guess this is one of those definitions from stats text books so I will defer to you on this.

How many more deaths/samples would have been needed to show a statistically significant increase? I would have thought 21 deaths after the war started and 1 death before it would have been a pretty good indication of a significant increase. But again I defer to the stats gods amongst us.