LancetIraq
tlambert | October 25, 2010The WikiLeaks Iraq archive, while incomplete, reveals many more previously unreported violent deaths in the Iraq war -- Iraq Body Count say that the archive reveals 15,000 people shot, blown up, had the heads cut off or killed in some other way that they had not recorded. So Tim Blair, who claimed that the Iraq Body Count was way way too high (and predicted that the coalition would suffer "below 50" casualties) has posted a correction. Ha ha, just kidding. Blair has a post claiming that the WikiLeaks archive, which is, as I have already noted, incomplete, proves that the Lancet study on…
tlambert | December 28, 2009Andrew Mack emails me to draw attention to his paper ("Estimating War Deaths: An Arena of Contestation" by Spagat, Mack, Cooper and Kreutz), which criticizes Obermeyer et al's paper Fifty years of violent war deaths from Vietnam to Bosnia. I commented on Obermeyer et al in this post.
I agree with some of their criticism. The regression that used for correcting PRIO estimates of war deaths is wrong and the conclusion that they drew using this correction -- that there is no evidence that war deaths have decreased is unfounded.
I'm not persuaded by their general criticisms of survey…
tlambert | March 6, 2009As promised, the Lancet has published a correction to the 2nd Lancet study:
Burnham G, Lafta R, Doocy S, Roberts L. Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey. Lancet 2006; 368: 1421--28--The Methods section of this Article (Oct 21, 2006) stated that "Participants were assured that no unique identifiers would be gathered." Upon review, it was determined that a significant number of the surveys contained names of respondents and household inhabitants. This was a lapse in the authors' obligations to protect participants. However, to the authors' knowledge…
tlambert | February 26, 2009This post contains some more notes on a reply to the badly flawed "Main Street Bias" paper.
In my previous post I showed that the MSB papers was wrong to claim that it was plausible that the unsampled regions was 10 times as large as the sampled region. In this post I look at their model. Their model is wrong because it assumes that there is no main street bias in the sampled region and because of this they massively overestimate any bias in the Lancet sampling.
Let's start with a correct model of the situation. I've adopted their terminology where possible.
We have a population of size N…
tlambert | February 23, 2009The Johns Hopkins press release states:
Data Collection
An examination was conducted of all the original data collection forms, numbering over 1,800 forms, which included review by a translator. The original forms have the appearance of authenticity in variation of handwriting, language and manner of completion. The information contained on the forms was validated against the two numerical databases used in the study analyses. These numerical databases have been available to outside researchers and provided to them upon request since April 2007.
Some minor, ordinary errors in transcription…
tlambert | February 19, 2009This post is some more notes on a reply to the badly flawed "Main Street Bias" paper.
The authors claim that it is plausible that the Lancet paper's sampling scheme could have missed 91% of the houses in Iraq. (That is, their parameter n, the number of households in the unsampled area divided by the the number in the sampled area could plausibly be 10 or more.) The only support they offer for this is a reference to this analysis of Iraqi maps.
To the right is a detail from their map. The red lines are main streets and the yellow are secondary streets. They assert that the blue areas are…
tlambert | February 6, 2009Debora Mackenzie, in the New Scientist reports on the AAPOR censure:
AAPOR charges that by refusing "to answer even basic questions" about data and methods, Burnham is preventing other researchers from evaluating his conclusions.
According to New Scientist's investigation, however, Burnham has sent his data and methods to other researchers, who found it sufficient. A spokesman for the Bloomberg School of Public Health at Johns Hopkins, where Burnham works, says the school advised him not to send his data to AAPOR, as the group has no authority to judge the research. The "correct forum", it…
tlambert | February 6, 2009Unfortunately, the Journal of Peace Research has published the badly flawed "Main Street Bias" paper. My earlier criticisms still apply, so I'm reposting them. Consider this the first draft of a reply to their paper.
The authors argue that main street bias could reasonably produce a factor of 3 difference.
How did they get such a big number? Well, they made a simple model in which the bias depends on four numbers:
q, how much more deadly the areas near main street that were sampled are than the other areas that allegedly were not sampled. They speculate that this number might be 5 (ie…
tlambert | February 4, 2009I asked Mary Losch (chair of AAPOR's Standards Committee) to comment on my previous post
I have read your entry and would note that the links you provided did
not supply the questionnaire items but rather a simple template (as
noted in the heading). The Johns Hopkins report provides only
superficial information about methods and significantly more detail
would be needed to determine the scientific integrity of those methods
-- hence our formal request to Dr. Burnham. The Hopkins website
refers to data release but, in fact, no data were provided in response
to our formal requests.…
tlambert | February 4, 2009The American Association for Public Opinion Research (AAPOR) has put out a press release alleging that Gilbert Burnham (who is not a member of the AAPOR) violated the AAPOR's code of ethics. What did he do? Their press release states:
Mary E. Losch, chair of AAPOR's Standards Committee, noted that AAPOR's investigation of Burnham began in March 2008, after receiving a complaint from a member. According to Losch, "AAPOR formally requested on more than one occasion from Dr. Burnham some basic information about his survey including, for example, the wording of the questions he used,…
tlambert | November 25, 2008The Iraq Family Health Survey, conducted by the Iraqi government and the World Health Organization, found that there were about 400,000 excess deaths in Iraq up to June 2006 associated with the invasion. The second Lancet survey conducted by researchers from Johns Hopkins and Al Mustansiriya University found that there were about 650,000 over the same time period. Both surveys missed the most violent period in Iraq -- if we project forward to the current day I estimate that the net cost of the war so far has been between 750,000 (using IFHS) and 1,250,000 (using Lancet2) deaths.
So how on…
tlambert | November 24, 2008I think it is worthwhile to update James Wimberly's comparison of surveys of deaths in Iraq. In the table below death tolls have been extrapolated to give a number of deaths due to the war up to Oct 08.
Survey
Violent deaths
Excess deaths
ILCS
160,000
Lancet 1
350,000
510,000
IFHS
310,000
740,000
Lancet 2
1,200,000
1,300,000
ORB
1,200,000
It is interesting to see that the IFHS ends up right in the middle, between the two Lancet studies. If you think that the IFHS study is reasonable then you must conclude that Lancet 1 has been confirmed and the critics of Lancet 1 were wrong.
What about…
tlambert | November 24, 2008
I've been remiss in not commenting on Obermeyer, Murray and Gakidou's paper in the BMJ, Fifty years of violent war deaths from Vietnam to Bosnia: analysis of data from the world health survey programme. OMG derive estimates of violent war deaths in thirteen countries from the World Health Survey and compare them with counts from passive surveillance (like the Iraq Body Count). Here's my graph of their results, showing 95% confidence intervals for the ratio between the survey estimate (WHS) and the passive count (Uppsala/PRIO). The green line is the weighted average of the ratios (2.6).
As…
tlambert | November 13, 2008Robin Meija writes about the Lancet studies:
In any case, such problems are common in war zones, according to nearly a dozen leading survey statisticians and epidemiologists I spoke with. "Iraq is not an ideal condition in which to conduct a survey, so to expect them to do the same things that you would do in a survey in the United States is really not reasonable," says David Marker, a senior statistician with the research corporation Westat. Even if the outdated population data led the researchers to a 20 percent overestimate, Marker explains, the revised death toll would still be at least a…
tlambert | June 23, 2008The punditariat at the Australian has lashed out at bloggers yet again (see here and here for previous examples). This time it's David Burchell, whose thesis is that all bloggers provide is a "vast outpouring of pseudo-expertise and vituperation". Naturally bloggers have responded, with Gary Sauer-Thompson writing
There is no attempt by Burchell to engage with any Australian political blogger. All are condemned and tossed into the waste bin without any argument. Burchell's position is one in which the reasoned arguments of Australian political bloggers on public issues is characterised by…
tlambert | March 29, 2008Julie Rehmeyer has a decent article about the Lancet studies in Science News. Unlike Neil Munro and Megan McArdle, she doesn't have an axe to grind. She talks to experts in the field like Jana Asher instead of non-experts like Michael Spagat:
The conflicting studies in Iraq show just how tricky it is to apply these methods in messy real-life situations. About the Lancet study, Asher says, "I don't think there was anything obvious in what they did that someone can point to and say this method is flawed. But the WHO study used appropriate methodology too."
The most suspect part of the Lancet…
tlambert | March 28, 2008Last year AP-IPSOS surveyed Americans and asked them to estimate how many Iraqi civilians had died in the war. They grossly underestimated the number, with the median estimate being just 9,890. The Atlantic has now published Megan McArdle's latest anti-Lancet screed, where she argues that it would be better if the Lancet studies had not been published at all because they make people more willing to accept higher estimates of Iraqi deaths. Yes, for war-advocate McArdle, the big problem is that people's estimates of Iraqi deaths are too high.
McArdle's piece reminds of me of Neil Munro's…
tlambert | March 18, 2008Les Roberts has a given a briefing to German parliamentarians on deaths in Iraq.
Hat tip: Media lens
tlambert | February 27, 2008Diane Farsetta has an excellent and comprehensive write up on the Lancet and other studies on deaths in Iraq. A few extracts:
Theoretically, the public health surveys and polls that have been conducted in Iraq -- at great risk to the people involved -- should help inform and further the debate. But the data is complicated by different research approaches and their attendant caveats. The matter has been further confused by anemic reporting, with news articles usually framed as a "he said / she said" story, instead of an exploration and interpretation of research findings.
These are the…
tlambert | February 26, 2008Mark Hoofnagle reviews a new group blog, Science-Based Medicine.
I must say I've loved much of the writing at the new blog Science-Based Medicine. These guys are fighting the good fight and presenting very sophisticated aspects of evaluating the medical literature in a very accessible way. In particular I'd like to point out David Gorski's critique of NCCAM and the directly-relevant articles from Kimball Atwood on the importance of prior probability in evaluating medical research.
The writng that Hoofnagle doesn't like are two poorly researched posts from Wallace Sampson, who uncritically…