Wall Street Journal on Lancet study

I missed this when it first came out, but Carl Bialik has written excellent summary of the issues in the Wall Street Journal.

Researchers concluded that about 100,000 more Iraqis had died outside Fallujah since the invasion than would have died had the prewar death rate continued. Yet the study, published in the British medical journal Lancet, was roundly criticized for discarding the Fallujah data from calculations. Others questioned the study for extrapolating from only 89 death reports outside Fallujah, including reports of 21 violent deaths.

The biggest concern with the Lancet study may be its enormous error range: The study said the number of dead from the war (again, excluding Fallujah) could be as low as 8,000, or as high as 194,000.

Les Roberts, epidemiologist at the Johns Hopkins School of Public Health and lead author of the Lancet paper, said that more than 80% of the time, the first two reported deaths in each neighborhood were confirmed by death certificate, and neighbors often independently corroborated the reports.

The Lancet study was dismissed by many critics, especially supporters of the war, because of two quotes that appeared in early press coverage of the report. Dr. Roberts told the Associated Press that he wanted the study to be published before the election (it was published online by the British medical journal Lancet in late October, just a month after the research was completed), so that "both candidates would be forced to pledge to protect civilian lives in Iraq." He also said he was opposed to the war. Meanwhile, Marc E. Garlasco, senior military analyst for Human Rights Watch, told the Washington Post, "These numbers seem to be inflated." Many commentators used that quote to argue that even Human Rights Watch, a group that has criticized U.S. forces' treatment of civilians in the war, didn't believe the study.

But neither quote provided just cause to dismiss the study. Dr. Roberts and co-author Richard Garfield, a professor of nursing at Columbia University who has extensively studied mass killings, say they had never discussed their views on the war before the AP interview appeared. Both also say that when the five researchers did discuss their views, they found that they were split on the war's merits.

I've written before about the importance of researcher bias, but in this case there's no indication that the research methods were chosen in a way that would intentionally inflate the numbers. Other researchers have told me and other reporters that the methods were sound; Dr. Roberts has experience measuring death tolls, including in the Congo, and his numbers have been widely accepted in less politicized conflicts. Furthermore, there's a difference between researchers letting their opinions affect their results and letting their results affect their opinions.

As for Mr. Garlasco's quote, he has since said that he hadn't yet read the study when he was interviewed. He told me last week, "I think the Lancet study is very valuable in some ways," and added, "I'm not a statistician. I don't really understand statistics. I try to stay away from numbers as much as possible."