Anjana Ahuja has written an extraordinarily one-sided article attacking the Lancet study. She drags out the same criticisms that were covered in the Nature story, but even though she cites the Nature piece, she carefully avoids mentioning the Lancet authors’ replies, or the opinions of the researchers supporting the study. Ahuja also makes many factual errors, even going as far as claiming that one of the interviewers contradicted Burnham when, in fact, they supported him. All of Ahuja’s errors are in the direction of supportting her case, suggesting that she is biased.
Iraq Body Count, an antiwar web-based charity that monitors news sources, put the civilian death toll for the same period at just under 50,000,
This is untrue. The IBC just counts deaths reported in the media. It is not a count of the total number of deaths.
One critic is Professor Michael Spagat, a statistician from Royal Holloway College, University of London. He and colleagues at Oxford University point to the possibility of “main street bias” — that people living near major thoroughfares are more at risk from car bombs and other urban menaces. Thus, the figures arrived at were likely to exceed the true number. The Lancet study authors initially told The Times that “there was no main street bias” and later amended their reply to “no evidence of a main street bias”.
Spagat is an economist and not a statistician, and has no experience in conducting surveys. His colleagues at Oxford are physicists who also have no experience in conducting surveys. Spagat and co know full well that most of deaths occurred outside the home so it matters little where they live. Spagat et al’s contrived analysis was only able to make the alleged “main street bias” matter by making absurd assumptions like that they only sampled 10% of the population, and that 90% of the population of Iraq virtually never used main streets for travel or shopping.
And don’t you just love the way Ahuja plays “Gotcha!” with the slight change in wording from the authors?
Professor Spagat says the Lancet paper contains misrepresentations of mortality figures suggested by other organisations, an inaccurate graph, the use of the word “casualties” to mean deaths rather than deaths plus injuries, and the perplexing finding that child deaths have fallen.
The authors acknowledged that their graph labeled “casualties” as “deaths” and erroneously compared rates and accumulated counts. So that’s is two errors that Spagat correctly reported, but the next mistakes are Spagat’s:
The word “casualties” does not even appear in the body of the paper. And the study actually found that child deaths increased. In one paragraph Spagat made as many mistakes as he was able to list from the Lancet study.
“The authors ignore contrary evidence, cherry-pick and manipulate supporting evidence and evade inconvenient questions,” contends Professor Spagat, who believes the paper was poorly reviewed. “They published a sampling methodology that can overestimate deaths by a wide margin but respond to criticism by claiming that they did not actually follow the procedures that they stated.” The paper had “no scientific standing”. Did he rule out the possibility of fraud? “No.”
I guess Spagat thinks that they should have got economists and physicists to review the paper instead of statisticians and epidemiologists. I think the reviewers were able to figure out what the sampling methodology was from the paper. Unlike Spagat, who came up with the interpretation that they said that they only sampled 10% of the country and when told that he had misunderstood what they had done accused them of not following the procedures that they has stated. And can we rule out the possibility of fraud in Spagat’s work? No.
If you factor in politics, the heat increases. One of The Lancet authors, Dr Les Roberts, campaigned for a Democrat seat in the US House of Representatives and has spoken out against the war. Dr Richard Horton, editor of the The Lancet is also antiwar.
And The Times, like every single Murdoch paper, was stridently pro-war. This might explain why Ahuja’s article is biased.
Dr Richard Garfield, an American academic who had collaborated with the authors on an earlier study, declined to join this one because he did not think that the risk to the interviewers was justifiable. Together with Professor Hans Rosling and Dr Johan Von Schreeb at the Karolinska Institute in Stockholm, Dr Garfield wrote to The Lancet to insist there must be a “substantial reporting error” because Burnham et al suggest that child deaths had dropped by two thirds since the invasion. The idea that war prevents children dying, Dr Garfield implies, points to something amiss.
No, Burnham et al do not suggest that child deaths dropped since the invasion (they increased), and Garfield did not say that the study suggested that either. Garfield suggested that because the child mortality rate was much lower than from surveys conducted in the 90s, the study had undercounted child deaths. And Ahuja has cherry picked her quotes from Garfield. Here is what he thinks about the accuracy of the study:
I am shocked that it is so high, it is hard to believe, and I do believe it. There is no reasonable way to not conclude that this study is by far the most accurate information now available.
Back to Ahuja’s story:
Professor Rosling told The Times that interviewees may have reported family members as dead to conceal the fact that relatives were in hiding, had fled the country, or had joined the police or militia. Young men can also be associated with several households (as a son, a husband or brother), so the same death might have been reported several times.
However, they were able to produce death certificates, so it is not credible that they invented the deaths. And if you wanted to conceal that someone had joined the militia, why not just not say that they had joined the militia instead of concocting a lie? As for double counting, that is very easy to check against and the researchers made sure that deaths were not counted twice.
Another critic is Dr Madelyn Hsaio-Rei Hicks, of the Institute of Psychiatry in London, who specialises in surveying communities in conflict. In her letter to The Lancet, she pointed out that it was unfeasible for the Iraqi interviewing team to have covered 40 households in a day, as claimed. …
Professor Burnham says the doctors worked in pairs and that interviews “took about 20 minutes”. The journal Nature, however, alleged last week that one of the Iraqi interviewers contradicts this.
Only if by “contradicts” you mean “confirmed”. Here’s the Nature story:
The US authors subsequently said that each team split into two pairs, a workload that is “doable”, says Paul Spiegel, an epidemiologist at the United Nations High Commission for Refugees in Geneva, who carried out similar surveys in Kosovo and Ethiopia. After being asked by Nature whether even this system allowed enough time, author Les Roberts of Johns Hopkins said that the four individuals in a team often worked independently. But an Iraqi researcher involved in the data collection, who asked not to be named because he fears that press attention could make him the target of attacks, told Nature this never happened. Roberts later said that he had been referring to the procedure used in a 2004 mortality survey carried out in Iraq with the same team (L. Roberts et al. Lancet 364, 1857-1864; 2004).
So the Iraqi researcher told Nature that they worked in pairs, which Spiegel says is doable. But Ahuja, after reading that very paragraph, carefully avoids mentioning Spiegel’s opinion, presumably because he is an epidemiologist with experience in such surveys as opposed to the psychiatrist Hicks. And she falsely claims that the Iraqi researcher says that they didn’t work in pairs when they did.
Ahuja’s piece is a disgrace.