Sometimes I think that there must be a qualifying exam in order to write for Tech Central Station. Fail the exam and you're in. They seem to have exams in at least physics, economics, statistics, and epidemiology. Tim Worstall, the author of today's article seems to have failed both the statistics and epidemiology exams.
Worstall is criticizing a recent study published in the Lancet that found very roughly 100,000 excess deaths in Iraq after the invasion, almost all of which were violent. He writes:
At the very least one would have to add The Lancet to that list of mainstream media which are worth 15% (or is it 5% now, the left have never really been any good at numbers) to John Kerry in the polls. What makes it a great deal worse is this, from the findings to the report. In fact, these are the findings in their totality:
"The risk of death was estimated to be 2.5-fold (95% CI 1.6-4.2) higher after the invasion when compared with the pre-invasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we exclude the Falluja data, the risk of death is 1.5-fold (1.1-2.3) higher after the invasion. We estimate that 98 000 more deaths than expected (8000-194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included. The major causes of death before the invasion were myocardial infarction, cerebrovascular accidents, and other chronic disorders whereas after the invasion violence was the primary cause of death. Violent deaths were widespread, reported in 15 of 33 clusters, and were mainly attributed to coalition forces. Most individuals reportedly killed by coalition forces were women and children. The risk of death from violence in the period after the invasion was 58 times higher (95% CI 8.1-419) than in the period before the war."Have a look at those confidence levels. Yup, 95%. That is, a one in twenty chance that the effect simply does not exist.
No, a 95% confidence interval does not mean that there is a 95% chance that the true value is in the interval. And even if it did, Worstall would still be wrong, since if there was a 95% chance that the true value was in 1.1--2.3, there would be a 2.5% chance that it was more than 2.3 and only a 2.5% chance that it was less than 1.1.
It gets worse:
Look at the relative risk ratios (leave out Falluja; I don't think anyone is really very surprised to see a higher mortality rate there): 1.1-2.3. It isn't just that it is an absurdly wide one (note, a relative risk ratio of 1 would mean no effect whatsoever) it is that if this paper was written to generally accepted statistical standards it would never have been published. With a 95% confidence level a relative risk ratio of anything less than three is regarded as statistically insignificant.
Actually, by "generally accepted statistical standards" the result is statistically significant at the 95% level since the 95% confidence interval does not include 1. The risk ratio does not have to be three or more to be statistically significant. Worstall is trying to tell his readers that if the death rate increased by a factor of 2.9 (which would be about 300,000 dead bodies), statistics could not detect this increase. Really. (Worstall's factor of three rule is probably a confused version of the GEP scam run by Philip Morris to try to show that cigarette smoke was harmless.)
Worstall goes on:
Just to clarify that, by "insignificant" no one is stating that it is not important to those people who undoubtedly have been killed during the War. What is being said is that we don't have enough information to be able to say anything meaningful about it. "Statistically insignificant" means "we don't know".
The result was, in fact, statistically significant, so we can be reasonably confident that the death rate in Iraq went up. The best estimate (not including Falluja) is 100,000 extra deaths, but the confidence interval is wide, so the estimate should be treated with caution.
Of course, it is obvious why Worstall's article was published despite his ignorance of basic statistics---Tech Central Station is campaigning for Bush and wants to deny that the invasion of Iraq has had bad consequences.
Update: Daniel Davies has a comprehensive take down of more defective critiques of the Lancet study, including Worstall's.
You are, of course, correct Tim, but the one portion of the methodology that should give the reader pause is the sampling method - not the actual size, but where the sample was taken.
I'd be a lot happier with the ability to generalize findings if some interviews had been conducted in other cities scattered throughout Iraq. I don't have a problem with the calculation of the data points, but I do have concerns about the robustness of the dataset WRT sampling bias. Yes, the authors and the editor said it was remarkable that they were able to gather data at all in such conditions [and the authors even expressed concern that these data could be a tail rather than representative], but adverse conditions doesn't make for tighter analysis.
D
Ironically, the absurd implication of Worstall's argument is that if the study found no effect (or a very small one) of the Iraq War on the death rate, it shouldn't have been published.
For example, suppose the study found a RR of 1.05 (CI 0.95 to 1.11). Worstall thinks that this fairly precise estimate should not be published. Why not? Because it suggests little effect of the war!
I don't want to defend the Lancet study (see my blog for why I think it was deeply flawed) but Worstall's rebuttal is literally laughable.
heh, great minds think alike, fools seldom differ. Worstall appears to be strikingly confused, even by the standards of TCS.
Dano: interviews were carried out across Iraq, in a number of towns and villages.
Ragout: I mention a critique similar to yours in my CT post. I don't think it's all that devastating; the authors appear to have taken something like it into account in estimating their confidence intervals.
There are two blogs that totally refute the Lancet report.
http://www.chicagoboyz.net/archives/002543.html
http://www.command-post.org/2_archives/016338.html And take the time to read through the comments sections for some additional information.
Sheesh. Cerebral Flaetus this morning. I have actually read the paper.
I think the hype is stronger than the paper, BTW, but Worstall certainly doesn't know how to read a paper.
His employer's continued attempts to discredit science in general are revealed here when they trotted out poor Tim to critique this paper.
Best,
D
A while back on my blog I posted a few links to Medical
Surveys on torture and falsification of death certificates in Iraq.
There is one small problem Tim, you have not provided a link to the article you criticise.
My error - you did, my sincere apologies
You might find this of interest:
http://mikejericho.blogspot.com/2004/11/100000-deaths-in-iraq.html
As noted in his comment above, Worstall has retracted his statistical claims (though he still does not believe the results). This is such a rare event at TCS that I think it deserves some sort of front-page acknowledgement on the blog.
What's lost in all of this is the fact that the US-Anglo invaders and occupiers have never taken any interest in the carnage and slaughter they have caused in Iraq or Afghanistan because, as author Mark Curtis points out, they are "unpeople" whose lives barely register as a blip on the collective consciousness of those in the west. The website "Iraq Body Count" made one effort after another to get the British government to census the deaths caused by the invasion and time after time they were rebuffed. Some of the statements by the UK government make utterly no sense; for example, on one occasion the Ministry of Defence replied by saying that they "regretted" civilian casualties but that they are "inevitable in such an operation". But what on Earth has this to do with censusing the deaths of those caused by US-British bombs? Nothing.
The fact is that civilians have always been a legitimite target but that we've always been able to sanitize it by arguing that 'military targets' were involved. If one includes Gulf War I, when the entire civilian infrastructure of Iraq was targeted by western bombs, the sanctions regime that resembled a medieval siege that claimed the lives of up to a million people, according to two senior UN officials and agencies, weekly bombing over that same time span, and the most recent agression, then the tally is up to 1.5 million corpses or more from "our side". But our victims don't count, because these are "unpeople". Its ironic that, as teams of investigators search for mass graves of Kurds, Shia etc. who were slaughtered by Saddam Hussein (with full western support during that time) as evidence for war crimes, that no effort will be made to look for mass graves of "our victims" because countries do not investigate their own crimes. If there was any justice in this warped world, criminals like Blair, Bush and their minions should be standing alongside lesser thugs like Suharto, Pinochet, Milosevic and others in the dock at The Hague for crimes committed against humanity. To many it is beyond the pale to suggest that "our leaders" are anything other than "noble servants of the people", and that our nations are "peace-loving democracies" who abide by "international rules of law". These lies need to be challenged.