Sometimes I think that there must be a qualifying exam in order to write for Tech Central Station. Fail the exam and you’re in. They seem to have exams in at least physics, economics, statistics, and epidemiology. Tim Worstall, the author of today’s article seems to have failed both the statistics and epidemiology exams.
Worstall is criticizing a recent study published in the Lancet that found very roughly 100,000 excess deaths in Iraq after the invasion, almost all of which were violent. He writes:
At the very least one would have to add The Lancet to that list of mainstream media which are worth 15% (or is it 5% now, the left have never really been any good at numbers) to John Kerry in the polls. What makes it a great deal worse is this, from the findings to the report. In fact, these are the findings in their totality:
“The risk of death was estimated to be 2.5-fold (95% CI 1.6-4.2) higher after the invasion when compared with the pre-invasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we exclude the Falluja data, the risk of death is 1.5-fold (1.1-2.3) higher after the invasion. We estimate that 98 000 more deaths than expected (8000-194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included. The major causes of death before the invasion were myocardial infarction, cerebrovascular accidents, and other chronic disorders whereas after the invasion violence was the primary cause of death. Violent deaths were widespread, reported in 15 of 33 clusters, and were mainly attributed to coalition forces. Most individuals reportedly killed by coalition forces were women and children. The risk of death from violence in the period after the invasion was 58 times higher (95% CI 8.1-419) than in the period before the war.”
Have a look at those confidence levels. Yup, 95%. That is, a one in twenty chance that the effect simply does not exist.
No, a 95% confidence interval does not mean that there is a 95% chance that the true value is in the interval. And even if it did, Worstall would still be wrong, since if there was a 95% chance that the true value was in 1.1–2.3, there would be a 2.5% chance that it was more than 2.3 and only a 2.5% chance that it was less than 1.1.
It gets worse:
Look at the relative risk ratios (leave out Falluja; I don’t think anyone is really very surprised to see a higher mortality rate there): 1.1-2.3. It isn’t just that it is an absurdly wide one (note, a relative risk ratio of 1 would mean no effect whatsoever) it is that if this paper was written to generally accepted statistical standards it would never have been published. With a 95% confidence level a relative risk ratio of anything less than three is regarded as statistically insignificant.
Actually, by “generally accepted statistical standards” the result is statistically significant at the 95% level since the 95% confidence interval does not include 1. The risk ratio does not have to be three or more to be statistically significant. Worstall is trying to tell his readers that if the death rate increased by a factor of 2.9 (which would be about 300,000 dead bodies), statistics could not detect this increase. Really. (Worstall’s factor of three rule is probably a confused version of the GEP scam run by Philip Morris to try to show that cigarette smoke was harmless.)
Worstall goes on:
Just to clarify that, by “insignificant” no one is stating that it is not important to those people who undoubtedly have been killed during the War. What is being said is that we don’t have enough information to be able to say anything meaningful about it. “Statistically insignificant” means “we don’t know”.
The result was, in fact, statistically significant, so we can be reasonably confident that the death rate in Iraq went up. The best estimate (not including Falluja) is 100,000 extra deaths, but the confidence interval is wide, so the estimate should be treated with caution.
Of course, it is obvious why Worstall’s article was published despite his ignorance of basic statistics—Tech Central Station is campaigning for Bush and wants to deny that the invasion of Iraq has had bad consequences.
Update: Daniel Davies has a comprehensive take down of more defective critiques of the Lancet study, including Worstall’s.