Conflict over conflict-resolution research

Mike Spagat writes:

I hope that this new paper [by Michael Spagat, Andrew Mack, Tara Cooper, and Joakim Kreutz] on serious errors in a paper on conflict mortality published in the British Medical Journal will interest you. For one thing I believe that it is highly teachable. Beyond I think that it's important for the conflict field (if I do say so myself). Another aspect of this is that the BMJ is refusing to recognized that there are any problems with the paper. This seems to be sadly typical behavior of journals when they make mistakes.

Spagat et al's paper begins:

In a much-cited recent article, Obermeyer, Murray, and Gakidou (2008a) examine estimates of wartime fatalities from injuries for thirteen countries. Their analysis poses a major challenge to the battle-death estimating methodology widely used by conflict researchers, engages with the controversy over whether war deaths have been increasing or decreasing in recent decades, and takes the debate over different approaches to battle-death estimation to a new level. In making their assessments, the authors compare war death reports extracted from World Health Organization (WHO) sibling survey data with the battle-death estimates for the same countries from the International Peace Research Institute, Oslo (PRIO). The analysis that leads to these conclusions is not compelling, however. Thus, while the authors argue that the PRIO estimates are too low by a factor of three, their comparison fails to compare like with like. Their assertion that there is "no evidence" to support the PRIO finding that war deaths have recently declined also fails. They ignore war-trend data for the periods after 1994 and before 1955, base their time trends on extrapolations from a biased convenience sample of only thirteen countries, and rely on an estimated constant that is statistically insignificant.

Here they give more background on the controversy. They make a pretty convincing case that many open questions remain before we can rely on survey-based estimates of war deaths. In particular, they very clearly show that the survey-based estimates provide no evidence at all regarding questions of trends in war deaths--the claims of Obermeyer et al. regarding trends were simply based on a statistical error. The jury is still out, I think, on what numbers should be trusted in any particular case.

Here's a summary of the data used by Obermeyer et al.:

This graph is excellent, except that I don't think they should label the axes in thousands! It would be much better to label the numbers directly (1 thousand, 10 thousand, . . ., 10 million). Also I'm surprised to see the low number for Guatemala; I've always heard that 200,000 people died in the civil war there.

Oddly enough, after making this graph, Obermeyer et al. fit a regression on the original scale. As Spagat et al. point out, fit a linear regression on the untransformed data is not such a good idea; here's their graph, which emphasizes that almost all the deaths in these data came from the single case of Vietnam, and, after you take out Vietnam, almost all the deaths came from the single country of Ethiopia:

This incredibly ugly plot indicates a bit of a conflict of goals, I think: On one hand, Spagat et al. are publishing a scholarly article and would like their conclusions to be clear. On the other hand their goal is to shoot down the analysis of Obermeyer et al. and so they have a motivation to make the data look as messy and inconclusive as possible. This second goal seems to have won out in this case.

Here's another ugly graph, this time with so many tick marks on the x-axis that it's impossible to tell which year is which:

On page 939, Spagat et al. write of "'battle-deaths' in 'state-based conflicts'--that is, those in which a government is one of the warring parties. Battle-death counts include deaths of soldiers from combat-related injuries and of civilians caught in the crossfire . . ." I'm not clear on how you count "battle-deaths" in a war like Guatemala's, where the government was one of the warring parties, but most of the deaths were not of soldiers. Also, note that in the above graph, the two different estimates from Guatemala are pretty similar, and both are much lower than the 200,000 figure that is usually stated.

P.S. We last heard from Dr. Spagat when he shot down the notorious article in the Lancet by Burnham et al. that estimated post-invasion deaths in Iraq using a sample survey.

P.P.S. Much more here.