More on surveys of violent war deaths

Andrew Mack emails me to draw attention to his paper (“Estimating War Deaths: An Arena of Contestation” by Spagat, Mack, Cooper and Kreutz), which criticizes Obermeyer et al’s paper Fifty years of violent war deaths from Vietnam to Bosnia. I commented on Obermeyer et al in this post.

I agree with some of their criticism. The regression that used for correcting PRIO estimates of war deaths is wrong and the conclusion that they drew using this correction — that there is no evidence that war deaths have decreased is unfounded.

I’m not persuaded by their general criticisms of survey measurements of war mortality:

The recall period for the WHO surveys–up to 40 years–was far in excess of recommended practice.

But it doesn’t seem likely that people would forget the death or circumstances of death for a sibling.

small surveys are inappropriate instruments for measuring violent deaths, because most civil wars today tend to concentrate in a few geographically localized areas. In these circumstances cluster surveys tend either to fail to detect any war deaths or–when they do–overestimate their impact by a wide margin.

In the case of an overestimate, it is clear what happened since you get one anomalous cluster (like Falluja in Lancet 1). More likely, you get an undersestimate, but other ways of counting war deaths seem to produce more severe under estimates.

Also commenting on their paper is Andrew Gelman, who was contacted by the first author, Michael Spagat. I wonder why Spagat didn’t contact me?

Comments

  1. #1 Vinny Burgoo
    December 29, 2009

    Do you have any thoughts on the recent Burke et al. paper Warming increases the risk of civil war in Africa. It claims to have found a small but robust correlation between war-deaths recorded for sub-Saharan Africa in the PRIO dataset and country-scale temperature increases in the period 1981-2002 and says that ‘this historical response to temperature suggests a roughly 54% increase in armed conflict incidence by 2030, or an additional 393,000 battle deaths if future wars are as deadly as recent wars.’

    I’m no statistician but …

    The period under review neatly straddles the end of the Cold War, a continent-wide ‘external forcing’ that the study totally ignores. Here’s an annual tally of African conflicts listed in the PRIO database. Is there a clear post-Cold War jump? Perhaps not. But I bet the apparent jump is more ‘robust’ than any correlation with temperature.

  2. #2 Bruce Sharp
    December 29, 2009

    I haven’t had time to read both papers yet, but this looks really interesting. There are some thoughtful comments accompanying [another Gelman article at Statistical Modeling, Causal Inference, and Social Science](http://www.stat.columbia.edu/~cook/movabletype/archives/2009/12/update_on_estim.html).

    The PRIO battle death data looks fascinating. The Obermeyer paper says that the PRIO “compiles and reconciles
    passive reports on violent deaths related to war globally
    from 1900 onward,” but that doesn’t appear to be at all an accurate description, based on the [PRIO data’s documentation](http://www.prio.no/sptrans/1555324504/PRIObd3.0_documentation.pdf).

  3. #3 Mark Shapiro
    December 30, 2009

    “The regression that used . . . ” maybe s/b “The regression that they used . . . “?

    Thanks for keeping this awful stuff in the public eye.

  4. #4 Robert Shone
    December 31, 2009

    A recent paper from Spagat, Johnson, et al (on conflict deaths in Iraq, Afhganistan & Columbia) made the cover story of Nature last week:

    http://www.nature.com/nature/journal/v462/n7275/covers/index.html
    http://dissident93.files.wordpress.com/2009/12/gourley_nature_cover.jpg

  5. #5 Marion Delgado
    December 31, 2009

    Knock on wood, but in Big Lebowski terms, it seems our Donnie on this issue has stopped wandering in and saying “I am the Walrus.”

    However, it also looks like the registration system at pharyngula is causing the insane to spill out all over scienceblogs, the way the crazed homeless went out after Reagan closed the mental health facilities in California.

  6. #6 andrew mack
    January 2, 2010

    Tim

    There is indeed evidence that people misremember past deaths. [See](http://www.ingentaconnect.com/content/apl/spub/2003/00000031/A062s062/art00003)

    Like you, I found it odd that individuals might misremember the death of siblings. But remember that the WHO surveys have recall periods going back 40 odd years. (The SMART protocol states flatly that recall periods should not exceed ONE year because of concerns about recall bias.) Remember too that many of the individuals who were asked about the violent deaths of siblings may well have been children when the wars were being waged. It wouldn’t be so surprising if they misremembered the death of an infant brother or sister.

    I should note though that the recall bias issue is peripheral to our critique of Obermeyer et al.

    On population health surveys and excess death estimate more generally, the Human Security Report Project has a study due out on January 20th which examines the use of surveys to examine all excess war deaths––those from disease and malnutrition as well as violence. It makes the case that, while such surveys are a vitally important source of data in war-affected poor countries, major challenges arise when they are used to estimate excess deaths.

    The challenges lie not so much with surveys themselves, but with how to figure out the baseline mortality rate. Here the need is for pre-war trend data, not just a point estimate. The counterfactual ‘what would have happened had there been no war’ has to be taken into account in estimating excess deaths, but this requires understanding not just the mortality rate in the immediate pre-war period but the trend in mortality over several years.

    Peacetime mortality rates are declining in almost all poor countries where most wars are waged today and if this isn’t taken into account in estimating excess mortality, the death toll will be underestimated.

    But, as our report points out, the practical challenges of “measuring from the slope” of a declining (or increasing) mortality trend line are almost insurmountable.

    Census data can sometimes be used to estimate excess deaths from all causes, but incident reporting of the sort undertaken by PRIO and Uppsala can’t be. There is no way of determining from reported deaths whether those who succumb to disease and malnutrition in wartime wouldn’t have died anyway had there been no war.

    Andrew

  7. #7 Tom
    January 3, 2010

    You folks sure like to think about who died… why did they die… should the memories of survivors count? Right through the New Year, you guys keep thinking for the unwashed, bloody sheep… Help us please with the wounded numbers in all these and more conflicts, would you please? The wounded divided by the dead. In most wars the ratio runs from 3-5 for each person killed. Where can I find these numbers please? In Iraq? In Gaza? In all areas of the world I have watched as the number of wounded has shot up over the years. In “Intifada”(to shake off), 1 & 2; this ratio got to well over twenty to one. To me this says that people are being wounded for sport by soldiers. Is this a war crime if proven to be true? It is not because of improved medical care. In most cases this only applies to troops in the area of operations. Not civilians. So please, do not bring up this as a red-herring. Also why don’t the type of wounds count in your reports. How may shot in the hand? Arm? Head? Center of mass? Leg? You all surly see how important this data will be, we all can… Where can I get that info? Thanks to all.

  8. #8 David Kane
    January 3, 2010

    Gelman writes:

    We last heard from Dr. Spagat when he shot down the notorious article in the Lancet by Burnham et al. that estimated post-invasion deaths in Iraq using a sample survey.

    Tim: Would you agree with Gelman’s assessment? If not, you should explain where he goes wrong. I, of course, agree with Gelman that Burnham’s article was “notorious” and that Spagat succeeded in shooting it down.

  9. #9 Dannyd
    January 3, 2010

    Tim’s claim that you’ll always know of an overestimate because of one obvious cluster seems like a baseless claim, to put it mildly.

  10. #10 andrew mack
    January 3, 2010

    Tom

    I don’t think there any good answers to your question.

    Research organisations like the International Peace Research Institute in Oslo and Uppsala University’s Conflict data program don’t try and collect injury data because they believe the reports are just too unreliable. Getting good data on fatalities is hard enough. Plus what constitutes an “injury” can vary dramatically from place to place. A minor wound that didn’t prevent an individual continuing in a combat role might counted as an injury in some contexts — not in others. But there is little room for ambiguity when someone is killed.

    One of the few serious studies we know of notes this:

    “The ratio of the number of people wounded to the number killed ranged from 1.9 to 27.8. Two additional articles gave the proportions of people wounded who eventually died in major conflicts since 1940, without giving absolute numbers. Total deaths were never more than 26% of all casualties, a wounded to killed ratio of 2.8.”

    This is from a lit. review in 1999 by Robin Coupland and David Meddings, both then at UNHCR. See:http://www.ncbi.nlm.nih.gov/pmc/articles/PMC28193/

    I think that there is actually quite a lot of evidence that the ratio of injuries to deaths is much higher in modern armed forces like those of the US because excellent first aid facilities prevent soldiers from dying who would certainly have died in earlier periods of warfare.

    In most poor country wars people bleed to death from what would otherwise be non-fatal wounds because there are no medics to stop the bleeding, or medevacs to get them quickly to a field hospital. So here the ratio of deaths to injuries is low.

    I think quick access to medical services would likely also explain the Palestine case where political violence takes place in urban settings––as against remote jungles as is the case in many conflicts in the developing world. Civilians get treated as well as fighters here. Moreover Palestinians have health services that are far superior than those in most wars in Africa and elsewhere––and lots of practice in using those services to treat victims of violence.

    Andrew

  11. #11 anon
    January 3, 2010

    David Kane: “I, of course, agree with Gelman that Burnham’s article was “notorious” and that Spagat succeeded in shooting it down.”

    Here’s what appears to me to be the most relevant quote from Gelman at the link: “P.P.S. A reporter called me about this stuff a couple months ago, but I’m embarrassed to say that I offered nothing conclusive, beyond the statement that these studies are hard to do, and for some reason it’s often hard to get information from survey organizations about what goes on within primary sampling units.”

    I feel this shows David Kane to be a person of dubious character and that Andrew Gelman may be prone to loose language on occasion. I’ve no doubt whatsoever that Kane can take Gelman’s comments far further than Gelman has and far further than he would want them taken. That’s what people like David Kane do, the devil finding work for those with idle minds etc etc.

  12. #12 Bruce Sharp
    January 3, 2010

    Anon, I don’t think it’s fair to criticize David, when he’s quoting Gelman accurately. But I think you’re probably right about Gelman, since his “shot down” link goes to a [March 2008 post](http://www.stat.columbia.edu/~cook/movabletype/archives/2008/03/ethical_and_dat.html) in which he admits that he hadn’t read Spagat’s paper. Presumably he’s read it since then, and agrees with it… but if so, he ought to have said something to that effect.

  13. #13 Tim Lambert
    January 4, 2010

    DK: “Would you agree with Gelman’s assessment? If not, you should explain where he goes wrong”

    I’ve already explained, in detail, where Spagat goes wrong. One of the posts is even linked in my post above. Gelman has not offered any assessment of the Lancet studies.

  14. #14 Marion Delgado
    January 4, 2010

    Alas, we didn’t make it through the movie before the child started speaking.

  15. #15 Robert Shone
    January 5, 2010

    I recall Les Roberts, Gilbert Burnham and Medialens (who were, IMO, Roberts’s inept proxy for attacking Iraq Body Count) touting the false claim (wrt the Obermeyer study) that:

    “But a study of 13 war-affected countries presented at a recent Harvard conference found that more than 80 percent of violent deaths in conflicts go unreported by the press and governments.”

    http://web.mit.edu/humancostiraq/further-reading/burnham-roberts2007.html

    http://tinyurl.com/yfdh3l2

    http://www.medialens.org/forum/viewtopic.php?t=2613

  16. #16 Jeff Harvey
    January 5, 2010

    Robert,

    Medialens were the “inept proxy” in your frankly inept opinion. I think Medialens did a great job explaining exactly why the mainstream (= corporate) media downplayed studies reporting the horrific toll of the illegal invasion of Iraq; had the Lancet studies been based on body counts as a result of the military activities of “officially designated enemies”, and had these had been juxtaposed with those of Iraq Body Count, then I am sure the media would have focused on the larger toll for purely propaganda purposes. The bottom line is that the western media constantly downplay or ignore crimes committed by ‘us’ and our proxies whilst focusing laser like on crimes committed by official enemies. This is hardly suprising, since the msm is generally owned by commerical elites who clearly see the media as a means of promoting elite explanantions.

    The IBC toll of the carnage in Iraq – almost certainly way too low – was only given full atterntion by the msm when the Lancet studies were published. The reason for this should have been obvious; in public relations terms the media were desperate to ‘manage the outrage’ that may have resulted had it been made clear that the real death toll of the Iraq war was many hundreds of thousands or even more than a million. Suddenly IBC came in from the cold because 30,000 deaths sounded ‘acceptable’ whereas 600,000 plus did not. What I found most annoying about IBC was that, when Bush and Blair began to downplay the death toll in Iraq using IBC figures, IBCs response to this abuse was apparent silence. They should have spoken up and argued that the war was illegal and that the death toll was horrific and that the war parties should not be using their own figures to defend the war. But I did not see that happen. The war party was allowed to abuse IBC figures to downplay the human toll of the war. The media went along for the ride.

    I appreciate the enrormous efforts made by David Edwards and David Cromwell and Medialens to expose the media’s cupability as ‘vital cogs in the machinery of industrial western killing’ as well as the hypocrisy of the IBC. As far as I am concerned, Robert, you should be expending more effort in exploring the real reasons for US military adventures and occupations in Asia and the Persian Gulf and its human toll, instead of attacking those who expose western actions and the actual motivations behind them (e.g. imperial expansion and efforts to prevent the rise of ‘near-peers’). For that I salute Medialens, Pepe Escobar, William Engdahl, Paul Street, Tom Engelhardt, Andrew Bacevich, Noam Chomsky, John Pilger, Robert Fisk, and others who are saying it like it is.

  17. #17 Robert Shone
    January 5, 2010

    Thanks for the lecture, Jeff. I too “salute” Noam Chomsky and Robert Fisk (or rather I greatly admire their work) but, unlike Medialens, they’re generally careful with the facts, and they don’t conduct smear campaigns.

    If you’re interested in why I find Medialens inept on this topic (I doubt, somehow, that you are), see my ZNet article which lists their many errors on IBC:

    http://www.zmag.org/znet/viewArticle/22309

    See also the following piece on their heavy-handed censorship, which also mentions how ZNet recently refused to publish their attempt to smear George Monbiot:

    http://dissident93.wordpress.com/2010/01/04/medialens-embarrassing-archive-08/

  18. #18 Jeff Harvey
    January 5, 2010

    Robert,

    Methinks you are resorting to the word “smear” too easily here. Medialens were annoyed with IBC for the same reason that I alluded to above. IBC appeared unconcerned that their estimates of deaths in Iraq were being (ab)used by members of the war party to downplay the carnage. Bush was asked what he thought the death toll of the Iraq invasion was (back in December 2005 or 2006 I think it was) and he responded with the 30,000 estimate that appeared to come straight from the pages of IBC. The next day I expected those at IBC to respond angrily, accusing Bush of misinterpreting the intentions of their survey. Did they? Did they hell. Their only response was silence, as if they were pleased to see the msm and western leaders who waged or supported the war citing their work. Their silence spoke volumes in my opinion.

    You also appear unconcerned at the much more egregious smear campaign that was launched at Roberts, Burnham, Gilbert et al. after the publication of their Lancet studies. Given the results of these studies (subsequently supported by the Opinion Business Research survey) apparently showed western culpability in mass murder, a finding which contradicts the carefully cultivated image of our basic benevolence, it was hardly surprising that the corporate media did everything it could to marginalize them.

    But I digress. As I said above, given the underlying actual causes of the conflicts in Iraq and Af-Pak, I am bemused as to why you expend such efforets at defending IBC. Fiddling while Rome burns is my take on it. As for Medialens, I admire their tireless efforts to expose the msm for what it is: subservient to commerical elites and powerful vested interests. Their two books, “Guardians of Power” and “Newspeak in the 21st Century” are outstanding reads. They have criticized George Monbiot in the past for some of his views, and good for them. I admire Monbiot but the so-called “liberal” media, which is hardly liberal as far as I am concerned, needs to be held to account. We know where the right wing press stands, but as David Edwards and David Cromwell have shown, it is the allegedly liberal press which appears equally beholden to power. Along with the likes of Robert McChesney, Richard Falk, Fairness in Accuracy and Reporting, Democracy Now and other outlets, it is good to see those media sources apparently championing liberal causes to be held to account.

  19. #19 Robert Shone
    January 5, 2010

    “Apparent silence”, “appeared unconcerned”, “fiddling while Rome burns” – the way Jeff Harvey frames his “criticisms” of IBC reminds me of Frank Luntz’s manual – or perhaps of witch-hunts from earlier times.

    Yes, of course, one has to be really “unconcerned” to devote nearly seven years of one’s life, as a volunteer, to corroborating and documenting violent deaths. It takes so much more concern and integrity to lecture people on web forums about how they should spend their time, and on what premises they must accept to avoid the accusation of “being silent”. Of course, of course.

  20. #20 Jeff Harvey
    January 5, 2010

    Robert,

    Your rather shallow posts shed a considerable amount of light on the way that you think. All of the gibberish you wrote in post 19 could be equally (if not more so) applied to people like Les Roberts and Gilbert Burhnam who are both professional epidemiologists. Yet I have never detected a scintilla of concern from you over the way these scholars have been treated and dragged through the mud. Have they not dedicated their lives to uncovering the scale of the carnage in Iraq? Where are your sympathies for them? After all, the IBC investigators are not professional epidemiologists but (as far as I can tell) have careers in other fields like the arts.

    The IBC team has not, as far as I have seen, responded to the clear abuse of their ‘estimates’ by defenders of the war party, including those who carried it out (e.g. Bush and Blair) nor of the intent of those in the media who routinely cite IBC estimates as ‘facts’ whilst downplaying or ignoring other studies that have produced much higher estimates of death. Care to speculate why the IBC team have remained silent? Or why the corporate/state msm often treat IBC estimates as reliable while ignoring or attacking the Lancet findings? I have my own suspicions. In fact, it should be patently obvious. But I have yet to see you discuss this rather important point on your web cite or elsewhere. Why not, Robert?

    In my opinion you are full of hot air. Lots of bluster but little substance. And especially little when it concerns the Anglo-American *modus operandi* or *casus belli* with respect to the illusory ‘war on terror’ or of the consequences of our ongoing military and economic policies in the Middle East and Caucasus region. Its just the same ‘attack Medialens for daring to criticize IBC’ refrain.

  21. #21 Jeff Harvey
    January 5, 2010

    To give just one example of BBC bias, today their is an on-line discussion asking the question:

    “Can the UK and US prevent extremism in Yemen?”

    Again, all of the usual assumptions are packed into this one question. First of all, it assumes that the intentions of the US and UK governments in the region are purely benign; second, that our governments are anything but extreme in the policies they pursue. In other words, western policies are always packaged by the msm into heart-rending tales of concern for human rights and freedom, irrespective of the actual facts on the ground. Crimes committed by the west are gnerally ignored, or forgotten. The real agendas are seldom, if ever, disclosed. This is the kind of media hypocrisy that Medialens challenges on its web site. And I for one think that they do a fantastic job.

  22. #22 Robert Shone
    January 5, 2010

    Jeff writes:

    Yet I have never detected a scintilla of concern from you over the way these scholars have been treated and dragged through the mud.

    That shows only that you haven’t looked very far. I once highlighted the contrast in BBC coverage between the Tsunami deaths (which BBC covered daily) and Iraqi deaths (eg the Lancet 2004 estimate, which they headlined on one day, but then forgot about). In fact, Medialens (no less) once used my material (correspondence with the head of BBC news) in one of their alerts. I’ve also defended Les Roberts in the past from attacks by rightwing pundits. (That was before Les Roberts ran for Congress, before he started attacking IBC via Medialens, and before he said that the shock-and-awe campaign was “very careful”).

    Not that I should have to defend myself against accusations of “appearing silent” on issues of your choosing. When witchfinder-generals such as yourself start to accuse people of “appearing silent”, you might want to take a look at yourself. (Incidentally, I can tell from your ignorant remarks about IBC that you haven’t spent much time looking at what they’ve written. Clue: they have addressed Medialens’s baseless accusations in full. But Medialens have “appeared silent” on that, which is possibly why you don’t seem to know about it).

  23. #23 Jeff Harvey
    January 5, 2010

    Robert,

    You seem singularly obsessed over this IBC-Medialens issue. Get a life, pal. There’s far more important issues to address in this world. It seems that you have dedicated yourself to the IBC cause. Are they not intelligent enough to take care of themselves?

    I know that IBC have responded to Medialens criticisms. I just do not know why they have not expended so much effort to counter msm articles which cite their estimates for the death toll in Iraq as a means of downplaying the results of other studies that produce far higher estimates. Curious, that one. Or perhaps not.

    Do you not think you are making a mountain out of a molehill? What about all of the other material medialens covers? Or does this IBC thing stick in your craw for some reason?

  24. #24 Robert Shone
    January 5, 2010

    Jeff writes:

    Do you not think you are making a mountain out of a molehill?

    Well, I’m responding to accusations. I guess it depends on whether you consider your own (actually Medialens’s) accusations to be mountains or molehills.

  25. #25 Marion Delgado
    January 6, 2010

    shorter Robert Shone:

    Comment by Robert Shone blocked. [unkill]​[show comment]

  26. #26 Robert Shone
    January 6, 2010

    funnier Marion Delgado:

    http://fascistoar.blogspot.com/

  27. #27 Bruce Sharp
    January 6, 2010

    Ater having read both the OMG paper and the Spagat/Mack/Cooper/Kreutz (SMCK) paper, I’d say that the SMCK paper makes a convincing case that there are significant flaws in OMG. Previously, I didn’t understand at all how OMG arrived at their conclusion that war deaths in the most recent period were higher than they were in the 1965-1974 reporting period. After reading SMCK, I understand how OMG came to that conclusion, and I think their method was just plain wrong.

    Like Tim, however, I have a few minor misgivings. Pointing to the [SMART Protocol](http://www.smartindicators.org/SMART_Protocol_01-27-05.pdf) as though it were some sort of definitive authority on what should and should not be allowed in mortality surveys strikes me as just plain silly. The introduction to the SMART Protocol states that “This manual is designed to be used in conjunction with the accompanying software. NUTRISURVEY for SMART which is free from http://www.nutrisurvey.de/ena/ena.html.” I don’t see why the software manual for a nutritional survey should be regarded as gospel for demographers who are trying to assess war casualties.

    In general, yes, longer periods would lead to greater uncertainty, but a one-year limit is arbitrary and often unrealistic. The Protocol notes that the recall period is inevitably a compromise: “If the recall period is too short, a very large number of households will need to be visited and interviewed, which makes the survey unwieldy.”

    And while I agree that surveillance projects like the Iraq Body Count provide an absolutely essential metric, I’m only in partial agreement with SMCK’s statement that these projects “reveal trends – whether things are getting better or worse – and can do so in near real-time.” They can reveal trends, but they don’t necessarily reveal trends. As SMCK points out in their own report: “Governments may forbid reporting of war deaths – particularly of their own forces. Journalists and other observers are sometimes banned from war zones, as in Sri Lanka in 2009, or may stay away because conditions are simply too dangerous. And in wars with very high daily death tolls – like Iraq in 2005 and 2006 – violent incidents with very small numbers of deaths may go unreported.” Those factors can have significant effects on our perception of the overall trends.

    Regards,
    Bruce

  28. #28 Robert (not Shone)
    January 7, 2010

    Bruce wrote:

    Pointing to the SMART Protocol as though it were some sort of definitive authority on what should and should not be allowed in mortality surveys strikes me as just plain silly. […] I don’t see why the software manual for a nutritional survey should be regarded as gospel for demographers who are trying to assess war casualties.

    Well, I agree that it’s not definitive gospel but I don’t think it’s silly to have good guidelines gathered together in an accessible document. Doing field work under crisis conditions is not a walk in the park so almost any advice you get from people who’ve actually done it helps you avoid egregious errors. I have no particular problem with the SMART protocol if it’s viewed in the context of “best practices under way less than best conditions.” However, like many manuals of the sort, it’s limited in what it can and cannot cover. One of the topics that field manuals just don’t have the room to cover is “how rotten will the estimates be if you don’t follow best practices to the letter?” Of course, you *want* to follow best practices but often in the field you can’t. Thinking that any deviation from best practices vitiates an entire study is the fallacy of the “Aha! I found a critical error” type. You can see lots of evidence of that in this topic. Some errors are far more egregious, and therefore consequential, than others.

Current ye@r *