Michael Spagat is back with another attack on the Lancet study. Most of it is stuff we've seen before, like absurd assumptions he makes for Main Street Bias, and the false claim that Soros funded the study. But there is some new stuff, including this (L2 is the second Lancet study):
The above graphic shows results from three mortality surveys. The first is the Kosovo study of Paul Spiegel and Peter Salama: War and Mortality in Kosovo, 1998-99: an epidemiological testimony", published in the Lancet in 2000. This paper is cited in L1, L2 and the MIT paper that is a companion piece to L2. This is, thus, a paper that the L2 authors know well.
There was an exchange of letters in the Lancet of January 13, 2007. The letter of Deberati Guha-Sapir, Olivier Degomme and Jon Pedersen questioned the L2 finding that roughly 90% of all excess deaths in Iraq were violent, contrary to findings in other war studies such as those done on the DRC. The L2 authors responded:
"We feel a better comparison would be to the data collected during that war which showed that 1.8% of the 19.9 million people in the eastern part of the country died of violence in the first 33 months of the conflict, a proportion similar to that measured in Iraq."
To back up this claim they cite a study of the Democratic Republic of Congo (DRC) done by Les Roberts and other people called "Mortality in Eastern Democratic Republic of Congo: Results from Eleven Mortality Surveys." This is the second point in the above graphic.
The third and final data point is L2 itself.
What is highly suspicious is that the three studies are in near-perfect alignment. A regression line drawn through them has an Rsquared of 0.9996. One could make slightly different assumption and feed in slightly different numbers but under any plausible scenario the fit is nearly perfect with an Rsquared of at least 0.99. All of these studies have quite large confidence intervals so the chances of their central estimates lining up so well would appear to be virtually zero.
The Kosovo and DRC studies were in the literature for a number of years before L2 was done. Draw a line between these first two central estimates and the slope suggest that an additional 15 months of conflict will result in deaths of an additional 1% of the population. Extending the line, the 8 months by which the L2 period exceeds the DRC period would bring the total percent killed during the L2 period to just over 2.3. The fact that the L2 authors cite the DRC study as being similar to L2 in terms of the number of months and percent of population killed and the fact that the L2 authors are well aware of the Kosovo study reinforces the relevance of the graph.48
My conclusion is that this graphic suggests data falsification by at least one of the L2 authors.
Well, the graph does suggest that there has been some data manipulation going on, but the manipulators are not the L2 authors.
The first thing to notice is that although the L2 authors said the DRC survey covered "first 33 months", Spagat's graph has it as covering 32 months. Oddly enough, this helps the three points to line up better.
Second, these three surveys aren't the only ones ever to have been done. The most glaring omission is the first Lancet survey. I mean if you were trying to extrapolate deaths in Iraq, wouldn't you start with a survey of Iraq rather than the Congo? And while the two papers containing the numbers plotted above were cited in L2, the numbers were not, while those for Darfur were. Bosnia was also mentioned. I've plotted those extra points and drawn some more lines in the graph below. There are two points for L1 -- excess deaths and violent deaths, and four points for L2 -- excess and violent deaths times uncertainty whether the population should be taken as 26 million or 27 million.
With all those points and lines, it's not hard to find two points that line up with one of the four possible points of L2. (In fact, as well as the ones Spagat shows, L1 and DRC also work). Once you've done this, all you have to is erase all the other points and lines to wind up with Spagat's graph. Needless to say, doing this is dishonest cherry picking, especially when you are doing it to accuse researchers of fraud.
Spagat says
"This graphic was passed to me by researchers who asked to remain anonymous."
And no wonder!
Since Spagat didn't produce the deceitful graphic, he isn't guilty of fraud, just of incompetence. As is David Kane, who describes Spagat's paper as a "tour de force".
Unbelievable! Who could possibly look at a 0.99 r-squared and not look for what was wrong with that?
What readership is Spagat's writing aimed at?
Spagat is insane.
please take a detailed look at the paper guys. start with section one "ethics" problems. hwere he can t decide whether they recorded TOO MUCH or TOO LITTLE information.
one of the last sentences of the paper reads:
I conclude that there should be a formal investigation of the second Lancet survey of mortality in Iraq.
ever heard of anything like that?
"I conclude that there should be a formal investigation of the second Lancet survey of mortality in Iraq."
as long as we don't make a formal official survey of the mortality, though. that would ruin the whole war.
He appears to use careful wording to avoid technically libeling one or more authors.
Spagat
What's the difference between this and saying "I just made up this graphic from thin air"? Who are these invisible "researchers", and what do they do anyway?
Well, at least Spagat put his cards on the table. He made a bold claim, and is trying to substantiate it.
I just skimmed through the paper and my impression is that despite much fanfarre, it does not deliver what it purports to do.
Tim's point about the graphic is very well taken. And since Spagat admits that the graph is his "main exhibit" for the falsification/fabrication claim, it does not bode well for the rest of his arguments.
But I would be very interested in following the discussion of some of its other main points, especially the parts where a comparison of L2 and other pieces of research about mortality in Iraq is attempted.
Tim wrote:
Hmmm. I guess it could look like a tour de force to someone who doesn't know how to calculate a crude mortality rate.
Well Kane he wasn't too far out. It's a tour de farce.
However I think Tim is being too kind in granting Spagat a fool's pardon. (Kane is of course fully entitled to one; "Others abide our question, thou art free.") I'm quite sure Spagat wouldn't let his students away with this kind of thing, but it will go down a treat with his target market - hacks like Fumento and Munro.
1) Thanks to Tim for the link. It sure would be helpful if he were to start new threads for other Lancet related work so as to provide a forum for discussion. Robert, for example, may not be impressed with Spagat and me, but what does he say to Debarati Guha-Sapir and Olivier Degomme (pdf)?
2) Unsurprisingly (!?), I agree with Tim that this is the weakest part of Spagat's paper, and I have told him so. But, you see, we Lancet critics are not monolithic in our opinions.
3) I still argue that the paper is a tour de force. But I'll leave it to Tim to try to respond to the evisceration of the L2 authors' many conflicting accounts of the sampling plan. Instead of picking on the weakest part of the paper, why not attack the strongest part? If you are really interested in the truth, that's what you do. In fact, that's why I read Deltoid!
4) The world would be a better place if Tim paid Spagat (and me!) the compliment of taking us seriously, as we do to him. Is it really your hypothesis that Spagat gathered all that data, produced the graph that you show above and then erased the points/lines he didn't like? Give me a break. That is just stupid. It is fine to claim that the graphic is stupid, that it doesn't provide any evidence for Spagat's claim and so on. I half think this myself. But to claim that Spagat has cherry-picked things, when you have no evidence that he did, hardly helps the discussion.
5) The real problem with the graphic, as far as I am concerned, is that it does not fit my theory of the crime. I think that there is a good chance that the interviewers sampled houses with deaths preferentially (or just made up data) rather than doing the random survey they were supposed to do. If that is what happened, then, obviously, the fact that the deaths line up or don't line up on some graph is random and/or irrelevant.
6) Not withstanding that, it looks to me like the graph does use 33 months and that only the label is wrong. Also, does your L1 datapoint include Falluja? It ought to. (And surely Spagat, if he redraws the graph, gets to make that call.) And, I think, doing so would make things line up fairly nicely.
7) "those for Darfur were. Bosnia was also mentioned." Can you give more details on this? What studies are you using for those numbers? Spagat is claiming (and this would all be more productive if you made this more clear) that we can be certain that the L2 authors knew about the studies he mentions. It is fair for you to note that they surely knew about L1 as well. But how do you know that they also knew about the (unnamed) studies that you are using to add the Dafur and Bosnia datapoints? For all I know, they might have. But we need to know what studies you are referring to (and their dates of publication!) to evaluate this.
8) And charming Robert Chung! Here is a question for you: Would you also claim that Mark J. van der Laan (pdf) does not know how to calculate a confidence interval?
David Kane:
But of course! Maybe God Almighty Himself delivered the graphic to Spagat in a dream. Or maybe Spagat is channeling the souls of researchers past to get the graphic. Maybe he got it from aliens. We don't know.
Two topics, offered in all seriousness, for Tim.
1) Two years ago, you wrote that "I would like to see a more complete description of the actual [sampling] procedure followed ..." Did you ever get that "complete description?" Could you share it with us? You can be sure that the Lancet authors wrote down their sampling plan before they started the survey. How about asking Roberts for a copy? Future historians will thank you. Do you disagree with any of the factual claims made by Spagat on pages 9-14? Could you clarify for all of us exactly how the sampling was done in L2?
2) You claim that Spagat et al make "absurd assumptions" in their Main Street Bias paper. Well, the paper is about to be published in the Journal of Peace Research. Why not write to the editors and tell them that this work does not belong in the peer-reviewed scientific literature?
So the previous 8 topics weren't serious at all! I see!
3) The "Main Street Bias" stuff isn't new and has been discussed already. Spagat is trying to make a mountain out of the molehill of an oversimplified description of the sampling procedure in the paper.
4) If you take me seriously why didn't you read all of my post? I didn't say that Spagat produced the deceitful graphic, but whoever did so cherry picked.
6) Spagat's graph uses 32 months. I measured it. The L1 points I plotted exclude Falluja. Adding two more points for L1-including-Falluja would let you draw even more lines and give you even more chances of finding a three-in-a-row.
7) The Darfur numbers were quoted in L2:
>Recent estimates are that 200 000 people have died in Darfur over the past 31 months. 3
The Bosnia number comes from L2 reference 34 (Coghlan et al)
3) MSB is not new, but this is the first extensive public discussion of the authors constantly shifting stories of what happened. So, just for starters. Were all streets included in the sample (including back alleys) or just streets which intersected main streets? Also, cluster 33 provide pretty compelling evidence that Main Street Bias is a problem.
4) I misread the title "Spagat's cherry picked graph" to imply that you thought Spagat did the cherry picking. I retract my complaint on this point.
6) Now that I check the cite (thanks for the link), I read:
So, isn't 32 months correct? Surely, you don't blame Spagat for not repeating mistakes in L2?
As long as we are having fun with graphics, I think that adding a spot for L1-with-Falluja would be helpful.
7) Where does the denominator come from for Darfur? Can you give more details on 34? Does that article itself include a percentage of population estimate or did you calculate it? I knew that the Bosnian war was bloody, but I had no idea that it killed 5% of the population.
David, Spagat's main street bias paper is an analysis of bias in identifying dead bodies using a particular sampling plan, not the bias in identifying a history of deaths within a household. If it were the latter, it would not say things like:
There is no sense in which the time of day at which a death occurs is relevant to the survey carried out by Burnham et al, since they ask about deaths in the household that occurred at any time, including before the invasion. They didn't go wandering around the streets counting dead bodies, which is the survey they would have to have done for MSB to be relevant.
This subtle difference - between the paper Spagat claims to be analysing, and the actual paper - is kind of important. It probably explains why their article on bias in epidemiological studies of conflict mortality is not going to be published in an epidemiology journal.
In order to properly conduct their analysis of main street bias, Spagat et al would have to have done a monte carlo simulation on an actual or modelled grid, using random samples on large numbers of simulated data, to show that a random sampling scheme of the sort described in the paper actually produces bias. They have the computer time, Johnson is a physicist and probably used to doing simulations on 2-dimensional grids, judging by the code in his paper. Why didn't they?
Of course, I would expect no better from a researcher (Spagat) who calculated an R-2 statistic for 3 data points, each of which was drawn from a different sampling distribution. That's pure genius, that is.
SG, in his 2nd last paragraph in #15, has nailed Spagat. SG's point wrt the three differentially sampled points essentially crucifies Spagat.
3) They've already said that the description in the paper was oversimplified. It is wrong to claim that this amounts to "constantly shifting stories".
6) The 32 month figure in the introduction seems to be an error, since elsewhere they have 33 months. If the contention is that they used this number to concoct the Lancet number, the correct number to use is the one that the L2 authors believed.
7) For both Darfur and Bosnia I looked up the population figures on Wikipedia. Other estimates of the death toll in Bosnia are lower (about 2%), but I used the one I found in the paper cited by L2.
I should also note that the DRC study was published by the International Rescue Committee, which gets funding from Soros, so by Neil Munro logic, it's not to be trusted.
David Kane asked:
Not at all. I've said that there are legitimate criticisms of the Roberts and Burnham pieces. What I've been saying is that your criticism isn't one of them. OTOH, I think Mark's criticism about death certificates is off.
David Kane wondered:
I hadn't previously read that particular working paper so thanks for pointing it out. I wouldn't have used the ILCS or the IBC for triangulation but other than that I think their comments on heterogeneity and migration are pretty similar to my own. And I'm pretty sure that, unlike some, they know how to calculate a crude mortality rate.
Robert, for example, may not be impressed with Spagat and me, but what does he say to Debarati Guha-Sapir and Olivier Degomme (pdf)?
the Guha-Sapir paper is RUBBISH.
http://www.cedat.be/Documents/Working_Papers/CREDWPIraqMortalityJune200…
5 full pages of critisism of the Lancet paper.
a single page on the IBC, with basically NO critisim at all. (missing some deaths in certain governates, but can use ITSELF (!!!) to correct the data..)
then the paper ends with this gem:
We therefore conclude that their Baghdad mortality estimate is close to complete, further corroborated by the ILCS estimates
this claim is simply insane. we are dealing with NEWSPAPER REPORTS in ENGLISH language papers.
here is my challenge: take any month and find out, how many ADDITIONAL deaths IBC does report over the days FOLLOWING an incident. (casualties of attacks who happen to die from their wounds)
then consider: bomb blast injury in a country with a failing medical system. might some people die from their wounds occasionally?
Spagat appears at the end of this short article in the NYT Sunday Magazine. He disagrees with Patrick Ball (who is the focus of the piece) about the number of murders occurring in Colombia. Spagat, as one might guess, thinks the number is smaller than Ball does.
http://www.nytimes.com/2008/02/17/magazine/17wwln-idealab-t.html?ref=ma…
He also appears in this letter of complaint from Human Rights Watch from last year, and lo and behold, he's making all sorts of accusations of methodological flaws on the basis of not very good evidence.
Re :Kane and Spagat's MSB:
Dear David,
What do you think of the sensitivity of the (so-called) main street bias effect to the f estimator in the author's original work?
The paper assumes the result in any case, jodyaberdein, so his f estimate is irrelevant. Spagat's paper showed the level of bias in the death sample if the sampling method of the Lancet papers failed to penetrate into ares of different death rate to the main streets.
It explicitly fails to show that this is the case for the sampling plan used. To do so would require repeated simulations of sampling processes on known grids (even modelled ones would be a start) to show that some regions were vulnerable to undersampling. He clearly didn't do this.