Democracy Now has an interview with Les Roberts about his survey. Roberts comments on Bush answering a question about Iraqi deaths:
I guess, politically, he has to downplay this issue, but for him to say a number, that of the eight estimates out there is probably the lowest one, really is not a strategic thing to do in terms of winning hearts and minds in Iraq. Secondly, I'm even more struck that here a year after our study came out, the first time the President has been asked about this was not by a reporter, but by someone from the public when he took a question.
Regular readers will be familiar with the rest of his comments, but there is this extra point: the death rate seems to have increased since the survey was conducted.
Iraqi Body Count has actually found a higher rate per day after than before, and another surveillance system, which has since gone defunct for lack of funding, called the NGO Coordinating Committee in Iraq, found something similar, but as far as us following up on the ground, no, that hasn't happened.
Nature did a study on wikipedia, reported here.
http://www.nature.com/news/index.html
Nothing, unfortunately, about Fumento's use of sock puppets.
Get real, the death rate is far below what was happening when Saddam "Genocide" Hussein was bribing the French, Russians and UN to keep him in power.
Odd that this olitically biased survey gets so much press and yet the deaths of tens of thousands of Europeans in a simple heat wave goes unremarked.
jm, the 100,000 or so excess deaths is the increase over and above what was happening while Sadaam was in power.
How do you get from 'Iraqi Body Count has actually found a higher rate per day after than before' to 'the death rate seems to have increased since the survey was conducted'?
In context, it looks like he is talking about before and after the war, not before and after the survey.
soru
"Odd that this olitically biased survey gets so much press and yet the deaths of tens of thousands of Europeans in a simple heat wave goes unremarked."
Actually, opponents of the Bush administration head-in-the-sand climate change mitigation strategy remark upon it fairly often. Does that satisfy your objection?
Soru: he is talking about before and after the Lancet study was conducted. The IBC soes not measure pre-war deaths at all.
It's a little odd to me that Les Roberts uses the IBC numbers as anything more than an undercount. They almost certainly can't be sampling all sources of violent deaths equally--the Western press is much more likely to hear about murders caused by insurgents than about civilian deaths caused by any other source. So if the death rate has gone up in IBC's count, it tells us mainly about the deaths caused by insurgents, which have definitely gone up. The killings done by pro-government death squads, common criminals, and especially US military action are not likely to be reported as often. We only have a sense of what might be going on from death squads and crime because, for instance, Robert Fisk went to the Baghdad mortuary and found there had been 1000 violent deaths there in July (not counting bombings). I don't know if IBC would include that number. Since I think Baghdad has about 20 percent of Iraq's population, if you assume half of Iraq is that violent, that'd be 2500 deaths per month (not counting bombings). But if I understand IBC's methodology, they couldn't do any sort of extrapolation of that sort and so unless Fisk or someone goes back to the morgue every month, and obtains the new figures, and people do this all over Iraq, a lot of violent deaths are going to be missed by the press.
Theoretical Headline of Tomorrow:
Darfur Body Count Figures Reveal Situation There Not Nearly as Bad as Previously Feared
Z, the Bush admin certainly had nothing to do with a bunch of Europeans going on vacation while letting their parent's die in a heat wave.
As to the Lancet study, its been exposed as a fraud many times already. How about we talk about what is real, like the democratic elections in Iraq?
Hi all;
I'm convinced the Lancet Study is misleading (and is effectively 'biased' upward). Basically the argument goes like this:
(1) Mortality is seasonal (has a 12 month cycle and is higher in some months than others).
(2) The study's time windows on each side of the war are different (they're longer post-war than pre-war).
Therefore;
(3) It's possible that the excess death count can be 'artifically' high or low because variable time windows oversample seasonal mortality peak or troughs before or after the epidemic.
An example - if you double counted winter months after an event in the UK, but not before the event, you'd get a high level of excess mortality. As more people die in winter. The result would be 'correct' in the sense that the excess mortality measured is real. But it's also an artifact of sampling windows being superimposed upon a cycle. It wouldn't make sense to attribute the resulting mortality to the event you based the study around. The result would tell you that excess mortality was higher post event, even though we know mortality in the UK is currently falling.
The above just shows a possible source of error. It doesn't demonstrate that this existed or whether this is biased high or low. I could at this point be about to say the estimate is too low or too high. Which choice I go for depends on the data.
(4) If you look at the data from the study (from the timeseries they give) double counted post-War months have higher mortality that single counted post-War months. This implies that they double counted seasonal peaks post-War - and the estimate is high.
NIkolai,assuming your effect exists it would apply to some subset of the Lancet deaths and not to the 60,000 deaths from violence.
I'm not sure there's any reason to suppose that violence isn't seasonal. Violent deaths and violent crime are seasonal in the UK.
It'd be interesting to try and estimate the magnitude of the effect. But I'm not sure this is possible without the raw data.
"As to the Lancet study, its been exposed as a fraud many times already."
Wow, how did I miss that?
Cross referencing page 19 of
http://www.brookings.edu/dybdocroot/fp/saban/iraq/index.pdf
and Table 1 in
http://www.zmag.org/lancet.pdf
shows an interesting pattern.
All 5 of the most violent provinces post-war were sampled. All but 1 of the 5 least violent were not sampled.
Baghdad: 7
Al Anbar: 1
Salah al Din: 3
Ninawa: 4
Diyala: 2
... (skip 8 provinces of which 5 sampled)
Arbil: 0
Al Muthanna: 0
An Najaf: 0
As Sulaymaniayh: 3
Dahuk: 0
In general, the most peaceful provinces post-war were the most violent and poorest pre-war.
Note that in (As) Sulaymaniay(h), the one sampled low-violence province, measured mortality was, post-war, 3 times lower than pre-war.
Has that apparent pattern been discussed here before?
soru
Maybe non-violent deaths in Iraq follow a seasonal pattern, but is there any reason to suppose that there is a single annual peak? Peaks in both mid-summer and mid-winter would seem more likely, as hot days and cold nights are both likely to kill. Figure 2 of the Lancet report has pre-invasion troughs in spring and autumn, with peaks in summer and winter. It would be interesting to know whether that is typical of such regions. If it is then the pre- and post-invasion periods are roughly comparable: one summer and two winters versus two summers and one winter.
As to the seasonality of violent deaths, the Najaf pilgrimage season certainly stirs things up, which is why Saddam imposed restrictions. Supporters and opponents of Operation Iraqi Freedom ought at least to be able to agree that greater freedom has a price in greater mortality.
Dear soru,
Tim Lambert utlizes a feature called categories on his blog. Please click the one labeled "Lancet study" or some such. I'm sure you will find some number of previous posts where your concerns are addressed. If jm would also have a look, that would be great.
It would be more accurate to say that the Lancet study has been more thoroughly reviewed -- by the competent before publication, and by the incompetent afterward -- than any other epidemiological study in recent history, without a single valid objection having been raised against its methodology or its results.
But you get points for chutzpah, jm!
Supposing that violence is seasonal, I'm having a little trouble understanding how an 18 month sample would produce a violent death rate that is dozens of times higher than a 14 month sample, unless the violent death rate went up by a factor of dozens. You could get a doubling this way, if for instance the most violent month happened to be April. Then since the invasion occurred in March, the prewar period would only contain one April and the longer postwar period would contain two.
I emailed Les Roberts and sent him the post I trackbacked with above. He has responded and I will be doing a follow-up probably after the holidays have settled down. I'm going to have to get more comments from him though, and he wouldn't let me publish any of his answer without me asking specifically on certain points in a reply.
jre claims that, with regard to the Lancet student, "not a single valid objection [has] been raised against its methodology or its results."
Hmmm. Allow me to raise one. The central problem with the Lancet study was that it was conducted by people who, before the war started, were against the war, people who felt that the war was likely to increase civilian casualities and who, therefore, had a expectation/desire (unconscious or otherwise) to find the result that they found.
Consider the Iraqis who did the actual door-to-door surveying. Do you think that they appreciated having such a well paying job? Do you think that they hoped for more such work? If you were them, would you be tempted to shade the results just a little so that the person paying you was happy?
Consider a counter-factual world in which the Pentagon (or the AEI or the Heritage Foundation) did a study with exactly the same methodology, sampling plan and so on. If that study had shown 100,000 fewer deaths, would you have been as quick to believe it?
Again, this objection in no way invalidates the study. But don't pretend that valid objections have not and could not be made.
David Kane,
"The central problem with the Lancet study was that it was conducted by people who, before the war started, were against the war, people who felt that the war was likely to increase civilian casualities and who, therefore, had a expectation/desire (unconscious or otherwise) to find the result that they found.... Again, this objection in no way invalidates the study. But don't pretend that valid objections have not and could not be made."
These statements are based on your personal distaste for the opinions of the Lancet researchers, not their methods or data. This is essentially an argument from authority (or in this case accusations of anti-authority)--a classic logical fallacy--NOT valid reasoning. I am of the "opinion" that the earth is round, and that it's at the center of a heliocentric solar system--both opinions that were shared by Galileo and not the Inquisition. This does not make them invalid.
To make your point you must demonstrate that bias has corrupted the Lancet team's data and/or methods. And, I might add, such a demonstration should be based on a far better knowledge of cluster sampling and statistical nethods than that of Seixon and other commentators here with pro-war "opinions". All the best.
Scott,
I am not sure that I follow your point. You can make it clear to me by explaining what you would do with the counterfactual. What if the Pentagon had used exactly the same methodology as Roberts et al and concluded that 100,000 fewer lives had been lost? Would you have no grounds for being suspicious of this claim? Would your only recourse be to "demonstrate that bias has corrupted" its "data and/or methods"?
Obviously not. By assumption, the methods are identical. And you have no grounds for disputing the data (just as I have no grounds for disputing the Lancet data) beyond a general prior belief that people tend to find what they are looking for (and that employees like to make their employers happy).
Again, the original claim by jre was that "not a single valid objection" has been made. I have falsified that claim. (Note that the Lancet study could still very well be perfectly correct, just as our counterfactual Pentagon one could be. My point is that, just because Tim and others have done great work in debunking many, many stupid objections, does not mean that all the non-stupid ones have been debunked.
I don't mind a bit stepping up to the plate on that one. If the Pentagon had tried honestly and competently to determine the effects of the war on Iraqi mortality, that study would have been entirely worthy of credence. It is precisely because the Pentagon has not attempted to assess the war's effects on civilian deaths that the US has been accused of avoiding its moral obligation to safeguard civilians wherever possible.
So -- you honestly believe that simply poisoning the well by impugning the motives of Roberts et al. qualifies as a valid criticism of the Lancet study's methodology and results? That is in itself a telling comment -- what you are saying, in essence, is that you've got nothing.
David Kane,
"What if the Pentagon had used exactly the same methodology as Roberts et al and concluded that 100,000 fewer lives had been lost? Would you have no grounds for being suspicious of this claim? Would your only recourse be to 'demonstrate that bias has corrupted' its 'data and/or methods'?
"
Of course I would be suspicious. But suspician is not properly presented evidence--if it we're, Joeseph McCarthy would've been the greatest detective of all time and a national hero. And yes, my only professional recourse absolutely would have been to demonstrate that data and methods had been corrupted, and do so in a proper, professional manner that others could replicate. This, David, is what separates science from the Salem Witch Trials, and until you come up with something more substantial than that you haven't falsified anything.
As for the methods being "identical", if they were, the conclusions would have been similar. If there were "no grounds" for disputing the data, it would speak for itself and there would be no grounds for suspicion--the Lancet study would stand refuted. This is exactly what has NOT been done yet by the pro-war faction. All we've been given is "suspicions" and accusations of "liberal bias".
NOT good enough.
"Maybe non-violent deaths in Iraq follow a seasonal pattern, but is there any reason to suppose that there is a single annual peak?"
Not that I'm aware of. All that my suggestion requires is an oversampling of "high" months post-event.
"Supposing that violence is seasonal, I'm having a little trouble understanding how an 18 month sample would produce a violent death rate that is dozens of times higher than a 14 month sample, unless the violent death rate went up by a factor of dozens."
I'm not suggesting that double-counting peaks would have produced a death rate inflated by dozens of times. Just that the result is 'biased' upwards.
For the record my data is:
Pre-Invasion Jan-02 Mar-03 15 months.
Post-Invasion Mar-03 Sep-04 19 months.
AVERAGE DEATHS/MONTH
Pre 3.53 (15mths) Post 4.68 (19mths).
Pre SC 3.33 (9mths) Pre DC 3.83 (6mths).
Post SC 3.00 (5mths) Post DC 5.29 (14mths).
Obviously the time series isn't broken down to the same level as the raw results. You have to use whole months and I included Mar in both pre and post sets.
All that my suggestion requires is an oversampling of "high" months post-event.
According to your figures there was also an oversampling of high months pre-event, albeit less pronounced. However whereas January was a double-counted high month pre-invasion it was a single-counted low month post-invasion.
I don't think there is much sense in trying to identify the effects of the seasons on violent deaths given that there is only one such death in the pre-invasion sample.
1) Scott is in favor of doing things in "a proper, professional manner that others could replicate." So am I. Question: Was the Lancet study done in this way? In particular, is the data available for replication by others? Here is one version of what "replicate" should mean in this context:
http://gking.harvard.edu/replication.shtml
2) jre claims:
That's just delusional. Imagine a world in which the US hires Roberts et al to do their study each month, providing regular updates on Iraqi mortality. Are you claiming that no one would accuse the US of failing to "safeguard civilians wherever possible"? Hah! There would still be plenty of such accusations. In fact, I would bet that there would be more. (Now, that doesn't mean that the US shouldn't hire Roberts et al. Perhaps we should.)
3) jre accuses me of "impugning the motives of Roberts et al." Well, I guess it depends on what you mean by "impugning." I think that the following facts are not in dispute:
a) Roberts et al were against the war before it started.
b) One of the reasons that Roberts et al were against the war was that they believe that wars lead to many more civilian deaths than would occur if there were no war (surely true on average).
c) Roberts et al "rushed" the publication into print because they thought that the findings were important, that other people would/should feel the same and that some of those people were US voters.
If these claims are not right, please correct me.
3) Scott claims that I haven't "falsified anything." Well, I certainly haven't falsified the Lancet study. Indeed, without access to the data (and a better description of the methodology --- see the "Replication Standard" above), it is impossible to do so. There are no faults that I can see in the article itself. But all that I have claimed to do is falsify jre's claim that "not a single valid objection [has] been raised against its [the Lancet study] methodology or its results."
I have hereby raised such an objection. Actually, I have now raised two. (The second one is the lack of data availability for replication, although if someone will kindly point me to the data, I would be happy to retract that objection. I have looked for Roberts' e-mail address to ask him myself but have been unable to find it. Does he have a homepage of some sort?)
You could check the Lancet Article itself to find Robert's email address, since it is right there. Here is a copy at http://www.zmag.org/lancet.pdf. Look on the right bottom where it says "Correspondence to:" Took me all of five seconds to find it.
What is more a mystery is how you can think that merely claiming bias is somehow a valid without demonstrating bias. Down here in Texas, we call that putting the cart before the horse.
First you must show that Robert's antiwar bias corrupted the results of the study. Until you do, what you have is mere speculation. Mere speculation is evidence of nothing.
Let me get this straight, David. You claim that the data is not available for replication, but you clearly have not emailed the authors to ask for it.
As for your other "objection"---if an interviewer makes up data it tends to stand out like a sore thumb when you analyze the results. Apparently you think Roberts is incompentent and wouldn't have noticed such a thing.
Nikolai; there is a bar chart showing deaths by month on p5 of the survey. Eyeballing the chart, there is a bit of seasonality in it, but the treatment effect is still the effect that jumps out and hits you.
"One of the reasons that Roberts et al were against the war was that they believe that wars lead to many more civilian deaths than would occur if there were no war (surely true on average)."
You're claiming that the study is suspicious because the people who conducted it had an intuitively correct grasp of the field? Huh?
As for your other "objection"—if an interviewer makes up data it tends to stand out like a sore thumb when you analyze the results. Apparently you think Roberts is incompentent and wouldn't have noticed such a thing.
I'm remembering something about Les Roberts talking about a certain medical journal publishing a study back in 1991 that contained falsified data in it Nah, couldn't be. ;)
I have had a nice dialog with Roberts through e-mail, which I will continue after the holidays. He has already ignored my underlying point, misread two of my three bulleted points, and acknowledged misreading one of them already. This ought to turn out well.
1) Thanks to Dominion for providing me with Roberts' e-mail address. I am not as good at finding things on the Web as I should be. I have e-mail Professor Roberts (and cc'd Tim) to see if the data is available for replication. I will keep Tim posted on what happens. Perhaps he could then post something to everyone else.
2) Dominion writes "What is more a mystery is how you can think that merely claiming bias is somehow a valid without demonstrating bias." People tend to find what they want/expect to find. Does that surprise you?
3) Tim writes:
Yes. (And I have now e-mailed them.) Many researchers now make their data publicly available to all. A recent famous example is Foote and Goetz.
http://www.bos.frb.org/economic/wp/wp2005/wp0515.htm
It is an evolving academic norm that serious scholars make their data available to all (unless there are good reasons why this is impossible). No one should have to ask Roberts et al for their data. It (and the exact computer programs that they ran to produce the results) should be available on the web for all to see.
4) Tim also writes:
I did not say and do not think that Roberts is incompetent. As Tim knows better than most (!), it is not always easy for most people to know when someone is making up data! The issue of bias is different from the issue of fraud. Do I really need to rehearse for this audience how easy it is for survey results to be misleading?
5) To z, I am suspicious anytime someone finds the effect that he expected/wanted/hoped to find. Call me paranoid.
"People tend to find what they want/expect to find. Does that surprise you? "
Largely because people don't normally search for something they don't expect to find. Fundamentalists don't become paleontologists; likewise, agnostics don't study the Shroud of Turin. Nobody thinks "I have no idea whether or not ketchup will cure glaucoma, therefore I am the perfect person to try it".
Dominion writes "What is more a mystery is how you can think that merely claiming bias is somehow a valid without demonstrating bias." People tend to find what they want/expect to find. Does that surprise you?
Not at all. What surprises me (and what you so carefully ignored) was that you seem to think that the mere mention of bias makes your argument valid. It does not. You have to demonstrate that the bias led to the results led to the data being skewed towards that bias.
Once again, until you do, you have nothing more than speculation. Speculation is evidence of nothing. Merely saying "they are bias" without showing how that bias effected the results of the study is hardly a valid argument!
dsquared - I don't disagree with you. I'm not by any means saying that there isn't an effect from the invasion, just that seasonality will have 'biased' the study's result upwards. I'm aware of the bar chart showing deaths by month, that's what's providing me with the evidence I need to argue that the result is skewed upwards.
The best I can do to get an idea of the size of the effect with the data I have is to estimate excess deaths for the sample based upon the deaths per month figures.
Multiplying the difference between the average deaths per month pre- and post-invasion by the number of months post invasion gives you 21.35 excess deaths (out of 89 post-invasion).
If you do the same, but with a seasonally weighted average (double counted months weighted as 0.5, single counted months as 1), you get 16.74 excess deaths.
So 21% of the excess deaths disappear when you deseasonalise the data. If the effect on the study's result using the studies methodology is the same that knocks the 98,000 estimate down to 77,000.
nik, that's just wrong. The 100,000 figure isn't an annual excess death rate---it's a total for the post-invasion period.
Tim, I think I abbrevated the description of what I did too much. And I think you may have misunderstood me because of that, just to clarify:
I accept the 100,000 figure is the excess deaths for the post-invasion period. The figures I calculated above are also intended to be excess deaths for the post-invasion period.
To do this I calculated the difference between the average monthly deaths pre-invasion and post-invasion. I then multiplied this by the duration of the post-invasion period. So the result I get is excess deaths for the post-invasion period.
I calculated seasonally adjusted results the same way. To remove seasonality I simply weighted each calendar month the same when the average was calculated, so months don't gain additional influence because they're double counted. This should remove the effect of oversampling seasonal peaks or troughs. I then used these adjusted averages to calculate excess deaths for the entire post-invasion period (by multiplying the difference between seasonally adjusted averages by the number of months post invasion).
The method's a very crude approximation of the Lancet study. But I'm looking at total post-invasion excess deaths.
All scientific research is subject to the same criticism.
Ah well, back to burning heretics and getting our description of the physical universe from the locally-favored version of the revealed Word of God.
Seixon: "He has already ignored my underlying point"
You mean, like every other statistically numerate individual you've talked to, he's dismissed your objections to the way in which cluster sampling was applied?
"All scientific research is subject to the same criticism."
Well, technically speaking, we do double blind experiments whenever possible. But in cases like this, it would be hard to design.
You mean, like every other statistically numerate individual you've talked to, he's dismissed your objections to the way in which cluster sampling was applied?
Presumably, back in the day, someone actually addressed his argumants, instead of mocking them?
It seems to me that common sense says his point is correct, that if you randomly or semi-randomly end up with a sample set that includes all the areas of most unrest and none of the areas of highest economic improvement, then it is rather questionable to say that the half way point of the CI is the 'most likely' value (as the author and supporters of the study repeatedly do outside academic contexts where they presumably would get called on it).
Sample Texas, Louisiana, New Mexico, Alabama, Kansas and Maine, would you use the value to place a bet on the outcome of a US election?
soru
soru, yes his point has been addressed again and again and again and again and again. Your point is different from his. Randomly moving certain clusters into a paired Governorate did not oversample violent areas. Seixon even presenting IBC data that showed this but somehow managed to conclude that it showed that the Lancet study was wrong.
"Presumably, back in the day, someone actually addressed his argumants, instead of mocking them?"
I'm still not sure what they are. Originally he insisted it was evidence of "bias" but seemed to be arguing not that it was biased, but that the confidence interval was in fact larger than the paper specified, a possibility that was suggested in the paper itself. He got a resounding "could be" on that. Eventually, he seemed to have dropped the "bias" argument and settled on that; but now, clearly he's back on the bias bandwagon again. But although the terminology is the same, it's not the same argument; it's a new argument towards the same preselected conclusion, the former arguments having been addressed somewhat exhaustively and found less than conclusive.
That's not pathological in itself; it's quite possible to look at a study and get a feeling that something's fishy and worry the thing until you finally get to the bottom of it. But this posing as the lone gunslinger of truth and justice being ostracized by the corrupt powermongers is indeed pathological and tends to take focus away from the substantive arguments themselves, to the point where one wonders which hypothesis is the major point and which merely supporting evidence.
A quick and dirty assessment of Seixon's argument would be this:
So at present, the idea that Seixon's argument gives anyone any very good reason to believe that the true excess deaths number was not positive and in the order 100k, isn't looking good.
The IBC data isn't a good indication of relative violence levels in any case, since violence in places like Baghdad were there are more reporters is more likely to get reported.
He's using IBC data now? That's interesting, though flawed for the reason Tim just mentioned. I'm sort of annoyed with IBC ever since their July analysis of their data came out--they pay lip service to the obvious fact that their methodology undercounts the deaths and never mention the highly probable fact that it provides us with a skewed sample of those that do die (people killed by American forces in areas where reporters can't go are highly unlikely to be counted) and then they go and calculate the percentage of this and that to 3 decimal places, which is fine if you keep in mind that this only applies to the deaths they've counted. But they don't make that clear. They've fallen in love with their data set and can't bear to admit that it's of limited utility.
I thought Seixon's earlier approach, using American troop deaths as a measure of violence levels, was an interesting one. He found that five of six of the paired provinces ended up with the more violent one picked. But I remember looking at the numbers and it didn't make a huge difference. And then when you take into account the fact that Anbar Province was thrown out, apparently the provinces that actually produced the 100,000 figure had almost exactly the same death rate (for US troops) as the one for Iraq as a whole.
Randomly moving certain clusters into a paired Governorate did not oversample violent areas.
So when 1 cluster from a non-violent governate was moved into a violent governate that already had 2, that doesn't oversample the violent area? Do tell.
dsquared,
I am also by no means trying to show that there was not a positive amount of excess deaths from the war - that much is obvious. I have even said that the number might even be 100,000. The only thing I am saying is that this study is horribly imprecise and full of flaws resulting from compromising the methodology in the name of convenience and practicality.
The IBC data isn't a good indication of relative violence levels in any case, since violence in places like Baghdad were there are more reporters is more likely to get reported.
Someone already tried making this argument at my blog. To that I say: do you seriously think that reporting bias has led to Ninawa having 500/M death rate, while Dehuk (its pairing partner) has 0? How likely is that, Mr. Statistics Whiz?
OK, fine. I'm wrting off this thread and declaring it a Seixon zone. Seixon, if you mention the Lancet study in any other thread I will delete your comment.
I will also be strictly enforcing my rules on civility. Seixon, if you make personal attacks on other commentors again I will not edit your post but delete it entirely.
And last point, Seixon, if folks can't be bothered refuting you yet yet again, it doesn't mean that they agree with you.
Ah Lambert, it would help if you actually refuted me for a first time. I asked you two simple questions in my last post, and instead of answer them, you once again try to put me in your lock-box. To think you could have spent those 3 paragraphs answering my questions! Courage!
To that I say: do you seriously think that reporting bias has led to Ninawa having 500/M death rate, while Dehuk (its pairing partner) has 0? How likely is that, Mr. Statistics Whiz?
Really quite likely as Ninawa governorate contains Mosul and Tal Afar, while Dehuk does not have any major city.
replying to comments:
Yes they did; they asked their local representatives which pairings made sense. Can we agree please that the relevant standard is not what "sounds scientific" to you but what might have actually caused a bias in the results. This was a time-saving measure which made the study possible at all, and nothing so far has given any reason to believe it caused a practical problem.
It obviously doesn't and you've admitted in 1) that combining North and South Dakota would not cause problems. There is nothing statistically sacred about the governorate boundaries of Iraq, so the only question here is whether the governorates which were paired are sufficiently similar to allow one to proxy for the other.
No it doesn't. The IBC data, plus everything we know about that dataset (that it has an English-language media bias and that it covers the entire period since the war rather than the period covered by the survey) does not prove anything of the sort. And in any case, in as much as it shows anything, it shows that on average the effect of the pairing process was to sample slightly less violent governorates.
Seixon, you're not really answering any objections here; you're just repeating your own talking points. I'm not surprised that Tim is losing patience with this.
"That is if you have established previously that they are similar, which this study made no attempt to do at all. They only cited a 'belief'. Doesn't sound all that scientific to me."
It also doesn't make any difference to the estimate if they are randomly chosen, although the closer you can pair them the less additional variance will be in your final result. Of course, if you can accurately pair them a priori by the violence level, one might ask why you would need to do the survey.
Seixon continues to claim that Tim Lambert doesn't respond to his points. Yet I see this in another of the four threads devoted to his case:
That was on December 17th. So far no response on that from Seixon.
Really quite likely as Ninawa governorate contains Mosul and Tal Afar, while Dehuk does not have any major city.
Alright, so you think it is likely that Dehuk was underreported at a factor of around 500 deaths/million due to this???
Let me remind some of you that the IBC doesn't only cite press reports, but also mortuary officials and medics. How likely is it that Dehuk actually was similar to Ninawa? Give me a straight answer.
Yes they did; they asked their local representatives which pairings made sense.
Yes, according to previous levels of violence and economic standard. I still haven't gotten anyone to explain to me how in the hell that would relate to mortality during war where most of the casualties were caused by violence/bombing. In that case, I guess Hiroshima was a really poor and violent part of Japan before we dropped the bomb. ;)
This was a time-saving measure which made the study possible at all, and nothing so far has given any reason to believe it caused a practical problem.
Yet the UNDP managed to sample every single province, almost 22,000 households. Time-saving, sure, but that's exactly why the study is so imprecise that it is basically meaningless. As for no reasons to believe it caused practical problems, well of course not - you keep playing down the Achilles heel of their study, that the governates were not similar in war-related deaths as they claimed they were.
It obviously doesn't and you've admitted in 1) that combining North and South Dakota would not cause problems. There is nothing statistically sacred about the governorate boundaries of Iraq, so the only question here is whether the governorates which were paired are sufficiently similar to allow one to proxy for the other.
You're right, there is nothing sacred about the borders of the governates, which proves my point even more. If they had combined pairs before distributing the clusters, then none of this would have been a problem. Of course, that would have meant that they most likely would have had to visit both governates. Damn.
Yes, it would not cause a problem with North and South Dakota in an opinion poll for example, hypothetically. Those two states are very similar in their demographical make-up, as far as I know. In the 2004 election, N. Dakota went 63% for Bush, South Dakota went 60%. If we're doing an opinion poll, those states are so similar that you could choose one or the other and the result would not be affected.
That doesn't work in Iraq, in a time of war, in counting war-related mortality. Especially when two sets of data show that the pairs were not similar at all.
No it doesn't. The IBC data, plus everything we know about that dataset (that it has an English-language media bias and that it covers the entire period since the war rather than the period covered by the survey) does not prove anything of the sort. And in any case, in as much as it shows anything, it shows that on average the effect of the pairing process was to sample slightly less violent governorates.
Denial, such a nice thing. You can keep saying "no, no, no" but that doesn't really change the truth here. You are clinging to this media bias thing, when if you actually read the IBC data, it most frequently cites mortuary officials and medics.
Your last statement is false. In 4 out of 6 pairs, the more violent one was chosen.
Seixon, you're not really answering any objections here; you're just repeating your own talking points. I'm not surprised that Tim is losing patience with this.
Kettle. Black. Something along those lines.
That was on December 17th. So far no response on that from Seixon.
Where's that? I'll have a look around to see if I can find it.
Where's that? I'll have a look around to see if I can find it.
Here, in response to your own comment:
http://scienceblogs.com/deltoid/2005/10/seixon.php/#comment-12399
Seixon,
"Let me remind some of you that the IBC doesn't only cite press reports, but also mortuary officials and medics."
Iraq Body Count presents a partial list of their sources at their web site along with a discussion of how their sources are chosen.
There is no mention here of mortuaries and medics, and an extended perusal of their database revealed no non-media sources to me. Perhaps their media sources gathered information from some, but this is still media reporting in areas where they investigate. They even state quite clearly in their FAQ's (accessible as a flyout window off their main page) that their counts are almost certainly low because they rely only on verified deaths.
In other words according to them their estimates may even be low by a factor of two or more. This would put the true number of deaths well within one standard deviation of the Lancet mean. It follows that their raw counts, useful and valuable as they are, are not a valid comparison to Roberts et al.
Scott,
Read their actual PDF report. About 50% of the deaths had a mortuary or medic as a primary source. Come on, don't be shy, read it. Get informed about what you're trying to talk about.
Also, I am not alleging that the IBC count is precise, in fact I have said many times that it is obvioust that the IBC count is low. That's not what I am using it for. I am using it to show differences between governates since the Lancet study claimed that there were 6 "similar" pairs of governates.
It won't matter about the undercounting because you would expect this to apply generally to all of Iraq, with perhaps a bit more underreporting in rural areas.
The comparisons of the governates with the IBC data show that 3 of the pairs were COMPLETELY dissimilar, while the 3 others are very questionable in similarity as well. Even taking into consideration any inherent biases due to reporting.
You mean like the passage on page 22 which reads:
You display a remarkable grasp of English for someone for whom it isn't their first language but in this case I think you may have made an error in reading.
"Primary source" refers to where the journalists got THEIR information. The inclusion of categories for "friends", "relatives" and "eye-witnesses" might have made that clear to you.
In other words, mortuary figures are included only if they're reported in the English-language media.
Ian,
English is my first language, Norwegian is my second language...
Anyways, my point was that the sourcing of the casualties comes mostly from mortuaries and medics, not simply from journalists. I have already agreed that the IBC numbers will not include all deaths, that is obvious. However, you're side-stepping the point that this would apply generally to all of Iraq, and not enough to a point in the rural areas of Iraq to make up for the vast discrepancies between at least three of the paired governates.
Let's just face the facts: three of the pairings were completely dissimilar.
We can argue about the other three if you wish.
Out of curiousity, has anyone tried to do the same sort of survey that IBC is using Arabic and Kurdish language sources? It strikes me that a. The coverage of Iraq would be better and b. It would be useful to compare the list to IBC.
However, you're side-stepping the point that this would apply generally to all of Iraq
No he isn't; no it wouldn't; you are. The presence of English languge media is not evenly dispersed across Iraq; it's concentrated in big cities like Mosul and not in tiny rural backwaters like Dahuk. Ian as far as I can tell is dead right that when IBC quotes a "mortuary" as a source it means the mortuary reports which appear in the press. They don't have anyone who pops down to the mortuary himself.
Seriously, underreporting would happen all over Iraq. Would it not? That's what I was saying. I also said that yes, this would be more prevalent in rural areas, but not to the point where Dehuk would be underreported by 500 deaths per million.
Unless you are alleging that there is nobody in these other provinces in Iraq, such as Iraqi journalists (who contributed to English language reports) or that rural areas do not have morgues or medics or Coalition soldiers or Iraqi officials or police.
I don't see how you can have someone as a primary source without even having some form of contact with them, lol.
Eli,
That would be quite an interesting experiment, although I doubt Al-Jazeera would make a good source...
And that poinjt was incorrect.
Your underlying argument was that the English-language reporting bias didn't matter because information wasn't coming exclusively from English language media sources.
This was wrong and the fact that those sources attributed the reports to doctors and morgues doesn't alter that.
"Seriously, underreporting would happen all over Iraq. Would it not?"
No - because the western media is clustered largely in Baghdad with lesser numbers in Basrah and Anbar.
"Unless you are alleging that...rural areas do not have morgues"
Most rural areas probably DON'T have morgues.
Additionally, muslim custom requires bodies to be buried within 24 hours of death if at all possible.
In most parts of rural Iraq, it would be impractical to send bosdies to a morgue and then recover the body for burial within that timeframe.
Your underlying argument was that the English-language reporting bias didn't matter because information wasn't coming exclusively from English language media sources.
Ehm, no. My argument was that the source of information wasn't just journalists, but most of it was sourced to Iraqis, such as mortuary officials, medics, government officials, and policemen. These types of people exist all over Iraq, and thus a journalist doesn't have to be where they are to report these types of things since they are not the primary source.
But thanks for trying to rewrite my argument so that you could nail it. ;)
No - because the western media is clustered largely in Baghdad with lesser numbers in Basrah and Anbar.
So are you saying that there was no underreporting in Baghdad, Mosul, etc? If so, you would be at odds with Lambert, and just about anyone else, including IBC. ;)
Most rural areas probably DON'T have morgues.
Medics? Policemen? Government officials? No morgues in any of the provinces I cited? Wow, astounding.
Additionally, muslim custom requires bodies to be buried within 24 hours of death if at all possible.
Well then it's quite a mystery that they even have morgues then, now isn't it? ;)
In most parts of rural Iraq, it would be impractical to send bosdies to a morgue and then recover the body for burial within that timeframe.
That's fine, so now you are left with explaining away medics, government officials, policemen, Coalition soldiers, Iraqi stringers.... the list goes on.
Hack away!
Seixon,
"Read their actual PDF report. About 50% of the deaths had a mortuary or medic as a primary source. Come on, don't be shy, read it. Get informed about what you're trying to talk about."
I did read it Seixon--and it states exactly what I said. Mortuaries, medics, Iraqi officials etc. were all primary sources from which the media outlets used by IBC claim to have gotten their figures. Fact Sheet 1 (page 20) discusses the primary sources that press and media reports have relied on. IBC drew their information from these sources making the primaries second-hand. Which is fine, but it does not make them actual sources that IBC spoke with directly. Which means that ultimately we're dependent on how thorough, and accurate, the media sources used by IBC were in reporting the primaries.
In fact, the report specifically states (Fact Sheet 2 - page 22) that overall, nearly 3/4 of their actual sources used only one of the stated primary sources across all categories, including mortuaries and medics. Though individual categories are mostly high, only one in 5 mortuaries had their reports independently confirmed. Because they constitute the bulk of the primaries, they pulled the overall average down. The same Fact Sheet also states that,
Which again, is exactly what I said, and in fact underlines the point that the bulk of IBC's information comes, for better or worse, directly from the media, and therefore likely overlooks non-reported deaths and the large number of deaths from regions inaccessible to the press, and urban civil sources--exactly as IBC said.
For someone so proud of not being "shy", perhaps you should read your own sources a little more thoroughly.
Last summer Fisk wrote a piece about the murder rate in Baghdad in July--it was about 1000 for that month. The impression given in that article was that these figures aren't in common circulation. Fisk went to the morgue and asked. But in general it doesn't seem to be the case that Western reporters are able to obtain monthly figures from Iraqi morgues. I doubt they can get monthly figures from Iraqi policemen either, or any of the other sources Seixon mentions. I don't doubt that every violent death that occurs in Iraq is witnessed by some Iraqi or many Iraqis, but there's no reason to necessarily think that any individual in Iraq has access to all this information, and certainly no reason to think that the Western press is able to collect it all.
Scott,
So what are you alleging, that Western reporters didn't chase every ambulance they could to rack up civilian death reports? Didn't get information from all over the place the best they could to gather up more death reports? I guess you and me have not been reading the same press reports.
Let's face it, right now we are tap-dancing around uncomfortable facts that it seems no one here wants to admit.
Ninawa: 500/M
Dehuk: 0/M
Can anyone write with a straight face that Dehuk, in reality, is actually similar to that of Ninawa?
Dhi Qar: 660/M
Qadisiyah: 70/M
Can anyone write with a straight face that Qadisiyah, in reality, is actually similar to that of Dhi Qar?
Basrah: 900/M
Missan: 40/M
Can anyone write with a straight face that Missan, in reality, is actually similar to that of Basrah?
As for all this "oh well it's so rural that there isn't anyone to report any deaths there" stuff. Let's have a look at the numbers.
Sulaymaniyah
- population: 1.7M
- death rate: 50/M
Anbar
- population: 1.3M
- death rate: 1730/M
Well golly gosh, the more "rural" governate has about 35 times the reported death rate than the more "urban" governate. How is that possible guys?
Karbala
- population: 0.8M
- death rate: 1170/M
Najaf
- population: 1M
- death rate: 760/M
Golly gosh guys, how did the more "rural" governate end up with a higher reported death rate, even within a pair?
Could it, could it, could it be that they are not similar?
Naw, never.
I see you have left out Dahuk from that little list. Dahuk's largest town (also called Dahuk) is a city of 47,000 inhabitants. That's very small. It would be entirely possible that there was not a single journalist, English or Iraqi (or Kurdish) in a town that size. There aren't any journalists in Cwmbran, for example, which is a town in South Wales the same size as Dahuk.
The death rates in Anbar province is driven by the city of Fallujah and you know it. You may or may not know that Karbala is not more "rural" than an-Najaf, but this is the case. You really are quite profligate in spending credibility that you earn at a much slower rate.
This is quite ludicrous, you know. On the other thread you are ragging on Les Roberts for pairing governorates to carry out his survey (while admitting that this could not have had much effect on the qualitative nature of his conclusions). On this thread, you're throwing around wildly inappropriate and partial data, claiming that it "obviously" shows things about Iraqi governorates that are not only not obvious but probably not true.
Ah yes the International Left-wing journalist's Conspiracy.
Personally I think they actually kill Iraqis themselves just to make the US look bad.
I see nobody took me up on my offer to write with a straight face that those three governates were actually similar.
dsquared,
Of course I left Dehuk off that list, as it is obviously very rural. That doesn't mean one can assume that deaths there have been underreported by 500 deaths per million. Does it? How likely is that? How about the Coalition death rates, showing that there were close to 0 deaths of Coalition soldiers there? Does that have any relevance for you? I guess not.
Also, a journalist does not have to be in a governate to report on something happening there. If that were the case, how in the world have the journalists covered all the deaths happening outside of the Green Zone? Hardly any journalists venture outside of that zone. Perhaps they conducted journalistic work and talked to and contacted people outside of their immediate area?
I guess reporters in New York cannot report on things happening in New Orleans. After all, they aren't there. How can they report on things?
Well, that seems to be the logic you are proposing here, while altogether avoiding the harsh fact that at least three of the pairings were completely fraudulent, no matter how much you want to pretend otherwise.
Ian,
For the nth time, there is no conspiracy of journalists. Just as there is no conspiracy that most Democrats in Congress don't like Bush. Journalists are typically people who bring it upon themselves to right wrongs in the world that they see, and also the entire "if it bleeds, it leads" mantra.
Is it also a conspiracy that most American journalists vote for Democrats? Or is it perhaps the result of the type of personality people who aspire to be journalists have? Or perhaps the type of professors they have had in journalism school, which also overwhelmingly vote Democrat?
It's not a conspiracy, it is just the way things are.
"The way things are" is that there a relatively small number of western journalists in Iraq most of whom are confiend to the Green Zone except for stage-managed trips out under US military supervision.
you keep claiming you don't believe there's a conspiracy then dismiss any and every news report you dislike as the result of a massive and pervading left-wing bias which leads to every single journalist totally ignoring their professional ethics and the possible consequences of being caught lying in order to present false reports.
Just think - if not for those damn left-wing reproteres fabricating the My Lai massacre; the bombing of Cambodia, the Gulf of Tonkin incident, the Pentagon Papers and Watergate, Vietname would undoubtedly have been a glorious victory for the US right.
Ian,
No, Vietnam was destined to be lost from the beginning.
I don't dismiss every news report I dislike, I dismiss any news report that is full of it. And no, it doesn't lead to "every single journalist" ignoring their professional ethics. Reporters have immense power and have the ability to shape people's minds about issues. Why wouldn't they take advantage of that to their own political leaning? They do all the time.
Andrew Gilligan, Eason Jordan, Dan Rather... all had to leave their jobs because they let politics cloud their judgment. For all of them, there are hundreds more journalists who do the same thing, but on issues that aren't as explosive.
Can you explain why the New York Times were against filibusters under Clinton, but were for them under Bush?
I can.
Can you explain why ABCNews were pimping Saddam-bin Laden connections under Clinton, but have disavowed any such thing under Bush?
I can.
Can you explain why Norway's #2 TV channel claimed on the news that the USA had used chemical weapons in Iraq? Even though the Norwegian military has the same weapons (something which conveniently never made it into their story)?
I can.
How many more examples do I have to do to show that there are a good amount of journalists and reporters who are anti-Bush or pro-Democrat??
Here's the link to the Fisk article I mentioned earlier--
http://www.robert-fisk.com/articles528.htm
Reading through the article, it appears the violent death rate in Baghdad went up during the post-invasion months to typically 700-800 per month, at least for the months he mentions, and even higher last summer. 700 per month just for Baghdad would give 17,000 deaths for the first two years, the period for which IBC counts 24,000 for the entire country. I don't know if the murder rate has always been at least 700 per month in Baghdad since March 2003, of course, but even if the average has been lower I doubt IBC has counted more than a fraction of them.
For my next trick, I shall solicit the data on deaths from the ILCS, to add yet another data source to check the similarity of the paired governates. This should appease everyone here who do not like the IBC data since it is based on press reports, although none have taken me up on my challenge to say that any of the governates are actually similar.
Hopefully I will be able to acquire this data from the UNDP or FAFO, and when I do, I will publish my findings as usual. If there was still any doubt...