I haven't commented on Kaplan's shoddy critique of the Lancet because Daniel Davies already demolished it here. Kaplan did have one argument that Davies did not address, so I will deal with that in this post. Kaplan wrote:
The survey team simply could not visit some of the randomly chosen clusters; the roads were blocked off, in some cases by coalition checkpoints. So the team picked other, more accessible areas that had received similar amounts of damage. But it's unclear how they made this calculation. In any case, the detour destroyed the survey's randomness; the results are inherently tainted. In other cases, the team didn't find enough people in a cluster to interview, so they expanded the survey to an adjoining cluster. Again, at that point, the survey was no longer random, and so the results are suspect.
It's quite remarkable how many things that Kaplan got wrong in just one paragraph:
- It is not true that they could not visit some clusters because the roads were blocked off.
- They did not pick other more accessible regions. To reduce travel, they paired governorates with similar violence levels and move all the clusters in the pair into one of the governorates.
- It was not unclear how they did this calculation since the report explains it in excruciating detail.
- The moving was clusters was randomized so that it did not destroy the survey's randomness.
- It is not true that they expanded the survey to an adjoining clusters if there weren't enough people in a cluster. The sampling unit was households, not people. And a cluster was the thirty households that were nearest a randomly chosen location.
The last mistake is particularly egregious. The first four seem to have resulted from a bizarre misreading of the study's Governorate pairing procedure, but the last one seems to have been made up out of whole cloth.
Out of curiosity, do you believe that the survey's Fallujah numbers were total garbage, or just an outlier that might have been a bit off...but probably pretty reasonable?
They were most likely a correct sample of a cluster of households that had experienced unusually heavy violence.
Has the author of this site seen the latest articles from the authors of the Lancet study, released online last week, answering some of the criticisms of the report? I read them on medialens a few days ago. The strange thing is, the authors of the Lancet study seem a great deal less certain about their conclusions than most people who are ferociously arguing in favour of every tiny detail, such as yourself. That was the impression I got, anyway.
I located the article to which I believe you are referring
here.
(click in the center pane and scroll about halfway down, or search for "Roberts" or "Garfield").
It was not my impression at all that Roberts or Garfield seemed uncertain about their results. Rather, they seemed comfortable with the decisions they had made (e.g. fewer clusters means a wider confidence band, but also less risk to field staff), and very direct and unembarrassed in discussing the limitations of the methodology. Roberts was particularly undefensive in responding to the
statement by the PM's spokesman.
Contrast Roberts' and Garfield's unruffled manner with some of the superheated criticism directed against them -- that should tell you something right there.
No I haven't; could you provide a link, please, as the JHU site doesn't appear to have them, and I can't find anything on medialens either.
Neither Tim nor myself have argued in favour of "every tiny detail", much less claimed to be "certain about their conclusions". We've simply argued, at times forcefully, that their conclusions were reached by sound statistical processes, and that (so far) every critique except one has been very ill-founded.
A direct link is below. The first is the editorial that came with the original report, unless it's been updated. And below is the response to the government and other critics. I thought what came across is how haphazard the whole process is. Certainly not the bible of truth approach that some have taken to this report.
http://members5.boardhost.com/medialens/msg/205117.html
There is nothing that I see in the 2 medialens pieces that indicates the authors are less certain of their conclusions than the ferocious dogs that snarl at every enabler and wargasmer that walks by. I don't even hear a chain rattling from the dog picking up its head when hearing an unfamiliar gait.
Were we not supposed to read the piece?
Best,
D
Paul D,
I think you're misunderstanding Tim's point. He isn't claiming that the study was perfect, or that the numbers are very reliable- after all, they come with a huge confidence interval. He's just saying that the study doesn't have any serious methodological or statistical flaws (so far), so it's conclusions are at least tentatively reliable.
We'd all like to see a better-funded study with a larger sample size to reduce the variance involved and get a more accurate picture. But that's very different from the knee-jerk inaccurate criticisms being leveled by those with an ideological need to not discuss civilian casualties at all (eg Kaplan). To some, discussing the concept of civilian casualties is somehow a breach of manners, if not outright treason...
Wu
I'll sign on with Carleton Wu about the statistical argument (and by extension with our host, d^2 and others). OTOH, I will also beat my drum, which everyone appears to have ignored:
Destroying the water and sewage systems in an urban nation WILL increase mortality. Then if you let all the pharmacies and hospitals be looted mortality will REALLY INCREASE. Anything else would be a miracle, which appears to be in short supply.
The scandal is not that mortality increased, but that it was trivially foreseable that it would do so, and no corrective actions were taken. There should be consequences.
Coming in late and ignorant for this whole hoo-ha (which is my standard MO).
Like 83.6% of the general public (adjusted for random outlier R^2, tachyon-positive confidence levels) I know fuckall about statistical praxis, and indeed practices. However Dsquared and your track records are good enough for me.
However I do know a good oneliner when I hear it. Can I suggest that the best response to Fumento's "I'm published more than you so I'm right" arguement could be best summed up as "Never mind the quality, feel the width".
PS. Next time yer in Melbourne, you should catch up with an evolving coterie of local bloggers who are now catching up for a drink or several and a bite or two in places that are not crowded doof doof nightspots, and where you can sit and talk, instead of stand and shout.
Sedgwick and Barista are acting as the focal points and the Daily Flute as the Sydney rep. I'd reckon you'd enjoy it and vice-versa.
I think the title of Brad DeLong's post best sums up the study in a single sentence: 98000 ± 90000 Excess Deaths in Iraq. That says says it all, doesn't it?
Pop trot, No, actually it does not. What would "say it all" in this context is something more like this: 98000 excess violence-related deaths in Iraq, with an exceedingly small possibility of +/- 90000, and a far, far more likely possibility of +/- a few thousand. That is how confidence intervals work in cluster sampled cohort studies like this one. It is interesting how many folks who review studies like Roberts et al. believe that confidence intervals do not follow the normal distributions which define them, and end up thinking that the truth is just as likely to be at the interval extremes as at the mean.
No, Pop Trot, it doesn't say it all. If you'd followed any of the online discussion over the past few weeks, you wouldn't be making that particular point.
Sometimes you need interpretation
"an exceedingly small possibility of +/- 90000"
exceedingly small = 5%, one in twenty, 0.05. I am not sure I would call 5% exceedingly small, or I might be accused of wilful exaggeration !
"far, far more likely possibility of +/- a few thousand"
I am not sure what "far, far more likely" means ! You start off with a range which brackets 95% of the possible population mean. As you reduce the range, your chance of including the population mean decline.
But this misses the point, anyway. You can say 98 thousand +/- 1; but this has such a tiny chance of covering the population mean that it is ridiculous. This is why they give a 95% confidence interval.
that looks like the confidence interval spans a twenty fold range:
98 000 +/- 90 000 excess deaths. Boy, is that a flaky confidence interval.
j
Pop,
Most people who follow current events closely bemoan the fact that stories are usually simplified in order to pass them off quickly to an uninterested public. Detail and nuance, we complain, are lost in these simplifications.
Of course, if the simplification fits one's partisan inclinations, it can be useful (if dishonest) to embrace them nevertheless. As you've demonstrated.
Wu
it is amazing how many people are upset, simply over one of the major findings from this study, which accurately sets out the 95% confidence interval. It is also visible just how wide this confidence interval is, but some people don't like that.
j
Scott, good point, but I prefer "98000 ± 90000 Excess Deaths in Iraq" to "Over 100,000 Iraqi deaths since war". Isn't the first more informative than the second?Donald, I didn't know I was making a point. "98000 ± 90000 Excess Deaths in Iraq" either is or isn't a true statement. Carleton, what partisan inclination have I demonstrated? Please clarify.
Pop,
It is slightly more informative that the first statement. But your desire to halt the flow of information at that point (ie that says it all) is indeed indicative of a partisan inclination. You are, in fact, as guilty as the folks who wanted to stop with "Study says 100k dead in Iraq"- you prefer simplifications when the suit your purposes.
Is it fair for me to infer purpose? Well, you're going from one site to another where more in-depth discussion of the study is taking place. But you'd apparently prefer less discussion and more simplification/distortion in the message. IMO that usually happens when one has an axe to grind. I didn't know I was making a point. Yes, well, then why are you wasting our time posting? Or do you have a point that you'd prefer not to admit trying to make?
But, to be fair, by all means take the opportunity to prove me incorrect by explaining why your statement says it all and that further, more in-depth coverage and discussion is unnecessary.
Wu
"98000 ± 90000 Excess Deaths in Iraq" isn't the half of it ! the authors Falluja data also indicates a point estimate of an additional 200 000 deaths.
So what their data shows is 298 000 deaths. Course, this isn't risible...
j