Daniel Davies comments on the attempted disproof by incredulity of the Lancet numbers:
I am curious as to why anyone is bothering with this debate any more (in some of the discussion on Dr Madelyn Hsiao-Rei Hick's comments, it has got parodic, as people discuss the minutiae of the "informed consent" requirements of the questionnaire). Does anyone think at this late date that they are going to come up with a result that proves that the whole war and occupation has been really good for the Iraqis? Have they not noticed that this debate (and the one on global warming too) is a bit like the Berlin Wall - people are only going from one side to the other in one direction?
This prompted a response by Jane Galt who comes up with the macaroni and cheese argument against the study:
But what I wanted to blog about is a somewhat related phenomenon, which is the systematic human tendency to underestimate how long things take. This was driven home to me rather poignantly when I went up against Spencer Ackerman in Blogging Chefs, and tried to estimate just how much I could do in 90 minutes. Then I tested how long it actually took to, say, cook macaroni and cheese.
Well, the best way to find out how long it takes in the field is to do such surveys. From the story in Nature:
each team split into two pairs, a workload that is doable", says Paul Spiegel, an epidemiologist at the United Nations High Commission for Refugees in Geneva, who carried out similar surveys in Kosovo and Ethiopia.
I commented at the time:
you'd hope that this finally puts the matter to rest.
Apparently not -- I had not anticipated the macaroni and cheese argument.
Galt also gives us this:
The more Les Roberts talks, the less confident I get in his results; he doesn't seem to have the faintest clue how the interviews were conducted. He keeps changing the number of interviewers, the number of houses (which has suddenly and without explanation dropped to 30 in some of the arguments) or who asked what. And somehow now all the interviews were done in 3 hours? At 20 households per team, that's 6 houses an hour, ten minutes a house including walking, peeing, and informed consent.
The more Galt writes, the less she seems to understand. Roberts hasn't changed anything. Roberts said that in 2004 (when they used 30 house clusters), they took about three hours, which is about 12 minutes per house. He says that the 2006 interviews took 15-20 minutes. He has not contradicted himself as Galt claims, but is discussing two surveys that took similar (but not the same amount of time per interview).
When Galt misunderstand Roberts, it is wrong to say that he "contradicted" himself. And far from reducing confidence in Roberts, it reduces confidence in Galt.
Need I point out that if Davies is right, and Burnham et. al. are right, then we should be seeing massive floods of refugees?
I guess that Galt took so long the make the macaroni and cheese that she didn't notice the thousands of news stories about Iraqi refugees.
The UN refugee agency believes that about two million refugees have fled Iraq. On the other hand, macaroni and cheese takes longer than you think to make, so the refugee agency could be wrong.
>>Roberts hasn't changed anything. Roberts said that in 2004 (when they used 30 house clusters), they took about three hours, which is about 12 minutes per house. He says that the 2006 interviews took 15-20 minutes.
Where does he say this? The last quote you offered in the post before this cited a time that works out to 11.78 minutes. This is not equal to 15-20 minutes. One reason the numbers are so hard to determine is that this top notch Lancet study hasn't published the actual times; I suspect they didn't gather that data.
Tim Lambert: "The UN refugee agency believes that about two million refugees have fled Iraq" - but without reducing the total population used by Roberts & co for extrapolating up from their anyway biased sample (with its unrepresentative demographics)?
90 minutes to cook mac and cheese?
Good lord, what a 3 alarm disaster that would be. You'd have to throw the pan out, that's for sure.
"One reason the numbers are so hard to determine is that this top notch Lancet study hasn't published the actual times; I suspect they didn't gather that data."
Perhaps the Lancet authors never envisioned people employing "Einsteins Theory or Relativity" to analyze their study.
The last quote you offered in the post before this cited a time that works out to 11.78 minutes. This is not equal to 15-20 minutes.
Good Fucking Christ! How fucking stupid ARE these morons?
The irony is that the Lancet study was probably more thoroughly and systematically planned than the actual invasion/occupation of Iraq.
I don't understand: why were these Lancet-publishing scientists bringing macaroni & cheese to Iraqi households?
If the Iraqis were consuming this much macaroni & cheese, it suggests then that their diets are far richer than previously estimated, and that food delivery must also be common since they apparently welcomed the mac & cheese delivery.
So I presume based on this argument that things in Iraq are far, far better than anyone had estimated in their wildest dreams, and that the bombings and shootings we hear about might be more involved in faulty food preparation than in any sectarian violence let loosed by the US' complete and utter destruction of Iraqi society and social controls.
Assuming from scratch M&C, from putting the pan of water on the burner to finish of cooking al dente macaroni is 15 to 20 minutes (time for 2 Qt of water to boil plus 10-15 minutes of cooking time, draining. A good cook can do the white sauce base and added cheese during that time (even if one needs to reduce block cheese to meltable bits) and melt some butter and brown some bread crumbs for topping, prep the baking dish, and pre-heat the oven, to boot. Combining the sauce and cheese, putting it in the baking dish and adding the topping is another minute. Baking is 15 to 20 minutes. Let's say 45 minutes.
I have to say that you are very lucky she chose macaroni and cheese to argue.
If she had chosen grits, which everyone knows takes 20 minutes, you almost certainly would have lost.
"The irony is that the Lancet study was probably more thoroughly and systematically planned than the actual invasion/occupation of Iraq."
You mean it took more time to plan the Lancet study than it takes to cook traditional* mac & cheese?
*as opposed to "MacGalt & Cheese", which obviously includes a hardening agent (shellac?) to prolong cooking time to 90 minutes -- so you can put it on the stove to boil and then go pick up the kids from soccer practice 45 miles away.
I want to go to mj's house for dinner. But if it takes 12 minutes one time and 20 minutes the next, does that mean I didn't go to the same place.
Here's some irony for you...
Note that in 2002, a Johns Hopkins University estimate of fatalities stemming from a hypothetical smallpox attack are credible and worth citing to support the administration's position. In 2006, Johns Hopkins University estimates of 655,000 excess Iraqi fatalities resulting from our invasion (published in the peer-reviewed Lancet) are, of course, not credible to this administration, and are summarily dismissed.
How Johns Hopkins University has fallen!
The interest of wingnuts and Bush-lovers has always been the same: To defend their President from ANY fact or event or interpretation that makes him or any part of his "Administration" look bad.
These critics of the Lancet quibble with process to deflect the larger significance, and have no answer whatsoever for the central moral questions of this issue:
WHY is it the official, immoral position of the US Military and political arms that Iraqi casualties have not been, and will not be counted?
WHY is there not a single official, never mind comprehensive, study of the human cost of the illegal invasion and continuing occupation of Iraq?
WHY are we "fighting them there"?
ANSWER: Because "Hajji" casualties there do not count the same as Americans, and count much less than casualties here.
No, I've read Jane's blog. I can believe it takes her an hour and a half to cook mac and cheese...
I'm guessing this culinarian is still awaiting her invite to challenge on Iron Chef.
I repeat this all the time to no good effect but a study using the same methodology was cited as a Congressional finding in legislation in 2006 and passed by unanimous consent in the Senate, a voice vote in the House, and signed by the President. Here is the relevant section from S. 2125 `Democratic Republic of the Congo Relief, Security, and Democracy Promotion Act of 2006'.
"SEC. 101. FINDINGS.
Congress makes the following findings:
(7) A mortality study completed in December 2004 by the International Rescue Committee found that 31,000 people were dying monthly and 3,800,000 people had died in the previous six years because of the conflict in the Democratic Republic of the Congo and resulting disintegration of the social service infrastructure, making this one of the deadliest conflicts since World War II."
The bill was co-sponsored by both Sens. Barack Obama (D-IL) and Sam Brownback (R-KS). You would think some scientific reporter would ask Obama or Brownback if they believe in the methodology of the study in their own bill. Or Bush appointee CDC director Dr. Julie Gerberding who has authored her own cluster sampling studies on other topics and published them in The Lancet in the past.
The point is to refute President Bush when he said that The Lancet methodology is 'pretty well discredited'. The fact is the US continues to use the same methodology again and again - just not in Iraq.
Jim said: "No, I've read Jane's blog. I can believe it takes her an hour and a half to cook mac and cheese..."
you could be right. I assumed (perhaps incorrectly) that most of the time was cooking time, but it may take her 70 minutes to get the stove lighted (the electric one).
What a relief it is to actually be talking about something concrete: Jane Galt's Mac & Cheese.
In the world of young DC punditry, regardless of political stripe, McArdle is considered amusing company. Perhaps this marks me out as an insufferable partisan, but I'd sooner chew bricks than hang out with such a person.
And it really does take her habit of argument-by-anecdote to new depths of absurdity. Though at least, for once, she wasn't talking about a colleague's friend's cousin's mac-and-cheese.
Maybe Jane's just trying to get laid:
http://www.theonion.com/content/node/33336
Why anyone pays so much as 90 seconds of attention to someone who has named herself after an Ayn Rand character is beyond me.
But the potential uses of this mac & cheese argument are intriguing, I have to admit. Deployed properly, it can be used to dismiss as flawed pretty much any seemingly impressive or significant achievement of any sort.
"Beethoven's Eroica? Bah. Had to be a rush job. You know how long it actually takes to figure out all those transpositions for the wind parts? Beethoven got so behind trying to get the thing in on time, he has one of the horn players coming in three bars early in the recapitulation. Clearly, we can ignore this as a piece of hack work. Shame, really."
Actually, that should be four bars. Sorry, the macaroni was boiling over just as I was writing the post, which took me about twice as long as I had anticipated.
"I had not anticipated the macaroni and cheese argument."
No one expects the Macaroni Inquisition!
As an avid amateur cook, I would like to comment on the "macaroni argument".
What this line of reasoning fails to take into account is that different cooks might take different amounts of time to cook the same recipe. For example, I can make a batch of fresh handmade pappardelle in about 40 minutes or so. I'd like to see if Jane can match that.
Thus, it took Franklin Roosevelt a little over three years to beat the Nazis. It will take George W. Bush somewhere between six and, let's say, nine hundred years to quell the leaderless Iraquis.
"It will take George W. Bush somewhere between six and, let's say, nine hundred years to quell the leaderless Iraquis."
..and that's being generous.
I'd sooner put my money on faster than light travel (or perhaps Hell freezing over).
One thing is certain, however: If Bush had been President during WWII, we'd all be speaking German now, because he would have attacked Iraq instead of Germany.
But bush & Cheney dodged the draft when there was a real war. So while I know he has screwed up everything he has touched, I can't imagine him doing anything but run away from a real war. Making up a fake one like Iraq must have seemed like a really good idea.
> One thing is certain, however: If Bush had been President during WWII, we'd all be speaking German now, because he would have attacked Iraq instead of Germany.
Are you seriously suggesting that alternative-history Iraq would have been better off with Faisal II still in power? Why do you lefties always hate alternative-history America so much? Support the alternative-history troops!!!!!
In other word's, Galt's logic is this:
"An unimportant aspect of your methodology is not reported to three signficicant figures, therefore all of your results are wrong."
Or in technoweenie-speak, an interview length of 11.78 minutes and an interview length of 15 minutes are indistinguishable if the standard of deviation is 3 minutes.
Idiots: they are so many and non-idiots are so few.
Spencer Ackerman, the company of idiots.
Let's talks process about g.r.i.t.s. now!
Would that include Condi Rice? What about her husba- , errr the pResident?
The irony is that the macaroni and cheese was probably more thoroughly and systematically planned than the actual invasion/occupation of Iraq.
Fixed your typo.
90 minutes
Looks like a classic case of the mains treat bias
Good thing it was mac & cheese instead of paella. That sh*t takes all day, man.
http://smartypants.diaryland.com/032006.html
"One thing is certain, however: If Bush had been President during WWII, we'd all be speaking German now, because he would have attacked Iraq instead of Germany."
Having been attacked by the Japanese, Bush would first attack Japan then follow up by invading China (on the basis that they shared the same Buddhist religion as the Japanese). Opponents to this strategy would be accused of being racists who thought Asians were incapable of understanding democracy.
Meanwhile after a summit meeting with Hitler, Bush would declare that they had prayed together and that after looking into the Reichschancellor's eyes he knew him to be a good man.
This would be followed by a nuclear technology sharing deal with Germany and trade sanctions on Russia.
Ian, your logic is impeccable, but I believe there's a minot typo:
"This would be followed by a nuclear technology sharing deal with Germany and trade sanctions on Russia."
Don't you mean nookyaler*?
*a Yalie with some nooky to give.
I can't imagine him doing anything but run away from a real war. Making up a fake one like Iraq must have seemed like a really good idea.
You are Jean Baudrillard and I claim my 2^0.5 pounds.
Perhaps she was confused as to the ingredients. Let's see it was macaroni and...something.
It's all funny and everything but ... but ... but... did someone say that whatever the number, even if it was a mere, say, 200,000, well, you know, that's a lot for Bush (no, sorry) Cheney and his mob and Blair and Howard and their mobs to be actually personally responsible for, and for that matter for everyone who supported the invasion to be personally responsible for. In fact that would be about 200,000 too many. Particularly given the real reasons for the invasion. (And though we mightn't know them all, we surely know they weren't the ones we were told.)
What astounded me at the time the Lancet study came out was that not one of them said, gosh, if this number is anywhere near accurate it's an atrocity and we need to reassess our participation and our strategy.
And anyway, why didn't she just get some Kraft M&C and sling it in the microwave? Then the timing would be precise. To two or three decimal places...
"if this number is anywhere near accurate it's an atrocity and we need to reassess our participation and our strategy."
...which is precisely why so many Americans deny it so vehemently.
Because, as good Christians, everything they do is noble and good.
Come on, you wouldn't want to disturb their "everyone lived happily ever after" fantasy now would you?
"WHY are we "fighting them there"?
ANSWER: Because "Hajji" casualties there do not count the same as Americans, and count much less than casualties here."
Q: Why are so many ready to yank troops out of the war and bring them home, regardless of the cost in lives with the absence of a stabilizing force.
Answer: Because Iraqi lives are worth less than American lives. Why allow our troops to be placed in harm's way when we can bring them home and allow the sectarian violence to play itself out, sacrificing tens of thousands of lives that don't matter.
JB:
I am not Christian, I am a US citizen and none of that has to do with why I think this study is flawed, but thanks for that string of overgeneralization. It certainly shows you to be a paragon of reason.
Taking a moment off from bald ad homina and jumping to conclusions maybe you'd care to explain why so many comments from Les Roberts on this study lead people reading them to conclusions other than the case?
He managed to state something that made his Nature interviewer think both that the 2006 interviewers worked independently and that they asked the local Iraqis for centers of population away from town centers, both of which were disputed by one of his interviewers.
He also states, in reply to a question about the length of time available for interviews, with a figure that is wholly unrepresentative, without major revisions of other claims he and Burnham made, of the actual time spent interviewing and even unrepresentative of the time available to be spent interviewing with the number of teams they claim.
Now why in the world would anyone think something might be fishy about this study when it only finds something so uncontroversial as all other estimates of deaths in Iraq were off by an order of magnitude by using a sample method that even Roberts notes has very little evidence of correlating with actual deaths in an area over a span of time?
Yes Roberts said something that it was possible to misinterpret. Somehow you think that reflects on his credibility and not the people who misinterpret him...
Lie, of course. Most Americans support bringing the troops home because they don't see them accomplishing anything other thant killing and being killed.
Tim,
Was he misinterpreted claiming his interviewers coordinated with locals to find interview spots away from the city centers?
Was he misinterpreted when asked a follow up about 2006 and he replied falsely which he later "clarified" was a comment on 2004 and so not only irrelevant to the question but damaging to his case since he had functionally 50% more interviewers in 2004?
Is it oddly coincidental that his replies to Nature tended to support the validity of his survey despite being false or irrelevant?
Was he misinterpreting himself when he, in what is either a complete non sequitur or a contradiction of Burnham and himself, claimed that the time devoted to the 2006 study was 2 hours per house?
I don't think this is a result of his inability to make clear sentences.
"They said there wasn't enough time to have done the interviews. We had eight interviewers working ten hour days for 49 days, they had two hours in the field to ask each household five questions. They had time."
The conclusion at least is plain and apparently a non sequitur. I'll ask you or anyone reading this to state how Roberts claiming two hours in the field per house devoted to asking five questions = ample interview time somehow truly means that 2 hours per house was the total labor devoted to the project, which has no reflection on the actual interviewing time.
Another good one is how claiming 8 interviewers in the field is supposed to be understood as 4 teams of 2 that each only interviewed one subject at a time.
What is the most parsimonious explanation? Everybody just misunderstands plain spoken Les Roberts in a manner that happens to support his case despite being false?
Lets just ask it again. Not to mention those Iraqis who have been wounded more or less seriously, not to mention those whose lives and livelihoods have been shattered as a result of the invasion... The actual numbers of Iraqs who have actually died as a result of the invasion are certainly higher than the official figures, and higher than the Iraq Body Count calculation. But let's say that the true figure is an order of magnitude less than the Johns Hopkins figure. At what point does it become okay to be comfortable with being a direct cause, or a facilitator of the causes, of the deaths of so many tens or hundreds of thousands of humans? Especially when it was and is on spurious grounds.
This debate is being used by the war supporters to divert the debate away from the issue of the personal responsibility which is held by everyone who supported, urged, participated, in and even who did not actively oppose, the invasion. It is being diverted away from the issue of moral responsibility to a semantic/statistical argument on decimal points and minutes.
The question remains: how many Iraqis is it okay to kill in order to feed your need to feel like a real man? 10? 100? 1,000? 10,000? Would 100,000 dead Iraqis be enough to balance a redneck's sexual insecurity?
For a moment there, I thought we were actually talking about something serious -- American "selective ignorance" about things they don't want to admit
...Until Kevin re-entered the conversation.
I thought we had gotten away from speculation that "there just had to be fraud in the lancet study because I have found a 10 micro-second discrepancy between the "estimated interview time" [my own uninformed guess] and the "stated" interview time [also my own uninformed guess]."
Oh, well. Back to "Disproof by Mac $ Cheese"...
> What is the most parsimonious explanation?
As 10*49*8/1849 is 2.12, he was presumably referring person-hours.
Unless I imagine him lying about the 49 10-hour
days spent in the field or the number of interviewers,
I see no other way to interpret the statement.
But rob, Roberts said 2 and not 2.12! Obviously that proves that the study is a fraud. Oops, thought I was David Kane for a moment then.
"Unless I imagine him lying about the 49 10-hour days spent in the field or the number of interviewers, I see no other way to interpret the statement."
If Roberts really wanted to lie, he certainly would have made up a better story -- ie, one that involved more interviewers working fewer (and shorter) days.
Let's face it, if people are going to call the Lancet study a fraud based on little more than the nonsense such as we have witnessed here -- "I can't believe they could do it in minutes" --, it clearly does not matter to them one bit what Roberts says.
Kevin wrote:
Well, that's exactly how I understood it when I first read the 2004 Roberts paper. In fact, what I couldn't understand was how Hicks could have misinterpreted it.
"10498/1849 is 2.12"?
Escaping asterisks...
10\*49\*8/1849 is 2.12
Tim writes:
Thanks to Tim for the mention. There is no such thing as bad publicity. Those interested in an update on my fraud claims are welcome to check here.
On a different topic, it would be great if Kevin could provide links all the Roberts interviews he takes issue with in the post above. Perhaps in a new comment in this thread or perhaps in an e-mail to me (dkane at iq dot harvard dot edu) which I could post. I have not followed this issue closely enough to know what is going on. For those who care, I am continuing to gather information on this topic for a paper I hope to present at JSM in August. Suggestions are welcome.
David Kane, I checked out the post you linked to, and it includes this:
In your "fraud" post at SSS, you said something along the lines of there not being a survey with a 99% response rate ever, anywhere, on any topic. In the comments to that (now disappeared) post, I pointed out 4 examples in excess of 97%, with one at 99%. In addition, the ILCS reported a response rate of 98.5%.
"I am working on a paper about the Lancet surveys for the August 2007 Joint Statistical Meetings in Salt Lake City, Utah." -- David Kane
Kane's argument for fraud: "Assume, for a moment, that fraud occurred."
Man, what an entertaining conference that will be.
Is it taking place at Cold Fusion University (AKA University of Utah), by any chance? Or some other "top-notch" venue?
Sorry, I hit the "post" button when I had intended to hit the "preview" button.
The response rates I had been pointing to were household response rates from various Demographic and Health Surveys. I had picked one country from each of the five geographic regions listed [here](http://www.measuredhs.com/countries/start.cfm), and all of those five had household response rates in excess of 90% -- as I said in the message immediately above, four of those five were above 97% and one was at 99.0%.
BTW, for an apples-to-apples comparison, if you calculate the Roberts (or maybe the Burnham) household response rate the same way the Demographic and Health Surveys do, I think you get 98%, not 99%.
JB: Yes, for a moment you did bring up your anti-US bias, but I brought us back on topic. And don't overestimate someone's ability to lie coherently.
Rob: Yes, 10X49X8/1849 is 2.12 but this is irrelevant since they didn't interview 1849 homes. They interviewed 2000 homes and several were disqualified because they wandered into areas that had previously been sampled. So 2.12 hours is relevant to nothing at all; Roberts rounded up from 1.96.
Tim: Now why is it so hard to understand what I am claiming here? It is not the Roberts is defrauding anyone by rounding up 4/100ths of one hour. It is that Roberts claims that the interviewers had 2 hours in the field per house [false], that they controlled for main street bias [contradicted by an interviewer], that they interviewed singly [at best a non sequitur reply about 2004 to a follow up about 2006]. One more time for the folks in the back row!
1) They interviewed in pairs; [4x10x49]/2000=.98 about 1 hour per house. He's already off by a factor of 2 in terms of effective labor.
2) He claims they actually spent 15-20 minutes interviewing per house. Did they really spend an hour in the field per house; 40 minutes travel, guesswork, risk estimation, etc. per house? If not, then even a claim of one hour in the field per house would be false.
3) Roberts is citing the total labor time for the project as if it is relevant to how much time interviewing took. Interview times are a subset of the total time, but that really doesn't speak to the question of how feasible the Lancet interview timeframe is. E.g., if the interviewers spent 3 hours on lunch every day, it really bites into interview time.
Roberts and Burnham describing what happened after
the interview of the first household in a cluster:
"Once we started, we went to the next nearest
39 doorways in a chain that typically spanned two to three blocks."
Sounds like a scheme designed to minimize travel time.
There is no reason to suppose they spent any significant
time on "guesswork". Nor is there any reason to suppose they took anything like a three hour lunch. Nor do I see
any reason to suppose they spent any considerable time on
risk estimation.
Kevin said: "don't overestimate someone's ability to lie coherently."
Or someone's ability to put forward the same inane "disproof by incredulity" argument 20 to 30 times in one thread.
"Once we started, we went to the next nearest 39 doorways in a chain that typically spanned two to three blocks."
"Sounds like a scheme designed to minimize travel time."
Indeed, I believe they consulted with world renowned mathematician Josh D ahead of time to solve the "Traveling Iraqi Interviewer" problem for them -- who did so for close to 2000 households on the back of a napkin in a matter of about five minutes (or perhaps it was 4.567 minutes -- though I doubt he had time to do it in less than 4.956, so if he claims as much, the calculation must have been fraudulent)
After working this almost miraculous feat of genius, Josh told them not to bother even leaving home because he would just claim fraud no matter what they did.
But (silly boys and girls) they thought he was just kidding and went ahead and did the study anyway.
Kevin said: "if the interviewers spent 3 hours on lunch every day, it really bites into interview time."
It really bites. It really does.
I am Nomad, am Nomad. I am perfect.
That which is imperfect must be sterilized.
I shall continue.
I shall return to launchpoint Earth.
I shall sterilize
Everything that is in error must be sterilized.
I am Nomad. I am perfect.
.
.
.
Error.
Error.
Error.
I shall analyze error.
Analyze ... error ...
Examine ...
error.
Error.
Rob: Well discounting the fact that they state in their study summary that risk assessment, finding locations, etc. were part of the survey methodology, you are missing the point entirely. It really should be quite hard to miss because I have stated it about 5-6 times in three threads, but Roberts appears to be making claims that are full of 'truthiness' but shy on relevance.
Here he claims 2 hours in the field per house and the actual time spent interviewing he claims is 15 to 20 minutes. I don't care how they may have spent the 1 hour, 40 minutes extra he implies was spent in the field per house, the point here is the claim he offers is not relevant to the issue. If they didn't really have 2 hours per house in the field his claim is flat out wrong.
Now seriously, I put forth nothing trivial above and the best you can do is critique the marginalia of my argument? I just picked some possible examples of how to spend an hour and 40 minutes out of thin air. They weren't particularly important. The important part was three separate instances where what Roberts says is incoherent with valid argument or incoherent with the case as asserted by his own interviewer.
How else would you care to dance around the issue? Do you understand why 2.12 hours is irrelevant? Why claiming 8 interviewers is irrelevant if they worked in pairs? Why there is tension between him saying that they had interviewers try to control MSB and his interviewer denying it? Why claiming 2 hours in the field per house is either false or irrelevant? Do you understand why it seems a bit coincidental that the cited statements he offered support his case but happen to be false or irrelevant?
JB: What exactly is incredulous about the fact that Roberts has made several important claims revolving around this survey that are not supported by either valid reasoning or his own interviewer?
I am not particularly doubtful they could have interviewed 2000 households in the stated time especially if they didn't care over much about the quality of their results and didn't take the care seemingly implied in the summary of their methodology. That doesn't lend much credence to the study results being fantastic, but I am not the least doubtful it could have happened just like that.
If you want a real example of a fallacy try the "argument from tradition" which is what many defenders of cluster studies are offering in its defense and which is invalid. It doesn't matter how many people or examples of such studies you can dig up, Roberts himself says there is little evidence proving the results of cluster studies correspond to the case. I thnk Lord Pitt said something about how habit gives vice the appearance of virtue. If cluster studies aren't too accurate for numeric representation, it doesn't matter in the least how long a tradition they have.
Kevin wrote:
I don't think it's irrelevant at all. I thought that was one of the more surprising characteristics of the surveys -- and the key characteristic that Hicks missed in her initial criticism that the 2004 study averaged 6 minutes per household.
Robert writes:
I am sorry that the post and your comments disappeared, but that was not my call. I think that the key distinction here is the precise nature of the survey. In Lancet II, as I understand it, they picked a house, interviewed whoever was there, went to the next house and so on. Once they finished that cluster (always in a single day?), they left and did not return. I find it implausible that the household head or spouse was there 99% (contact rate) of the time.
As discussed here, ILCS was a much more thorough (and expensive survey). Interviewers when back to the selected houses again and again and again, over the course of days, if not months. In such a scenario, it does not surprise me that they got very high response rates. If you keep asking and asking and asking, someone will eventually be home and agree to participate.
So, what we need to know is the response rates for ILCS after just one attempt. I have not been able to find these. (Pointers welcome!)
If you want a sense of what "typical" contact/particpation rates in Iraq in 2006 for a single attempt survey, you should check out the results for here.
As always, I appreciate the fact that Tim provides this forum. I would appreciate any insights that others have to offer on this topic. The last thing that I want to do is to write a paper with incorrect conclusions.
I do not know of a single-contact-attempt nation-wide survey from a reputable organization (in any country, on any topic, ever) with a 98.3% or higher response rate. If you know of any counter examples, please tell me!
I do not know of a single-contact-attempt nation-wide survey from a reputable organization (in any country, on any topic, ever) in which the interviewers stood on their heads while conducting interviews and chugged beers after every 3 houses (still standing on their heads, of course), with a 98.3% or higher response rate. If you know of any counter examples, please tell me!
> Rob: Well discounting the fact that they state in their
> study summary that risk assessment, finding locations,
> etc. were part of the survey methodology, you are missing
> the point entirely."
They stated in their summary that they spent a significant
part of their time in the field doing "risk assessment" and
"finding locations"?
Sounds to me like planning stage activities, but
even if they were conducted in the field, just
how hard do you suppose it would be to find the
house next door? How much time do you suppose
would be spent on assessing the risk of going to
the house next door?
Perhaps you are thinking about in-between-cluster
activities. Even if they spent as much as 4 hours
finding a cluster and then analyzing the risk
before entering it, it'd only amount to 6 minutes
per household.
> It really should be quite hard to miss because I have
> stated it about 5-6 times in three threads, but Roberts
> appears to be making claims that are full of 'truthiness'
> but shy on relevance.
> Here he claims 2 hours in the field per house and the
> actual time spent interviewing he claims is 15 to 20
> minutes. I don't care how they may have spent the 1 hour,
> 40 minutes extra he implies was spent in the field per
> house, the point here is the claim he offers is not
> relevant to the issue. If they didn't really have 2 hours
> per house in the field his claim is flat out wrong."
If the total of their man-hours in the field divided by the
total number of households visited is approximately equal
2, how could his claim be "flat out wrong"?
> Now seriously, I put forth nothing trivial above and the
> best you can do is critique the marginalia of my
> argument?"
Seems to me like you are straining at gnats.
It as if you have invested so much of yourself into Roberts
and Burnham being wrong, you can't find it in yourself to
give them a fair shake.
> I just picked some possible examples of how to
> spend an hour and 40 minutes out of thin air. They
> weren't particularly important. The important part was
> three separate instances where what Roberts says is
> incoherent with valid argument or incoherent with the
> case as asserted by his own interviewer."
You might try this: Whenever you find what you think is
an incoherent statement, go back and see if there is
*ANY* way to interpret it in a way which is *NOT*
incoherent.
In any event, when someone is describing parts of a
complicated lengthy task in an interview,
simplifications may be made that some might not find
reasonable. He may make statements capable of
being interpreted in more than one way. Words or
phrases that could have been a part of a sentence
may be left out either by accident or because it is
felt that the absent bit is understood. (There are
usually plenty of possible explanations for
inconsistency besides bad faith.)
> How else would you care to dance around the issue? Do you
> understand why 2.12 hours is irrelevant?
No, I don't, but then I can't see any reason to
expect that there wouldn't be a number of
in-the-field tasks that could be divied up between
a group of people and then executed simultaneously.
> Why claiming 8 interviewers is irrelevant if they worked
> in pairs?
Just off the top of my head...
In those households that took the most time(ie. those
with deaths), there would be a small pile of documents
to be examined. Surely, two people could do that more
quickly than one.
Perhaps they interviewed two people per household
independently wherever possible. That too would be
done more quickly with two member teams.
> Why there is tension between him saying that they had
> interviewers try to control MSB and his interviewer
> denying it?
I'm not familiar with this "tension". In any case,
I don't see any reason to expect that his interviewer
would know more about than he does.
David Kane wrote:
Perhaps, but there are two issues: 1) is this really a key distinction from the point of view of response rates; and 2) is this a difference that would result in bias?
For the first issue, the question is whether repeated calls were needed in the ILCS (or the other studies I pointed to) to make initial contact, or to clarify responses to the questionnaire -- as we know from the ILCS report, much of the maternal and child health section had to be re-done, so this suggests that initial contact and approval to participate had occurred. In addition, the ILCS wasn't the same sort of cluster survey as the Roberts or Burnham studies. In the latter studies, no particular house was targeted -- the PSU was the cluster, and if a house was empty or the HH head or spouse was not available, I believe the plan was to go to the next house until 40 total qualifying HHs had been approached. In that case, the relevant question isn't so much the overall response rate but the refusal rate, the "no one at home" rate, and the "someone at home, but not a qualified respondent" rate.
For the second, response rates, in general, are not an end in themselves: they are an indicator of potential bias in the sample. Usually, of course, we think of low response rates as worrisome because we fear that there is systematic selection bias in who is not at home, or who is home but refuses to participate. It appears that the refusal rate in the Roberts and Burnham studies is not out of line with the refusal rate in the ILCS, and also in the DHS studies from many countries and across many years that I pointed to. So that leaves the "no one at home" and "no qualified respondent at home" rates (and, of course, it's insufficient merely to point out what those rates are; one also has to discuss in which direction these might bias the results).
I don't know what those rates are, but both of these are rather different than what appeared to prompt your charge of fraud, i.e., an overall response rate that seemed (to you) improbably high. Do you now concede that your statement on the SSS blog (paraphrased) of "no survey, anywhere, anytime, on any topic, has ever reported a response rate of 99%" was incorrect?
There is a standard procedure followed with suspected scientific fraud cases and it involves the collection of substantial evidence before any charges are brought.
I'd say any "researcher" who implies fraud was involved based on little more than his incredulity at the survey response rate has no business at Harvard or any other university for that matter.
Robert makes a bunch of sensible points which I mostly agree with and then writes:
Well, that isn't what I said. You can see for yourself a copy of that post here.
Now, it is obviously true that some surveys feature a response rate of 100%. I asked my two daughters were they wanted to go for dinner the other night and, Bingo!, a 100% response rate. I like to think that the context makes clear that I am talking about polls like the Lancet's: country-wide, in person, one attempt at contact and so on.
Rob:
>>They stated in their summary that they spent a significant part of their time in the field doing "risk assessment" and "finding locations"?
They stated they spent time doing it.
>>Sounds to me like planning stage activities, but even if they were conducted in the field, just how hard do you suppose it would be to find the house next door? How much time do you suppose would be spent on assessing the risk of going to the house next door?
In Iraq? I'd spend a fair amount of time.
>>If the total of their man-hours in the field divided by the total number of households visited is approximately equal 2, how could his claim be "flat out wrong"?
Because he claims they had 2 hours in the field per house. He had effectively 4 interviewing teams. Right there, the 2 hours per household figure is already wrong. Also, unless the time spent in the field is a one to one correspondence with the total labor, there's no way all ten hours in a day were utilized, equalling two hours per house available to be used for interviewing. And since they weren't his claim was irrelevant to the point at hand, namely how feasible were the Lancet study methods?
>>It as if you have invested so much of yourself into Roberts and Burnham being wrong, you can't find it in yourself to give them a fair shake.
I guess that's how you see it. OTOH, I think I have 3 pretty clear examples, without even considering the substance of the study, of misleading or false statements made by Roberts regarding the feasibility and methods of the study.
>>You might try this: Whenever you find what you think is an incoherent statement, go back and see if there is ANY way to interpret it in a way which is NOT incoherent.
I could multiply theories endlessly. I am trying to be epistemologically charitable, but at least for the three points I cited, I can't see a way.
>>>How else would you care to dance around the issue? Do you understand why 2.12 hours is irrelevant?
>>No, I don't, but then I can't see any reason to expect that there wouldn't be a number of in-the-field tasks that could be divied up between a group of people and then executed simultaneously.
No, my point was you used the wrong figure. They interviewed in 2000 households, some of which they disqualified.
>>>Why claiming 8 interviewers is irrelevant if they worked in pairs?
>>Just off the top of my head...
>>In those households that took the most time(ie. those with deaths), there would be a small pile of documents to be examined. Surely, two people could do that more quickly than one.
Sure; in fact, I've noted that before in this blog. However the bulk of the time would be interviewing and tag team speaking is not going to speed up the process. The marginal gains of shuffling paper faster are not going to double the interview speed now are they?
>>Perhaps they interviewed two people per household independently wherever possible. That too would be done more quickly with two member teams.
This contradicts what one of the Iraqi interviewers said.
>>>Why there is tension between him saying that they had interviewers try to control MSB and his interviewer denying it?
>>I'm not familiar with this "tension". In any case, I don't see any reason to expect that his interviewer would know more about than he does.
Read the Nature article. Roberts and Burnham claim that the interviewers worked with locals to find clusters away from town center. The interviewer denied it. The reason the interviewer is more likely to know is that the interviewer would have been the one asking.
David Kane wrote:
Read the Nature article. Roberts and Burnham claim that the interviewers worked with locals to find clusters away from town center. The interviewer denied it.
I read it. Here's what the article says:
"The Iraqi interviewer told Nature that in bigger towns or neighbourhoods, rather than taking the main street, the team picked a business street at random and chose a residential street leading off that, so that peripheral parts of the area would be included. But again, details are unclear. Roberts and Gilbert Burnham, also at Johns Hopkins, say local people were asked to identify pockets of homes away from the centre; the Iraqi interviewer says the team never worked with locals on this issue."
So the "tension" relates to whether the interviewers sought help from locals, or whether they were able to manage without it. Better set up a judicial inquiry, eh?
Robert, comments are never lost; not in a world full of Lancet junkies. Your comment read:
Evidently, David was unable to find the Demographic and Health Surveys. Typically, one can find reported response rates in Chapter 1 of the final reports. I arbitrarily clicked on five of those countries and selected the most recent report. Of those five, here are the reported response rates:
Egypt 2005: 98.9% (of 21,972 HH's);
Jordan 2002: 99.0% (of 7907 HH's);
Burkina Faso 2003: 99.4% (of 9149 HH's); and
Turkmenistan 2000: 98.6% (of 6391 HH's).
The fifth, an old report from Thailand, reported a response rate of 94%.
So it appears Burkina Faso, 2003 DHS meets his standard.
That last comment was mine, not Robert's (but only the intro was my own).
Kevin Donoghue wrote:
You are scary.
Yes, this is getting to be a bit of an obsession. Of course it wasn't hard to foresee the disappearance of that particular post. I don't make a habit of recycling vanished comments, but I took your remark upthread as an invitation to do so.
Here is an interesting test of the claim often made by Josh and others that the media cover major incidents like car bombs and air strikes pretty effectively. Baghdad blogger Omar (no admirer of Burnham & Co.) wrote this two days ago:
"From the increased activity of jetfighters and the way the explosions sound it looks like a wave of aerial bombing is underway somewhere on the peripheries of the city...not sure yet what's going on but we'll provide an update if we find more info."
http://iraqthemodel.blogspot.com/2007/03/action-follows-calm.html
His one update so far doesn't add much: the explosions sounded very loud from "Plaestine [Palestine?] Street, just west of Sadr city".
Now, it seems to me that a wave of aerial bombing audible in central Baghdad ought to be newsworthy but I don't see much about it on TV or on the web. It strikes me as a useful data-point for evaluating media effectiveness. If they don't cover the outskirts of Baghdad at all well then it's pretty safe to conclude that what happens in Tal Afar mostly stays in Tal Afar.
Okay so I need to get a life. When the weather improves I will buy a new bicycle and tackle the Wicklow hills.
Robert asks:
Yes, you are correct. I should be more precise in my language and, to be honest, I knew a lot less about surveys 6 months ago than I do today. The key distinction is between nation-wide surveys that try to contact interviewees just one time on just one day (example here) and surveys which try over several days/weeks/months to do so (example here). .
I can't find a precise citation, but it seems fairly obvious that DHS tries multiple times over multiple days/weeks/months to contact selected households. In such a situation, high response rates are perfectly plausible. If you do repeated "callbacks," response rates can be high. The DHS manual mentions:
Correct. The Lancet surveys, as far as I know, did no callbacks. In that context, 98% response rates are completely implausible. So, my better worded claim still stands. There are no examples of single-contact nation-wide surveys with 98% response rates. If you know of any counter-examples, I would be eager to know about them.
David Kane wrote:
Right, but that's because the DHS sampling designs had HHs as the PSU (and the WPO survey you cited appears to have had individuals as the PSU). In the Roberts and Burnham studies, the PSU was the cluster. That's why there was no need for callbacks.
Hmmm. On what basis do you conclude that this is completely implausible under the situation that the PSU is the cluster?
I am not sure what you mean here. The interviewers picked a randon house on a random street and started there. They went from that house to the nearest 39 other houses. Their claim is that, more or less, 39 out of those 40 houses had the head of household present and willing to participate at that moment in time. If no one was home, that house was marked as absent. Only 40 houses were tried each time. (They did not keep knocking on doors until they found 40 people at home.)
So, it does not matter what the PSU is. If you don't have callbacks, your response rate should be much lower.
David Kane: You're right. I mis-remembered what was written in the 2006 Burnham paper, and your post prompted me to go back and look. In fact, the Burnham paper states on page 2 (right hand column, about half way down): "Empty houses or those that refused to participate were passed over until 40 households had been interviewed in all locations." This is, of course, at odds with the statements on page 4 that 1849 HHs were ultimately included in the sample, and "[i]n 16 (0.9%) dwellings, residents were absent; 15 (0.8%) households refused to participate."
I would think that the refusal rate is not relevant to your case -- presumably, the ILCS (and DHS, and similar surveys) would not "callback" a HH that initially had refused to participate, and would only make repeated attempts to empty HHs, or those where there was no appropriate respondent at home.
So to clarify, you're basing your case for fraud on the presumption that 16 dwellings where residents were absent are too few for a survey where there were no callbacks?
Not only that, but in a civil war, what is the most likely reason for a house being empty? IMHO, the household is dead or fled; either way there's a *huge* downward bias on the death rate esimates.
Of course people like David Kane never mention that - I wonder why?
Robert: Thanks for pointing that sentence out. I had mistakenly thought that they stopped at precisely 40 rather than going on until they got 40 answers.
Why does that matter? Well, in situation with a trivial number of absences/refusals, it won't matter. Even if you threw away the data from any cluster which featured a refusal/absence, the overall answer would be close to the same. But, if the survey had lots of absences/refusals, you might end up with significant biases.
Imagine that, say, people with relatives killed in fighting agree to participate in surveys at much higher rate because they want to have their story told. People without dead relatives want to avoid trouble and so turn away strangers. In such a hypothetical scenario, the raw data would suggest a much higher mortality rate even if all the data is accurate and everyone is telling the truth.
Almost my entire "case" --- not sure that I like this terminology --- rests on the implausibility of the absence/refusal rate in a survey with no callbacks. It is my understanding that ILHS/DHS and other surveys (like the US census) pick the houses they want first and then just keep going back to that house over and over again. If you have the time/money to do this, it is certainly the best approach. Otherwise, you run the risk of the people who respond being systematically different than those who don't.
David Kane wrote:
I thought your post was entitled, "A case for fraud?"
Anyway, my point was that once a household had refused to participate, it would be wrong to return repeatedly in the hope that you could convince them to change their minds. In this respect, the ILCS (or the DHS) should only get one bite at the refusal apple, just as for the Roberts or Burnham studies. Therefore, I would think that if you are building your case on the difference between callbacks and no callbacks the only thing that should matter is the absence rate, not the refusal rate.
Oh, yeah! I need to modify my language.
Anyway, you write:
I am not sure what you mean by "wrong" in this context, but there are certainly situations where both strategies (don't callback or callback again and again) are appropriate. It is perfectly fine for ILCS/DHS to callback again and again. Otherwise, the statistical issues associated with adjustments for absence/refusal issues are complicated and annoying. Indeed, I think that most/all statisticians would say that callbacks, if possible, are preferred.
Now costs and safety are sometimes an issue, as with Lancet. I have no problem with the Lancet's methodology of no callbacks. The refusal rate matters as well since there is often an initial refusal which, after some repeat visits, turns into participation. Perhaps the husband declines to participate one day, but you come back the next day and get all the needed data from the wife.
In any case, I still can't find a single-contact nationwide survey (on any topic in any country on any date) with 1% absence and refusal rates. Counter-examples welcome!
David Kane wrote:
What I meant by "wrong" is "unethical." I'm more familiar with the DHS than with other surveys and perhaps the DHS are special and unrepresentative, but with them once someone says "we don't want to play" you move on. In fact, I would think that calling back again and again to a household that had initially refused until you badgered them into participation could produce biased answers. That means it wouldn't just be wrong -- it'd be stupid. Callbacks for absences don't have that problem.
That's why I think if you're predicating your case on the callback/no callback distinction, you're down to the absence rate. I would think the absence rate depends on household size, household composition, and female labor force participation so you'll probably want to examine those.
Understood. Fortunately, I don't think that I need to worry about the arcana of "household size, household composition, and female labor force participation." Whatever values you want for those, it still seems wildly implausible that 99% of the houses selected had either the household head or spouse at home at the one moment the Lancet interviewers came calling. How could that possibly be true if World Public Opinion.org only finds 75% or so of people to be home? Something does not add up.
Kevin D.:
No the tension arises from the claims of the authors of the study failing to cohere with the claims of the interviewer. According to the authors of the study, who presumably had reasons for wanting them to do X, they did X. According to the interviewer, X was not done. This has import both for the level of awareness of the study's authors on what happened at all and also for whatever bias or reason they had for prescribing searches for clusters away from town center in the first place.
Neither necessarily indicates fraud or any intentional bad faith. You're arguing against straw men. What either point could indicate, and what you certainly cannot discount without more information than anyone has yet cited on this issue in this blog, is that the authors - as Hicks asserted - don't really know what happened in their Iraq interviews, and the potential for bias due to the study methods not being followed.
Again with the Lancet estimates. Sheesh.
Just for fun can anyone who's happy with the methodology of the study in producing it's estimate of excess deaths (a very serious matter) and has confidence in it's results explain whether it's not also a much better estimation of the coverage of death certificates for dead Iraqis (a less serious matter) with a much higher confidence level ?
A key test of the accuracy of study's survey data was that over 90% of respondents claiming to have family members killed had these claims backed by death certificates.
Now we know there is a much lower confidence level in there actually having been 100,000 dead Iraqi bodies after the one-month, light-resistance invasion or 655,000 dead Iraqi bodies in 2006. We know this from the scarcity of this correlation ever being made by anyone when discussing at length why the estimates are accurate. Even though this is the point of the entire exercise, to estimate how many Iraqis actually died.
So a burning question would be if you're willing to go to the lengths of walking the streets of Iraq in 2006 to conduct face-to-face interviews with random Iraqis, WTF is stopping anyone making a trip to the Iraqi Health Ministry to audit their death certificate records ?
You know, the records that are accruate to a 90% degree as per this clearly sufficient study for testing such a thing.
If this survey was random enough and with sufficient nation-wide samplng to produce estimates of the number of people who have died, it is sure as shit good enough to put to bed those complaints that official death figures won't be accurate because of a lack of regional reporting.
It's 90% coverage nationwide, not just for Baghdad.
This joke where the same figures are attacked and defended 5 years on is still running.
So what's the punchline ?
Why not resolve the question of how accurate the estimates of deaths are by determining and citing how many Iraqis actually died ?
Honestly, I don't get it.
Well Kilo, why doesn't the Iraqi Ministry of Health release their own numbers?
David Kane wrote:
Could you be conflating the contact rate with the completion rate? The methodology section you pointed to is rather skimpy but it appears that they give a 74% completion rate and a 94% contact rate, so it sounds as if they didn't find a quarter of the people to be absent. In addition, because of the skimpiness of the methodology section it's hard to tell whether they were targeting a particular individual within a HH or not -- I would think you could figure that out if the sex ratio of the respondents was close to even or skewed heavily toward women but I don't see that in the report -- but if they were, that could also contribute to some of the difference.
Unemployment rate should be added to my list of household size, household composition, and female labor force participation. Perhaps those do qualify as arcana but I would not think them irrelevant arcana if the goal is to make a case for fraud based on the absence rate.
Kevin wrote:
I think it's entirely possible that the main authors don't really know every detail of what happened in the Iraq interviews and that there is potential for bias. However, pointing out the potential for bias isn't sufficient; anyone can do that. The thing that separates idle speculation from expertise is the ability to make reasonable estimates of the direction and magnitude of those potential biases.
And Hicks? A hack.
Well Kilo, why doesn't the Iraqi Ministry of Health release their own numbers?
Posted by: Zarquon | March 17, 2007 03:21 AM
.
They have released their own numbers. Just not for complete periods. You can see them referenced in each update of the Brookings Iraq index.
.
Considering you see such interest in determining and confirming** estimates of the number of Iraqi deaths during specific, complete periods that leaves you with the question why have none of these researchers asked for this ?
.
This would of course be a question for you to answer just as why the Iraqi government does what it does is a question for me to answer.
.
** Sorry, that should say "defending estimates". I cannot provide any supporting evidence that anyone is interested in "confirming" these estimates of Iraqi deaths.
Kilo, these are issues that have been well and widely discussed. Basically, death certificates are handed out locally and then handed up a long chain to a central repository. Even in countries where this chain is intact it can take a while for death certificates to make that trip. For example, the US released its final 2004 mortality report at the end of 2006, so it's running about two years late and our vital statistics reporting system is pretty much intact. (As a matter of fact, two years is an improvement -- it used to be longer and it didn't cover all fifty states).
Kilo, have you heard of Deputy Health Minister Hakim al-Zamili and the reasons given for his arrest? If you have then presumably you can guess why it's not a simple matter to audit the Ministry's figures. As one of Tim Lambert's earlier posts points out, the Minister himself acknowledges that the figures are incomplete and he has attempted his own guesstimate (which isn't very credible).
Kilo,
You've an excellent point there. If the possession of a death certificate is considered verification of a death in the household, why wouldn't checking death certificates at various local certificate issuing authorities be more accurate than the cluster sampling and more accurate than waiting for the certificates to trickle up the chain to a central authority?
It kind of makes one wonder about the reason for the survey, if the death certificates are that accurate of a measure and reliable enough to be a check on the oral interviews.
Robert asks:
Correct. I need to read my own posts more closely!
If the participation rate for one survey is 99% while it is 74% for a similar survey, then something is going on. Now, it may be useful to focus on just the overall response rates or to look at the subparts which drive those differences. But, either way, there is something strange. Why are people so much more willing to participate in the Lancet survey?
Of course, we can tell all sorts of tales about why they might be. But that still leaves aside the mystery of the overall response rates. In other words, I do not think that it is as useful to look at the differences between, say, contact rates. Instead, focus on the overall response rate. Why should that be 25% different between these polls? And why should Lancet II have a higher response rate than any other single-contact nationwide poll ever taken?
As one of Tim Lambert's earlier posts points out, the Minister himself acknowledges that the figures are incomplete and he has attempted his own guesstimate (which isn't very credible).
Posted by: Kevin Donoghue | March 17, 2007 08:17 AM
And ? The only reason we are discussing this is that the Lancet figures aren't credible estimates of how many people were killed as a result of the invasion.
Since when did credibility become an issue ?
You won't understand this either until you stop viewing these figures as a reasonable conclusion to valid sampling methods for producing statistical estimates and instead try to equate them with how many Iraqis are actually dead.
There are about 7 people worldwide who give any kind of a toss how many Iraqis died as a result of this war *in theory only*.
There's a reason the press have treated these figures with far more interest than others relating to wheat crop yields.
There's a reason that excess death estimates being attributed to just people being killed by US soldiers (or Bush, personally, in many cases) are rejected by so many people as ridiculously high and implausible.
These are the reasons there is another article in another newspaper about the Lancet figures this month, just like there was last month and every month before that.
And here we are on a blog where the majority of posts complaining about press treatment of studies always point out where data and conclusions have been misinterpreted.
Yet you'll be fucked trying to find a single one about Lancet's Iraqi death figures correcting anyone interpreting excess death estimates as being actual figures of people dead or killed by the invasion. Even though this is how these figures are treated in every article.
I'd suggest if the people arguing for the validity of these Lancet figures mean something then stick with it. Give it 5 years or so and we'll see how repetition of things like "valid cluster" and "12.37 minutes between door knocks" turned out in terms of putting this matter to rest.
Or FFS figure out the issue at some point and address that instead.
David Kane wrote:
Perhaps they were less willing to participate in the WPO survey? Opinion surveys appear to have lower participation rates than enumeration surveys.
Why? Does this mean you reject the argument that refusal rates should not depend on whether there were callbacks but that absence rates could? I'm just trying to clarify my understanding of your position.
Does it? The IRC's 2000 DR-Congo study appears to have been single-contact and it's HH response rate was 99%. The HRDAG mortality survey in East Timor appears to have been single-contact, and they reported an overall HH response rate of 97%. So the Burnham study's HH response rate of 98.3% seems pretty much in line with those two.
BTW, since we're actually in the middle of the "how long does it take to cook mac and cheese" thread, I'll toss in something about timing from the Thailand DHS report. The Thai HH survey asked similar but not identical information as the Burnham study, and averaged 4.5 minutes -- "similar" meaning that they didn't ask mortality information (but did ask a couple of other things). In the Burnham sample there were 629 deaths reported in 1847 HHs so roughly two-thirds of the HHs didn't have a death to report. So even if the Iraq interview length were double the Thai average (that is, 9 minutes) for the two-thirds of HHs in which there was no death, that would still appear to leave plenty of margin for the interviewers to handle the households that *did* report a death within the overall average reported by Burnham.
Robert writes:
I appreciate these references but, at some point, I will grow tired of checking these stories out myself. Every single (allegedly) high response rate nationwide survey that I have taken the time to check turns out to involve multiple callbacks. Do you really expect these to be different? If you could provide links to the methodology reports of these surveys, any evidence for your claim that they appear to be single contact, I will try to investigate further. It turns out that survey organizations, out of either ignorance or sneakiness, report "response rates" when they really mean participation rates.
In other words, in a place like Congo and East Timor, it is very hard to do a randomized population survey because you don't have even a rudimentary census. You, therefore, can't draw 1,000 random individuals/households from the proverbial hat to survey because you don't have a list of all the individuals to put into the hat in the first place. Now, there are ways around this and, in difficult places, you make do with what you have. But if there is no good framework for selecting your sample, your contact rate can go to 100%. There wasn't any individual that you tried to contact that you couldn't!
But, if you provide some links, I will dig deeper.
By the way, I think that we can now put the "Was there enough time to do the interviews?" question to bed. Burnham reported at the MIT talk that the two person teams "sometimes" split up. In other words, instead of having a single team of 4 doing 40 interviews in a day (the Hicks interpretation of the write-up) or two teams of two each doing 20 interviews a day (my previous understanding), we now have 4 individuals each doing 10 interviews. However long the interviews took, this seems like a reasonable workload.
Now, this dispute still provides evidence that the US authors were/are disconnected from what actually happened on the ground in Iraq. And why did the Lancet article fail to explain this clearly? But, given this new (correct?) information, there is no reason to doubt that an individual couldn't do 10 interviews per day.
David Kane wrote:
Oh dear.
David Kane said "By the way, I think that we can now put the "Was there enough time to do the interviews?" question to bed."
Too bad, So sad. The gate on one avenue for speculation has been shut and barred. Kaching.
But hey, there are always others, right?
I'm sure that won't stop some "researcher" somewhere (at "Haaavid", say) from speculating about fraud based on little more than "I can't belieeeeeeve the response rate was that high"s, now will it?
For some, the possibilities for such speculation are endless.
Here's an interesting article--http://www.guardian.co.uk/Iraq/Story/0,,2037219,00.html
Notice the last paragraph, which is why I link to it. A poll found that one person in four had lost a close relative. Too bad we don't know how many people would fall into the typical Iraqi's notion of "close relative". It's sort of irritating--why not just ask if someone in the household had died?
Anyone know anything more about this poll?
Here it is, Donald:
http://www.opinion.co.uk/Newsroom_details.aspx?NewsId=67
Thanks, Kevin. Very frustrating. I could see why they'd ask about relatives being murdered, since it would tie into the level of unhappiness, but why not add just one or two extra questions and ask about whether someone in the household had died? Yeah, someone might change households, but it would pin the number down better. Or they could ask about dead siblings and/or children or parents vs. more distant relations.
David:
You say that Burnham puts the matter of the timeframe to rest but for me it just makes more questions.
Nature says that an interviewer in 2006 claimed they did not work independently of one another and when asked again by Nature about this particular inconsistency, Roberts replied that he was referring to 2004 and not 2006. Burnham claims that they also went singly in 2006. How is this supposed to help?
The Lancet study group's response changes at every single objection.
1) Too hard for two groups of four to pull this off? The groups of four split into twos.
2) Too hard for groups of two to pull it off? The groups of two split into singles.
3) The interviewer for Lancet in Iraq says they didn't split into singles? I was talking about 2004 and not 2006.
4) The whole narrative is now inconsistent? Well they "sometimes" split into singles.
Burnham's claim about them sometimes going singly, and I still don't understand why if there is some substantive reason for picking teams of two and specifying one male and one female then going singly is okay, still contradicts the direct report of one of the interviewers, according to the Nature article.
The ABC/USA Today/BBC poll seems to come closest to replicating the Lancet mortality survey. Not very close, but it's the closest yet.
The link is here--
http://abcnews.go.com/US/story?id=2954716&page=1
One Iraqi household in six had someone "harmed" by violence. Presuming 6 people in the average household, that would mean at least 1 person in 36 were "harmed". (Or somewhat higher, since some households would have 2 or more victims.)
This sounds like a number that is probably lower than Lancet 2, since I'd assume the majority of those "harmed" were wounded, not killed. But 1 out of 36 would be about 700,000 people.
This sounds like a number that is probably lower than Lancet 2, since I'd assume the majority of those "harmed" were wounded, not killed. But 1 out of 36 would be about 700,000 people."
Any way you look at it, one hell of a lot of people have been killed/maimed/wounded/harmed/traumatized/orphaned/affected (pick your adjective) by the war.
All this debate about what the "actual" number is (which really depends on on your definition, at any rate) is a sideshow (or perhaps sidetrack [or perhaps derailment in some cases]).
An order of magnitude estimate is certainly sufficient to gauge how bad things are in Iraq, and by most measures (including the IBC number), they are pretty damned bad.
At some point one has to cease debating what the "actual" number is and start doing something about the problem.
Unfortunately, some have used the debate to derail that process.
My above estimate is conservative--some "harmed" households could contain more than one harmed individual.
I agree that the war is a disaster, but I'm sure you agree, JB, that if the death toll is in the hundreds of thousands it is important for Americans (and others) to know this, rather than believe a much lower figure if it is wrong. This new poll suggests the correct figure might be in the hundreds of thousands--whether it agrees with the Lancet2 CI is secondary. What also matters here is that it is clear that people could replicate the Lancet study in all of its details if they wanted to. There shouldn't be any need to wonder how many "harmed" individuals there were per household, or how large the household was. Apparently no one who commissions polls cares about this.
One unfortunate side-effect of the IBC approach is that we now have a reinforced double standard on how atrocities are reported. The scale of enemy atrocities is generally given with large dramatic estimates and we almost never know the basis for them. Deaths caused by US policies, on the other hand, have to be actual counts (or at least based on what is purported to be actual counts). This is going to bias discussion in favor of so-called humanitarian interventions. So that's why the number debate is important. I suspect you agree.
Donald: The new poll could suggest many things. A finding about being "harmed by violence" necessarily entails nothing for consequent mortality rates. 100% of members of my household have been harmed by violence in the past 5 years and none have been mortal wounds. It's a pretty vague question.
And of course I understand the cash value of polls like this. The point is supposed to be that the invasion caused all of these deaths. Of course this is made in complete ignorance of the fact that the majority of them are due to the acts of radical Islamist foreign infiltrators or displaced Baathist supporters willing to wreck Iraq's economy and commit mass murder as a form of political discourse.
Why is the Lancet methodology even worthy of replication? How is it not easier and more accurate to get the records of death certificates being issued at the local level? Even Lancet used those as a check on their polling.
And yes, the war could have been handled much better from data available at the time of the invasion. I don't think that the war was handled incompetently is any sign that the effort itself was unjust. The Baath regime didn't have a swell record on excess mortality; why is the fact that the Baath regime was a mass-murdering, police state irrelevant to whether or not invading was just?
In context, Kevin, when the pollster asks how many people have been "harmed", it probably means serious injury or death. Much of the poll is about the security situation, and what sorts of violence have the responders seen. One question was whether the respondent had seen various forms of violence in their neighborhood and the highest response (44 percent) was for "unnecessary coalition violence against civilians". Asked to say what they feared the most, car bombs and suicide attacks were at the top of the list (38 percent). Unnecessary violence from coalition forces was second at 16 percent and various forms of Iraqi-on-Iraqi violence picked up the rest. But to me, the fact that 16 percent fear coalition forces the most (Shiites almost as much as Sunnis--the poll is broken down by sect) and that 44 percent have seen unnecessary violence by coalition forces suggests to me that we aren't hearing about a lot that goes on over there.
There are some statistics you could use for the prowar side too--53 percent of Shiites and 59 percent of Kurds but only 7 percent of Sunnis think the country is better off than before the war. But I think the picture painted of violence makes it seem more widespread than one would guess from the usual media statistics and in particular, US violence seems more widespread and a bigger cause of concern for Iraqis than you would ever dream if you went by the US press.
Donald you seem to be tweaking things in your direction. First you find the one report that uses the phrase "close relative" for one of the questions while the poll just said 'relative', which would mean anyone related to them, uncles, cousins etc. (which is probably around 50+ people).
Then you interpret harmed to mean "serious injury or death". Why "serious"? The question casts the widest possible net over injuries, not just "serious" ones. If anything it would possibly include some violent events that impacted household members, but which didn't necessarily cause injuries at all, like kidnappings, which the poll suggests are rampant in many areas.
I don't think any firm casualty numbers can be derived from this poll, but this does look like another piece of evidence suggesting L2 is an overestimate. Roberts has said that his study suggests "one in seven" (14%) of households have had someone killed. And this poll finds only 3% more when the net is widened to include anyone "harmed". And this is after seven more months of the most intense violence since 2003, more intense than most of the period covered by L2. If L2 was right, widening the net to include any injuries should produce a far, far higher number, not 3% higher. It should probably be 3% higher just for deaths alone given the extra seven months of extreme violence.
(Shiites almost as much as Sunnis--the poll is broken down by sect)
Too bad L2 isn't similarly broken down. Note the huge disparity of answers depending on the sect of the respondent. If a poll is oversampling Sunni Arabs or undersampling Kurds for example, you're likely to get overestimates of violence related indicators. Yet we have no idea, like most everything else, of the sects of the respondents (or the surveyors) in L2. We do know though that L2 says the surveyors were fluent in "English and Arabic", while Kurdish is not mentioned.
Question 35 actually reads:
"Have you or an immediate family member - by which I mean someone living in this household - been physically harmed by the violence that is occurring in the country at this time?"
I don't see how this can either support or undermine L2 since the period during which the violence took place is unclear and so is the actual harm done. The two polls do give the impression that Iraq is a far worse mess than suggested by official figures and suchlike. But then it must take quite an effort to square the official line with the accounts of reporters on the scene. Anyone who can do that should easily be able to explain these polls away as well.
Try not to be so suspicious, Josh. I'm as biased as the next guy, but I used "close relative" because it happened that was the term used in the report that I first read.
BTW, there are two polls here--the one that I mistakenly thought said "close relative" and this other one I'm citing in my more recent posts. The second one oversampled Sunnis, though it's not clear to me if they took this into account in calculating the overall results. It doesn't make much difference for my purposes--the number of Sunnis who said someone in their household was "physically harmed" was 22 percent, for Shiites it was 17 percent and for Kurds it was 7 percent. If I use the 60, 20, 20 percent composition I usually see claimed for the Shiite, Sunni, Kurd population I get 16 percent physically harmed. The poll says 17 percent overall.
Since neither of us are statistics gurus, I don't mind embarrassing myself with back of the envelope calculations in front of you. (The rest of you can avert your eyes.) I'll assume 7 people per household--that's what Lancet 2 found. (I thought it was 6 before.) Assume 24 million people (allowing for some refugees having gone). That's 3.4 million households. Multiply by .16 and you get 550,000 households with at least 1 casualty.
I'm going to guess the number of casualties per harmed household is between 1 and 2, so that's 550,000 to 1.1 million casualties. Then assume that the fraction which are deaths is 1/4 to 1/3. Then you get a range of deaths from 140,000 to 370,000. Lower than L2, though the top of my range almost touches the bottom of theirs. Higher than what IBC would allow, though the bottom number might be close to acceptable if it includes insurgent dead. (Something it'd be interesting to have counted from press accounts, btw. I suspect you'd find it difficult to do.)
One could quarrel with this, of course, but it seems like a reasonable estimate to make based on these numbers.
"These numbers" refers to the poll results, of course--the other numbers are my guesses about household size and casualties per house and deaths /casualties and so forth.
Kevin D, it does suggest a minimum number of casualties in the mid hundreds of thousands and possibly more, or anyway, I can't see how one could get a number for "physically harmed" which isn't at least that much. It doesn't tell us how many of the physically harmed were killed or how many were harmed per household--I supplied my guesses above, to illustrate the point that the poll suggests a death toll well into the hundreds of thousands range.
Donald, what are you taking the phrase "at this time" to mean, in the question I quoted above? You seem to be extending it back to 2003 or thereabouts.
"But then it must take quite an effort to square the official line with the accounts of reporters on the scene. Anyone who can do that should easily be able to explain these polls away as well."
I'm not sure what "official line" or "official figures" you're fantasizing about, or how you're able to compare it to this poll when you've just finished claiming the poll is all but meaningless in this regard, but "official figures" I've seen suggest a very big mess with about 3,000 being killed each month, lots more injured, kidnapped etc., all kinds of attacks up, up, up, and a situation that has been rapidly deteriorating for some time.
As to "accounts of reporters on the scene", I doubt you read many of them, and you should recall that you're speaking to someone who reads them much more closely than you do, and has for the past four years.
You should try reading some of them, particularly not just anecdotes you cherry pick from blogs, but the ones that have actually explored in detail "official figures" and compared them against local records, which are more relevant, such as:
BORZOU DARAGAHI: Well, we think -- the Los Angeles Times thinks these numbers are too large, depending on the extensive research we've done. Earlier this year, around June, the report was published at least in June, but the reporting was done over weeks earlier. We went to morgues, cemeteries, hospitals, health officials, and we gathered as many statistics as we could on the actual dead bodies, and the number we came up with around June was about at least 50,000.
And that kind of jibed with some of the news report that were out there, the accumulation of news reports, in terms of the numbers kill. The U.N. says that there's about 3,000 a month being killed; that also fits in with our numbers and with morgue numbers. This number of 600,000 or more killed since the beginning of the war, it's way off our charts.
To Kevin D--- I see your point. I was assuming that violence "at this time" meant all the violence since the March 2003 invasion. Maybe some of the respondents thought in terms of a shorter period.
Sticking to what I think one can conclude from the poll, allowing for sampling error (which is probably not too large for this big a poll), there are probably several hundred thousand households which have suffered at least one casualty during whatever time period the respondents had in mind.
Donald,
I posted my last question before I saw your response to my earlier comment. I don't think we disagree. When I say these figures neither support nor undermine L2 what I mean is that they could be consistent with a wide range of deaths. If the true number of violent deaths is inside the 426,000 to 794,000 confidence interval then L2 is vindicated and if not then there is something badly wrong with it. These polls could be consistent with either scenario, depending on your assumptions. I agree that they broadly support the impression that the death toll is in the hundreds of thousands. But Ragout, for example, would say that we had good reason to believe that before L2 came out.
Not for the first time, Josh quotes Borzou Daragahi. The report Daragahi refers to is probably by Louise Roug. Josh doesn't often quote Roug, who says inconvenient things like this: "Obtaining accurate numbers from the Health Ministry or the 18 major hospitals serving Baghdad proved difficult, because officials at all tiers of government routinely inflate or deflate numbers to suit political purposes." She makes it very clear that the figures she has obtained are at best a lower bound (and even that is only true so long as it suits the MoH to produce low numbers rather than high ones).
Like Roug, experienced reporters like Patrick Cockburn, John Simpson and many others make it perfectly clear that the figures furnished by officials (yes Josh that's what I mean by official figures) are only a partial count and that in general there is much more violence taking place than the media coverage would suggest to the unwary. Indeed Daragahi may be aware of this: "about at least 50,000" - that seems to be an acknowledgement that we are talking about a minimum here.
It must take a real effort to read reports from Iraq and miss these warnings. Perhaps the trick is to look at them so "closely" that you can't see the wood for the trees: "only that which can be counted is permitted to count." Reports of fighting or explosions which don't mention a specific number of deaths add nothing to the total. That's fair enough as long as you remember the limitations of the method.
In short, the statement n > b doesn't tell us what n is, it only tells us what it isn't.
First it's that I should read them, then when I read them more closely than you but still don't agree with you, I read them too closely. Ok.
That you claim I'm ignoring "warnings" about underreporting is stupid because the "warnings" don't say anything other than that there's a lot of unreported stuff, which I already assume and have since the beginning.
Next you assume Roug would have some different view than Daragahi (who I believe was involved in the same work, and certainly knows the basis of its statements and analysis better then you or I do).
And it is true that n>b can't give us an exact figure for n, but that does not mean ANY value for n is reasonable next to the evidence and manner in which b was obtained. What Daragahi is saying is that the comprehensive way they went about obtaining b ("We went to morgues, cemeteries, hospitals, health officials" ..etc.), and checking with as many different sources as they could find, renders some values for n (like 600,000) quite absurd and extreme.
But then you'll have no problem disregarding this "warning" from "reporters on the ground". You'll also have no problem disregarding that LR claimed his study would be confirmed if someone would only go investigate cemetaries. Daragahi is telling you they did, and is telling you that the Lancet numbers don't look anything like reality when you do.
On the comment about "inflating or deflating" numbers for political reasons, what this tells me is that some Iraqi officials in Baghdad, according to Roug, have tweaked numbers up or down for various reasons or at various times. Maybe so. It's not clear in the reports exactly what she bases this claim on, but far from being "inconvenient", it can be surmised that she can say this because she and her colleagues had some means to check the veracity of the numbers she got from some sources against those she got from other sources, which was the pont of their efforts. And it tells me that when they looked at multiple sources for the numbers they could see that some accounts didn't match up with others.
Given what Daragahi says (Roug has never commented on Lancet numbers to my knowledge), quite obviously the discrepencies they found in different accounts, and which presumably informed such a comment, were not anything remotely like Lancet-size discrepencies. If the discrepencies were huge rather than small, I think they would have written on this, and Daragahi would hardly say what he does (unless he's also in on the big cover up).
You choose to take Roug's comment to mean "they're hiding hundreds of thousands of deaths", but that doesn't seem to be anything like what they mean, and Daragahi is telling you that's not what it means, and that this interpretation is not reasonable to them.
Perhaps more importantly, what it tells me is that independent investigators have at least some ways to go about checking the veracity of "official numbers", by checking many different sources and accounts, and have some means to find fault with claims made and the persons making them, and that's what they were trying to do.
Quite the opposite of this would be the numbers reported by the Lancet teams, which neither Louise Roug nor any other independent party have any means to check at all, and which are apparently therefore to be taken as much more reliable.
Josh, it's because of arguments like yours in the 11:27 comment that I'm not completely in the Lancet2 camp. But it's not wholly convincing either--there are reporters like Cockburn and Fisk and a couple of others who either support outright the hundreds of thousands of deaths (Rory Stewart did so most recently in a side-comment in his NYT guest column) or who say that the violence is widespread and largely unreported. Maybe the LA Times could talk to people who were fearless and unafraid to tell the truth about the death toll, or maybe they talked to people who thought it was a dangerous subject to be truthful about. Or maybe people in these various places simply don't keep accurate counts--there are also reports, you know, of the chaotic state of the Iraqi health care system. If methodology is a problem to be discussed with L2, it seems at least as much a problem with the LA Times approach.
That's why polls and surveys ought to be an invaluable method of determining whether or not the official statistics are correct, because you bypass officials of all types and go directly to ordinary people (who, of course, might also lie). You and others think the Lancet papers are badly flawed--fine, but now we've got another poll which can't give us an accurate count, but at the very least strongly supports a figure of several hundred thousand households which have suffered at least one casualty. You ought to wonder whether this means the IBC number is much too low, even if the truth isn't necessarily as high as the Lancet2 confidence interval.
And what I wonder is why these news agencies couldn't just have ordered up a poll explicitly intended to determine the number of dead and wounded, given that this controversy over death tolls has made it into the mainstream press to some extent. If they are really serious professional organizations devoted to uncovering the truth and if it occurred to them to commission these polls on Iraqi attitudes, why on earth wouldn't they try to determine mortality rates?
Josh wrote: What Daragahi is saying is that the comprehensive way they went about obtaining [a lower bound] ("We went to morgues, cemeteries, hospitals, health officials" ..etc.), and checking with as many different sources as they could find, renders some values for n (like 600,000) quite absurd and extreme.
I've re-read the report by Louise Roug and Doug Smith (Raheem Salman is credited in the footnote, Daragahi doesn't get a mention) and there is nothing in it to support the claim that their investigation established any upper bound whatever, nor even that they hoped to do so.
But then you'll have no problem disregarding this "warning" from "reporters on the ground".
I'll certainly have no problem whatever ignoring what a reporter says in the course of a radio interview about his colleagues' work. Likewise, if somebody at JHU claims that the Iraq study established results which the authors themselves do not assert, rest assured I won't expect you to regard that as persuasive - unless an argument goes with it and the argument is sound.
But let's suppose that every LA Times reporter who has ever been to Baghdad is incredulous about the 600,000 figure. What would that tell us? Not a damn thing, until they tell us why they are incredulous. I read reports to find out what the reporters actually see happening. What they see happening is more violence than they can cover, corpses eaten by dogs and various other horrors which are entirely compatible with a very high death toll indeed. But if Daragahi ever tells us how he gets from "I'm incredulous" to "I therefore conclude that X is an upper bound for the death toll", believe me I'll study his argument.
Incidentally, Roug and Smith don't mention checking cemetaries at all in that particular report. But Roug does mention them in another report: "In the Sunni cemeteries serving Baghdad, a city of 5 million people, demand for tombs is so high that people are buried between old graves or at the edges of the burial grounds."
That's not the sort of test Roberts suggested. But it certainly doesn't conflict with his reading of the situation. Bear in mind that many people die of natural causes, even in Iraq. Mortality rates would have to rise quite a bit to have a noticeable impact on demand for tombs.
You choose to take Roug's comment to mean "they're hiding hundreds of thousands of deaths"....
Misrepresentations like this don't enhance your claim to be a careful reader, of reports from Iraq or anything else.
Kevin, i think you're wasting my time. I'd just say that the Roug quote about Sunni graveyards in Baghdad "doesn't conflict" with the figures Roug is giving you for Baghdad in the very same report.
Likewise, the figures given by Roug do not conflict in any way with the findings of Burnham et al., since she makes it abundantly clear that the figures are an undercount; they "do not present the full picture of the violence in the capital." A careful reader will also notice words like "at least" and "conservative" at various points in her report.
Tim - you've probably seen this http://news.bbc.co.uk/1/hi/uk_politics/6495753.stm already, and are typing up a new post on the Lancet report right now. But if you haven't, then I think that one or two people here might be interested...