Tim Lambert (deltoidblog AT gmail.com) is a computer scientist at the University of New South Wales.
Daniel Davies comments on the attempted disproof by incredulity of the Lancet numbers:
I am curious as to why anyone is bothering with this debate any more (in some of the discussion on Dr Madelyn Hsiao-Rei Hick's comments, it has got parodic, as people discuss the minutiae of the "informed consent" requirements of the questionnaire). Does anyone think at this late date that they are going to come up with a result that proves that the whole war and occupation has been really good for the Iraqis? Have they not noticed that this debate (and the one on global warming too) is a bit like the Berlin Wall - people are only going from one side to the other in one direction?
This prompted a response by Jane Galt who comes up with the macaroni and cheese argument against the study:
But what I wanted to blog about is a somewhat related phenomenon, which is the systematic human tendency to underestimate how long things take. This was driven home to me rather poignantly when I went up against Spencer Ackerman in Blogging Chefs, and tried to estimate just how much I could do in 90 minutes. Then I tested how long it actually took to, say, cook macaroni and cheese.
Well, the best way to find out how long it takes in the field is to do such surveys. From the story in Nature:
each team split into two pairs, a workload that is doable", says Paul Spiegel, an epidemiologist at the United Nations High Commission for Refugees in Geneva, who carried out similar surveys in Kosovo and Ethiopia.
I commented at the time:
you'd hope that this finally puts the matter to rest.
Apparently not -- I had not anticipated the macaroni and cheese argument.
Galt also gives us this:
The more Les Roberts talks, the less confident I get in his results; he doesn't seem to have the faintest clue how the interviews were conducted. He keeps changing the number of interviewers, the number of houses (which has suddenly and without explanation dropped to 30 in some of the arguments) or who asked what. And somehow now all the interviews were done in 3 hours? At 20 households per team, that's 6 houses an hour, ten minutes a house including walking, peeing, and informed consent.
The more Galt writes, the less she seems to understand. Roberts hasn't changed anything. Roberts said that in 2004 (when they used 30 house clusters), they took about three hours, which is about 12 minutes per house. He says that the 2006 interviews took 15-20 minutes. He has not contradicted himself as Galt claims, but is discussing two surveys that took similar (but not the same amount of time per interview).
When Galt misunderstand Roberts, it is wrong to say that he "contradicted" himself. And far from reducing confidence in Roberts, it reduces confidence in Galt.
Need I point out that if Davies is right, and Burnham et. al. are right, then we should be seeing massive floods of refugees?
I guess that Galt took so long the make the macaroni and cheese that she didn't notice the thousands of news stories about Iraqi refugees.
The UN refugee agency believes that about two million refugees have fled Iraq. On the other hand, macaroni and cheese takes longer than you think to make, so the refugee agency could be wrong.
Comments
Where does he say this? The last quote you offered in the post before this cited a time that works out to 11.78 minutes. This is not equal to 15-20 minutes. One reason the numbers are so hard to determine is that this top notch Lancet study hasn't published the actual times; I suspect they didn't gather that data.
Posted by: Kevin | March 10, 2007 3:55 AM
Tim Lambert: "The UN refugee agency believes that about two million refugees have fled Iraq" - but without reducing the total population used by Roberts & co for extrapolating up from their anyway biased sample (with its unrepresentative demographics)?
Posted by: Tim Curtin | March 10, 2007 4:23 AM
90 minutes to cook mac and cheese?
Good lord, what a 3 alarm disaster that would be. You'd have to throw the pan out, that's for sure.
Posted by: JB | March 10, 2007 10:31 AM
"One reason the numbers are so hard to determine is that this top notch Lancet study hasn't published the actual times; I suspect they didn't gather that data."
Perhaps the Lancet authors never envisioned people employing "Einsteins Theory or Relativity" to analyze their study.
Posted by: JB | March 10, 2007 10:57 AM
The last quote you offered in the post before this cited a time that works out to 11.78 minutes. This is not equal to 15-20 minutes.
Good Fucking Christ! How fucking stupid ARE these morons?
Posted by: dave | March 10, 2007 11:39 AM
The irony is that the Lancet study was probably more thoroughly and systematically planned than the actual invasion/occupation of Iraq.
Posted by: Redleg | March 10, 2007 11:47 AM
I don't understand: why were these Lancet-publishing scientists bringing macaroni & cheese to Iraqi households?
If the Iraqis were consuming this much macaroni & cheese, it suggests then that their diets are far richer than previously estimated, and that food delivery must also be common since they apparently welcomed the mac & cheese delivery.
So I presume based on this argument that things in Iraq are far, far better than anyone had estimated in their wildest dreams, and that the bombings and shootings we hear about might be more involved in faulty food preparation than in any sectarian violence let loosed by the US' complete and utter destruction of Iraqi society and social controls.
Posted by: El Cid | March 10, 2007 12:01 PM
Assuming from scratch M&C, from putting the pan of water on the burner to finish of cooking al dente macaroni is 15 to 20 minutes (time for 2 Qt of water to boil plus 10-15 minutes of cooking time, draining. A good cook can do the white sauce base and added cheese during that time (even if one needs to reduce block cheese to meltable bits) and melt some butter and brown some bread crumbs for topping, prep the baking dish, and pre-heat the oven, to boot. Combining the sauce and cheese, putting it in the baking dish and adding the topping is another minute. Baking is 15 to 20 minutes. Let's say 45 minutes.
Posted by: mj | March 10, 2007 12:04 PM
I have to say that you are very lucky she chose macaroni and cheese to argue.
If she had chosen grits, which everyone knows takes 20 minutes, you almost certainly would have lost.
Posted by: jerry | March 10, 2007 12:16 PM
"The irony is that the Lancet study was probably more thoroughly and systematically planned than the actual invasion/occupation of Iraq."
You mean it took more time to plan the Lancet study than it takes to cook traditional* mac & cheese?
*as opposed to "MacGalt & Cheese", which obviously includes a hardening agent (shellac?) to prolong cooking time to 90 minutes -- so you can put it on the stove to boil and then go pick up the kids from soccer practice 45 miles away.
Posted by: JB | March 10, 2007 12:16 PM
I want to go to mj's house for dinner. But if it takes 12 minutes one time and 20 minutes the next, does that mean I didn't go to the same place.
Posted by: Ken Houghton | March 10, 2007 12:17 PM
Here's some irony for you...
Note that in 2002, a Johns Hopkins University estimate of fatalities stemming from a hypothetical smallpox attack are credible and worth citing to support the administration's position. In 2006, Johns Hopkins University estimates of 655,000 excess Iraqi fatalities resulting from our invasion (published in the peer-reviewed Lancet) are, of course, not credible to this administration, and are summarily dismissed.
How Johns Hopkins University has fallen!
Posted by: Malacandra | March 10, 2007 12:21 PM
The interest of wingnuts and Bush-lovers has always been the same: To defend their President from ANY fact or event or interpretation that makes him or any part of his "Administration" look bad.
These critics of the Lancet quibble with process to deflect the larger significance, and have no answer whatsoever for the central moral questions of this issue:
WHY is it the official, immoral position of the US Military and political arms that Iraqi casualties have not been, and will not be counted?
WHY is there not a single official, never mind comprehensive, study of the human cost of the illegal invasion and continuing occupation of Iraq?
WHY are we "fighting them there"?
ANSWER: Because "Hajji" casualties there do not count the same as Americans, and count much less than casualties here.
Posted by: CYberian | March 10, 2007 12:25 PM
No, I've read Jane's blog. I can believe it takes her an hour and a half to cook mac and cheese...
Posted by: Jim | March 10, 2007 12:40 PM
I'm guessing this culinarian is still awaiting her invite to challenge on Iron Chef.
Posted by: The Guilty Carnivore | March 10, 2007 12:46 PM
I repeat this all the time to no good effect but a study using the same methodology was cited as a Congressional finding in legislation in 2006 and passed by unanimous consent in the Senate, a voice vote in the House, and signed by the President. Here is the relevant section from S. 2125 `Democratic Republic of the Congo Relief, Security, and Democracy Promotion Act of 2006'.
"SEC. 101. FINDINGS.
(7) A mortality study completed in December 2004 by the International Rescue Committee found that 31,000 people were dying monthly and 3,800,000 people had died in the previous six years because of the conflict in the Democratic Republic of the Congo and resulting disintegration of the social service infrastructure, making this one of the deadliest conflicts since World War II."
The bill was co-sponsored by both Sens. Barack Obama (D-IL) and Sam Brownback (R-KS). You would think some scientific reporter would ask Obama or Brownback if they believe in the methodology of the study in their own bill. Or Bush appointee CDC director Dr. Julie Gerberding who has authored her own cluster sampling studies on other topics and published them in The Lancet in the past.
The point is to refute President Bush when he said that The Lancet methodology is 'pretty well discredited'. The fact is the US continues to use the same methodology again and again - just not in Iraq.
Posted by: joejoejoe | March 10, 2007 12:50 PM
Jim said: "No, I've read Jane's blog. I can believe it takes her an hour and a half to cook mac and cheese..."
you could be right. I assumed (perhaps incorrectly) that most of the time was cooking time, but it may take her 70 minutes to get the stove lighted (the electric one).
What a relief it is to actually be talking about something concrete: Jane Galt's Mac & Cheese.
Posted by: JB | March 10, 2007 1:05 PM
In the world of young DC punditry, regardless of political stripe, McArdle is considered amusing company. Perhaps this marks me out as an insufferable partisan, but I'd sooner chew bricks than hang out with such a person.
And it really does take her habit of argument-by-anecdote to new depths of absurdity. Though at least, for once, she wasn't talking about a colleague's friend's cousin's mac-and-cheese.
Posted by: pseudonymous in nc | March 10, 2007 1:19 PM
Maybe Jane's just trying to get laid:
http://www.theonion.com/content/node/33336
Posted by: MasonMcD | March 10, 2007 1:20 PM
Why anyone pays so much as 90 seconds of attention to someone who has named herself after an Ayn Rand character is beyond me.
But the potential uses of this mac & cheese argument are intriguing, I have to admit. Deployed properly, it can be used to dismiss as flawed pretty much any seemingly impressive or significant achievement of any sort.
"Beethoven's Eroica? Bah. Had to be a rush job. You know how long it actually takes to figure out all those transpositions for the wind parts? Beethoven got so behind trying to get the thing in on time, he has one of the horn players coming in three bars early in the recapitulation. Clearly, we can ignore this as a piece of hack work. Shame, really."
Posted by: mason | March 10, 2007 1:27 PM
Actually, that should be four bars. Sorry, the macaroni was boiling over just as I was writing the post, which took me about twice as long as I had anticipated.
Posted by: mason | March 10, 2007 1:40 PM
"I had not anticipated the macaroni and cheese argument."
No one expects the Macaroni Inquisition!
Posted by: Matt | March 10, 2007 1:50 PM
As an avid amateur cook, I would like to comment on the "macaroni argument".
What this line of reasoning fails to take into account is that different cooks might take different amounts of time to cook the same recipe. For example, I can make a batch of fresh handmade pappardelle in about 40 minutes or so. I'd like to see if Jane can match that.
Thus, it took Franklin Roosevelt a little over three years to beat the Nazis. It will take George W. Bush somewhere between six and, let's say, nine hundred years to quell the leaderless Iraquis.
Posted by: Carl from L.A. | March 10, 2007 2:52 PM
"It will take George W. Bush somewhere between six and, let's say, nine hundred years to quell the leaderless Iraquis."
..and that's being generous.
I'd sooner put my money on faster than light travel (or perhaps Hell freezing over).
One thing is certain, however: If Bush had been President during WWII, we'd all be speaking German now, because he would have attacked Iraq instead of Germany.
Posted by: JB | March 10, 2007 3:12 PM
But bush & Cheney dodged the draft when there was a real war. So while I know he has screwed up everything he has touched, I can't imagine him doing anything but run away from a real war. Making up a fake one like Iraq must have seemed like a really good idea.
Posted by: tom | March 10, 2007 5:50 PM
Are you seriously suggesting that alternative-history Iraq would have been better off with Faisal II still in power? Why do you lefties always hate alternative-history America so much? Support the alternative-history troops!!!!!
Posted by: huxley | March 10, 2007 6:06 PM
In other word's, Galt's logic is this:
"An unimportant aspect of your methodology is not reported to three signficicant figures, therefore all of your results are wrong."
Or in technoweenie-speak, an interview length of 11.78 minutes and an interview length of 15 minutes are indistinguishable if the standard of deviation is 3 minutes.
Idiots: they are so many and non-idiots are so few.
Posted by: Charles | March 10, 2007 6:39 PM
Spencer Ackerman, the company of idiots.
Let's talks process about g.r.i.t.s. now!
Would that include Condi Rice? What about her husba- , errr the pResident?
Posted by: Mr.Murder | March 10, 2007 7:55 PM
The irony is that the macaroni and cheese was probably more thoroughly and systematically planned than the actual invasion/occupation of Iraq.
Fixed your typo.
Posted by: melior | March 10, 2007 8:14 PM
90 minutes
Looks like a classic case of the mains treat bias
Posted by: anthony | March 10, 2007 8:50 PM
Good thing it was mac & cheese instead of paella. That sh*t takes all day, man.
http://smartypants.diaryland.com/032006.html
Posted by: Connie | March 10, 2007 9:46 PM
"One thing is certain, however: If Bush had been President during WWII, we'd all be speaking German now, because he would have attacked Iraq instead of Germany."
Having been attacked by the Japanese, Bush would first attack Japan then follow up by invading China (on the basis that they shared the same Buddhist religion as the Japanese). Opponents to this strategy would be accused of being racists who thought Asians were incapable of understanding democracy.
Meanwhile after a summit meeting with Hitler, Bush would declare that they had prayed together and that after looking into the Reichschancellor's eyes he knew him to be a good man.
This would be followed by a nuclear technology sharing deal with Germany and trade sanctions on Russia.
Posted by: Ian Gould | March 10, 2007 10:33 PM
Ian, your logic is impeccable, but I believe there's a minot typo:
"This would be followed by a nuclear technology sharing deal with Germany and trade sanctions on Russia."
Don't you mean nookyaler*?
*a Yalie with some nooky to give.
Posted by: JB | March 11, 2007 10:42 AM
I can't imagine him doing anything but run away from a real war. Making up a fake one like Iraq must have seemed like a really good idea.
You are Jean Baudrillard and I claim my 2^0.5 pounds.
Posted by: dsquared | March 11, 2007 11:57 AM
Perhaps she was confused as to the ingredients. Let's see it was macaroni and...something.
Posted by: Abe G | March 11, 2007 3:31 PM
It's all funny and everything but ... but ... but... did someone say that whatever the number, even if it was a mere, say, 200,000, well, you know, that's a lot for Bush (no, sorry) Cheney and his mob and Blair and Howard and their mobs to be actually personally responsible for, and for that matter for everyone who supported the invasion to be personally responsible for. In fact that would be about 200,000 too many. Particularly given the real reasons for the invasion. (And though we mightn't know them all, we surely know they weren't the ones we were told.) What astounded me at the time the Lancet study came out was that not one of them said, gosh, if this number is anywhere near accurate it's an atrocity and we need to reassess our participation and our strategy. And anyway, why didn't she just get some Kraft M&C and sling it in the microwave? Then the timing would be precise. To two or three decimal places...
Posted by: graham giblin | March 12, 2007 4:25 AM
"if this number is anywhere near accurate it's an atrocity and we need to reassess our participation and our strategy."
...which is precisely why so many Americans deny it so vehemently.
Because, as good Christians, everything they do is noble and good.
Come on, you wouldn't want to disturb their "everyone lived happily ever after" fantasy now would you?
Posted by: JB | March 12, 2007 7:39 AM
"WHY are we "fighting them there"?
ANSWER: Because "Hajji" casualties there do not count the same as Americans, and count much less than casualties here."
Q: Why are so many ready to yank troops out of the war and bring them home, regardless of the cost in lives with the absence of a stabilizing force.
Answer: Because Iraqi lives are worth less than American lives. Why allow our troops to be placed in harm's way when we can bring them home and allow the sectarian violence to play itself out, sacrificing tens of thousands of lives that don't matter.
Posted by: Frank | March 12, 2007 4:11 PM
JB:
I am not Christian, I am a US citizen and none of that has to do with why I think this study is flawed, but thanks for that string of overgeneralization. It certainly shows you to be a paragon of reason.
Taking a moment off from bald ad homina and jumping to conclusions maybe you'd care to explain why so many comments from Les Roberts on this study lead people reading them to conclusions other than the case?
He managed to state something that made his Nature interviewer think both that the 2006 interviewers worked independently and that they asked the local Iraqis for centers of population away from town centers, both of which were disputed by one of his interviewers.
He also states, in reply to a question about the length of time available for interviews, with a figure that is wholly unrepresentative, without major revisions of other claims he and Burnham made, of the actual time spent interviewing and even unrepresentative of the time available to be spent interviewing with the number of teams they claim.
Now why in the world would anyone think something might be fishy about this study when it only finds something so uncontroversial as all other estimates of deaths in Iraq were off by an order of magnitude by using a sample method that even Roberts notes has very little evidence of correlating with actual deaths in an area over a span of time?
Posted by: Kevin | March 12, 2007 9:06 PM
Yes Roberts said something that it was possible to misinterpret. Somehow you think that reflects on his credibility and not the people who misinterpret him...
Posted by: Tim Lambert | March 12, 2007 9:14 PM
Lie, of course. Most Americans support bringing the troops home because they don't see them accomplishing anything other thant killing and being killed.
Posted by: Barry | March 13, 2007 12:52 AM
Tim,
Was he misinterpreted claiming his interviewers coordinated with locals to find interview spots away from the city centers?
Was he misinterpreted when asked a follow up about 2006 and he replied falsely which he later "clarified" was a comment on 2004 and so not only irrelevant to the question but damaging to his case since he had functionally 50% more interviewers in 2004?
Is it oddly coincidental that his replies to Nature tended to support the validity of his survey despite being false or irrelevant?
Was he misinterpreting himself when he, in what is either a complete non sequitur or a contradiction of Burnham and himself, claimed that the time devoted to the 2006 study was 2 hours per house?
I don't think this is a result of his inability to make clear sentences.
"They said there wasn't enough time to have done the interviews. We had eight interviewers working ten hour days for 49 days, they had two hours in the field to ask each household five questions. They had time."
The conclusion at least is plain and apparently a non sequitur. I'll ask you or anyone reading this to state how Roberts claiming two hours in the field per house devoted to asking five questions = ample interview time somehow truly means that 2 hours per house was the total labor devoted to the project, which has no reflection on the actual interviewing time.
Another good one is how claiming 8 interviewers in the field is supposed to be understood as 4 teams of 2 that each only interviewed one subject at a time.
What is the most parsimonious explanation? Everybody just misunderstands plain spoken Les Roberts in a manner that happens to support his case despite being false?
Posted by: Kevin | March 13, 2007 1:49 AM
Lets just ask it again. Not to mention those Iraqis who have been wounded more or less seriously, not to mention those whose lives and livelihoods have been shattered as a result of the invasion... The actual numbers of Iraqs who have actually died as a result of the invasion are certainly higher than the official figures, and higher than the Iraq Body Count calculation. But let's say that the true figure is an order of magnitude less than the Johns Hopkins figure. At what point does it become okay to be comfortable with being a direct cause, or a facilitator of the causes, of the deaths of so many tens or hundreds of thousands of humans? Especially when it was and is on spurious grounds. This debate is being used by the war supporters to divert the debate away from the issue of the personal responsibility which is held by everyone who supported, urged, participated, in and even who did not actively oppose, the invasion. It is being diverted away from the issue of moral responsibility to a semantic/statistical argument on decimal points and minutes. The question remains: how many Iraqis is it okay to kill in order to feed your need to feel like a real man? 10? 100? 1,000? 10,000? Would 100,000 dead Iraqis be enough to balance a redneck's sexual insecurity?
Posted by: Graham Giblin | March 13, 2007 3:48 AM
For a moment there, I thought we were actually talking about something serious -- American "selective ignorance" about things they don't want to admit
...Until Kevin re-entered the conversation.
I thought we had gotten away from speculation that "there just had to be fraud in the lancet study because I have found a 10 micro-second discrepancy between the "estimated interview time" [my own uninformed guess] and the "stated" interview time [also my own uninformed guess]."
Oh, well. Back to "Disproof by Mac $ Cheese"...
Posted by: JB | March 13, 2007 9:49 AM
As 10498/1849 is 2.12, he was presumably referring person-hours.
Unless I imagine him lying about the 49 10-hour days spent in the field or the number of interviewers, I see no other way to interpret the statement.
Posted by: rob | March 13, 2007 11:35 AM
But rob, Roberts said 2 and not 2.12! Obviously that proves that the study is a fraud. Oops, thought I was David Kane for a moment then.
Posted by: Tim Lambert | March 13, 2007 12:13 PM
"Unless I imagine him lying about the 49 10-hour days spent in the field or the number of interviewers, I see no other way to interpret the statement."
If Roberts really wanted to lie, he certainly would have made up a better story -- ie, one that involved more interviewers working fewer (and shorter) days.
Let's face it, if people are going to call the Lancet study a fraud based on little more than the nonsense such as we have witnessed here -- "I can't believe they could do it in minutes" --, it clearly does not matter to them one bit what Roberts says.
Posted by: JB | March 13, 2007 12:55 PM
Kevin wrote:
Well, that's exactly how I understood it when I first read the 2004 Roberts paper. In fact, what I couldn't understand was how Hicks could have misinterpreted it.
Posted by: Robert | March 13, 2007 1:24 PM
"10498/1849 is 2.12"?
Escaping asterisks...
10*49*8/1849 is 2.12
Posted by: rob | March 13, 2007 2:00 PM
Tim writes:
Thanks to Tim for the mention. There is no such thing as bad publicity. Those interested in an update on my fraud claims are welcome to check here.
On a different topic, it would be great if Kevin could provide links all the Roberts interviews he takes issue with in the post above. Perhaps in a new comment in this thread or perhaps in an e-mail to me (dkane at iq dot harvard dot edu) which I could post. I have not followed this issue closely enough to know what is going on. For those who care, I am continuing to gather information on this topic for a paper I hope to present at JSM in August. Suggestions are welcome.
Posted by: David Kane | March 13, 2007 4:08 PM
David Kane, I checked out the post you linked to, and it includes this:
In your "fraud" post at SSS, you said something along the lines of there not being a survey with a 99% response rate ever, anywhere, on any topic. In the comments to that (now disappeared) post, I pointed out 4 examples in excess of 97%, with one at 99%. In addition, the ILCS reported a response rate of 98.5%.
Posted by: Robert | March 13, 2007 5:39 PM
"I am working on a paper about the Lancet surveys for the August 2007 Joint Statistical Meetings in Salt Lake City, Utah." -- David Kane
Kane's argument for fraud: "Assume, for a moment, that fraud occurred."
Man, what an entertaining conference that will be.
Is it taking place at Cold Fusion University (AKA University of Utah), by any chance? Or some other "top-notch" venue?
Posted by: JB | March 13, 2007 5:44 PM
Sorry, I hit the "post" button when I had intended to hit the "preview" button.
The response rates I had been pointing to were household response rates from various Demographic and Health Surveys. I had picked one country from each of the five geographic regions listed here, and all of those five had household response rates in excess of 90% -- as I said in the message immediately above, four of those five were above 97% and one was at 99.0%.
BTW, for an apples-to-apples comparison, if you calculate the Roberts (or maybe the Burnham) household response rate the same way the Demographic and Health Surveys do, I think you get 98%, not 99%.
Posted by: Robert | March 13, 2007 5:55 PM
JB: Yes, for a moment you did bring up your anti-US bias, but I brought us back on topic. And don't overestimate someone's ability to lie coherently.
Rob: Yes, 10X49X8/1849 is 2.12 but this is irrelevant since they didn't interview 1849 homes. They interviewed 2000 homes and several were disqualified because they wandered into areas that had previously been sampled. So 2.12 hours is relevant to nothing at all; Roberts rounded up from 1.96.
Tim: Now why is it so hard to understand what I am claiming here? It is not the Roberts is defrauding anyone by rounding up 4/100ths of one hour. It is that Roberts claims that the interviewers had 2 hours in the field per house [false], that they controlled for main street bias [contradicted by an interviewer], that they interviewed singly [at best a non sequitur reply about 2004 to a follow up about 2006]. One more time for the folks in the back row!
1) They interviewed in pairs; [4x10x49]/2000=.98 about 1 hour per house. He's already off by a factor of 2 in terms of effective labor. 2) He claims they actually spent 15-20 minutes interviewing per house. Did they really spend an hour in the field per house; 40 minutes travel, guesswork, risk estimation, etc. per house? If not, then even a claim of one hour in the field per house would be false.
3) Roberts is citing the total labor time for the project as if it is relevant to how much time interviewing took. Interview times are a subset of the total time, but that really doesn't speak to the question of how feasible the Lancet interview timeframe is. E.g., if the interviewers spent 3 hours on lunch every day, it really bites into interview time.
Posted by: Kevin | March 13, 2007 7:17 PM
Roberts and Burnham describing what happened after the interview of the first household in a cluster:
"Once we started, we went to the next nearest 39 doorways in a chain that typically spanned two to three blocks."
Sounds like a scheme designed to minimize travel time.
There is no reason to suppose they spent any significant time on "guesswork". Nor is there any reason to suppose they took anything like a three hour lunch. Nor do I see any reason to suppose they spent any considerable time on risk estimation.
Posted by: Rob | March 13, 2007 8:11 PM
Kevin said: "don't overestimate someone's ability to lie coherently."
Or someone's ability to put forward the same inane "disproof by incredulity" argument 20 to 30 times in one thread.
Posted by: JB | March 13, 2007 10:12 PM
"Once we started, we went to the next nearest 39 doorways in a chain that typically spanned two to three blocks."
"Sounds like a scheme designed to minimize travel time."
Indeed, I believe they consulted with world renowned mathematician Josh D ahead of time to solve the "Traveling Iraqi Interviewer" problem for them -- who did so for close to 2000 households on the back of a napkin in a matter of about five minutes (or perhaps it was 4.567 minutes -- though I doubt he had time to do it in less than 4.956, so if he claims as much, the calculation must have been fraudulent)
After working this almost miraculous feat of genius, Josh told them not to bother even leaving home because he would just claim fraud no matter what they did.
But (silly boys and girls) they thought he was just kidding and went ahead and did the study anyway.
Posted by: JB | March 13, 2007 10:27 PM
Kevin said: "if the interviewers spent 3 hours on lunch every day, it really bites into interview time."
It really bites. It really does.
I am Nomad, am Nomad. I am perfect. That which is imperfect must be sterilized. I shall continue. I shall return to launchpoint Earth. I shall sterilize Everything that is in error must be sterilized. I am Nomad. I am perfect. . . . Error. Error. Error. I shall analyze error. Analyze ... error ... Examine ... error. Error.
Posted by: JB | March 13, 2007 11:20 PM
Rob: Well discounting the fact that they state in their study summary that risk assessment, finding locations, etc. were part of the survey methodology, you are missing the point entirely. It really should be quite hard to miss because I have stated it about 5-6 times in three threads, but Roberts appears to be making claims that are full of 'truthiness' but shy on relevance.
Here he claims 2 hours in the field per house and the actual time spent interviewing he claims is 15 to 20 minutes. I don't care how they may have spent the 1 hour, 40 minutes extra he implies was spent in the field per house, the point here is the claim he offers is not relevant to the issue. If they didn't really have 2 hours per house in the field his claim is flat out wrong.
Now seriously, I put forth nothing trivial above and the best you can do is critique the marginalia of my argument? I just picked some possible examples of how to spend an hour and 40 minutes out of thin air. They weren't particularly important. The important part was three separate instances where what Roberts says is incoherent with valid argument or incoherent with the case as asserted by his own interviewer.
How else would you care to dance around the issue? Do you understand why 2.12 hours is irrelevant? Why claiming 8 interviewers is irrelevant if they worked in pairs? Why there is tension between him saying that they had interviewers try to control MSB and his interviewer denying it? Why claiming 2 hours in the field per house is either false or irrelevant? Do you understand why it seems a bit coincidental that the cited statements he offered support his case but happen to be false or irrelevant?
Posted by: Kevin | March 14, 2007 12:47 AM
JB: What exactly is incredulous about the fact that Roberts has made several important claims revolving around this survey that are not supported by either valid reasoning or his own interviewer?
I am not particularly doubtful they could have interviewed 2000 households in the stated time especially if they didn't care over much about the quality of their results and didn't take the care seemingly implied in the summary of their methodology. That doesn't lend much credence to the study results being fantastic, but I am not the least doubtful it could have happened just like that.
If you want a real example of a fallacy try the "argument from tradition" which is what many defenders of cluster studies are offering in its defense and which is invalid. It doesn't matter how many people or examples of such studies you can dig up, Roberts himself says there is little evidence proving the results of cluster studies correspond to the case. I thnk Lord Pitt said something about how habit gives vice the appearance of virtue. If cluster studies aren't too accurate for numeric representation, it doesn't matter in the least how long a tradition they have.
Posted by: Kevin | March 14, 2007 1:01 AM
Kevin wrote:
I don't think it's irrelevant at all. I thought that was one of the more surprising characteristics of the surveys -- and the key characteristic that Hicks missed in her initial criticism that the 2004 study averaged 6 minutes per household.
Posted by: Robert | March 14, 2007 2:29 AM
Robert writes:
I am sorry that the post and your comments disappeared, but that was not my call. I think that the key distinction here is the precise nature of the survey. In Lancet II, as I understand it, they picked a house, interviewed whoever was there, went to the next house and so on. Once they finished that cluster (always in a single day?), they left and did not return. I find it implausible that the household head or spouse was there 99% (contact rate) of the time.
As discussed here, ILCS was a much more thorough (and expensive survey). Interviewers when back to the selected houses again and again and again, over the course of days, if not months. In such a scenario, it does not surprise me that they got very high response rates. If you keep asking and asking and asking, someone will eventually be home and agree to participate.
So, what we need to know is the response rates for ILCS after just one attempt. I have not been able to find these. (Pointers welcome!)
If you want a sense of what "typical" contact/particpation rates in Iraq in 2006 for a single attempt survey, you should check out the results for here.
As always, I appreciate the fact that Tim provides this forum. I would appreciate any insights that others have to offer on this topic. The last thing that I want to do is to write a paper with incorrect conclusions.
I do not know of a single-contact-attempt nation-wide survey from a reputable organization (in any country, on any topic, ever) with a 98.3% or higher response rate. If you know of any counter examples, please tell me!
Posted by: David Kane | March 14, 2007 8:53 AM
I do not know of a single-contact-attempt nation-wide survey from a reputable organization (in any country, on any topic, ever) in which the interviewers stood on their heads while conducting interviews and chugged beers after every 3 houses (still standing on their heads, of course), with a 98.3% or higher response rate. If you know of any counter examples, please tell me!
Posted by: JB | March 14, 2007 10:41 AM
They stated in their summary that they spent a significant part of their time in the field doing "risk assessment" and "finding locations"?
Sounds to me like planning stage activities, but even if they were conducted in the field, just how hard do you suppose it would be to find the house next door? How much time do you suppose would be spent on assessing the risk of going to the house next door?
Perhaps you are thinking about in-between-cluster activities. Even if they spent as much as 4 hours finding a cluster and then analyzing the risk before entering it, it'd only amount to 6 minutes per household.
If the total of their man-hours in the field divided by the total number of households visited is approximately equal 2, how could his claim be "flat out wrong"?
Seems to me like you are straining at gnats.
It as if you have invested so much of yourself into Roberts and Burnham being wrong, you can't find it in yourself to give them a fair shake.
You might try this: Whenever you find what you think is an incoherent statement, go back and see if there is ANY way to interpret it in a way which is NOT incoherent.
In any event, when someone is describing parts of a complicated lengthy task in an interview, simplifications may be made that some might not find reasonable. He may make statements capable of being interpreted in more than one way. Words or phrases that could have been a part of a sentence may be left out either by accident or because it is felt that the absent bit is understood. (There are usually plenty of possible explanations for inconsistency besides bad faith.)
No, I don't, but then I can't see any reason to expect that there wouldn't be a number of in-the-field tasks that could be divied up between a group of people and then executed simultaneously.
Just off the top of my head...
In those households that took the most time(ie. those with deaths), there would be a small pile of documents to be examined. Surely, two people could do that more quickly than one.
Perhaps they interviewed two people per household independently wherever possible. That too would be done more quickly with two member teams.
I'm not familiar with this "tension". In any case, I don't see any reason to expect that his interviewer would know more about than he does.
Posted by: rob | March 14, 2007 12:52 PM