Nature has published a letter (subscription required) from Roberts and Burnham responding to this story.
In our opinion, your News story about our Lancet paper "Death toll in Iraq: survey team takes on new critics" (Nature 446, 6-7; 2007) has confused the matter rather than clarified it. You outline three criticisms of our work: that there was not enough time to have conducted the survey; that the sampling method suffered from a 'main-street bias'; and that the study team fabricated the data (the last being attributed to anonymous "researchers"). These criticisms have been previously addressed, and have little merit.
On the first point, the 1,849 interviews in 49 days described in our study suggest that 38 interviews had to be conducted each day by our eight interviewers. Although introducing themselves and explaining the confidentiality agreement might have taken interviewers several minutes, the five-question interview would take only a couple of minutes for most households that reported no deaths. The idea that eight interviewers could not conduct a total of 38 interviews in a day is not credible.
Second, we dismiss the suggestion that our sampling over-represented main streets, where car bombs are more likely. As stated in our paper (G. Burnham, R. Lafta, S. Doocy and L. Roberts Lancet 368, 1421-1428; 2006), when excluding the statistically outlying cluster of Falluja from the first report, we estimated 98,000 (95% c.i.: 8,000-194,000) excess deaths versus 112,000 (95% c.i.: 69,000-155,000) over the same period with the second survey. The first survey was done selecting random starting points with a Global Positioning System unit. The second used the random street-selection process, which is being criticized as biased. It rarely occurs in the field that two sampling methods are used allowing for comparison, and here the results are nearly identical. Moreover, there is no plausible mechanism for a significant main-street bias to operate, because only 15% of all deaths are from car bombs and other ordnance, and because most violent deaths are believed to occur away from the home.
Third, as for the accusation that researchers fabricated the data, we are ready, willing and eager to have an established international authority take a sample of the cluster forms and go to the field with our interviewers to verify the findings. Until that time, the Coalition and Iraqi governments' statements that during the first three years of occupation, Iraq's violent-death rate was lower than those of Russia, Estonia, Latvia, South Africa and Kazakhstan remain an implausible contrast with our findings.
When Nature called one of our study members in Iraq and asked if local officials joined them during the survey, that individual later clarified to Nature by e-mail that 'local officials' did not mean local clinicians and colleagues. This was inaccurately reported in the Nature summary along with a statement by our co-author that interviewers often worked alone. These points were wrongly cited as contradictions between the study team members in your News story.
All reports will eventually have "criticisms that dogged the study", if previously addressed criticisms with so little merit are given a voice in the press. -- Les Roberts & Gilbert Burnham
Nature stands by its version of the events described in the penultimate paragraph of the above Correspondence -- Editor, Nature.
Also in the news, Riyadh Lafta is giving a talk on April 20 about the study. He's giving his talk in Vancouver because the US refused to issue him a visa.
I thought the Editor's response was a little bit BS.
"[...]we are ready, willing and eager to have an established international authority take a sample of the cluster forms[...]"
And if the authority is brave enough.
Can someone define a "cluster form" for me? And how would someone sample one? And in what way is this supposed to relate to whether "researchers fabricated the data"? That whole section looks like equivocation.
Also, their comparison of L1 and L2 is misleading and evades the entire issue they are claiming to refute with it. The two studies measured extremely different phenomenon. They get an illusory similarity by conflating increased violent deaths with decreased non-violent ones in L2, misleading readers about the similarity of the two studies and evading the entire issue of MSB altogether.
They assert that these criticisms have been "previously addressed", but they've been "addressed" with misleading and evasive stuff like this, which is why these issues very rightly continue to "dog the study".
The claim that "15%" is car bombs and ordinance is also misleading and evasive. It's misleading because it's more like 27% of violent deaths (they conflate all non-violent with violent again to bring it down to 15%). And it's an evasive straw man because MSB does not say this would only be limited to car bombs or ordinance.
And this is a falsehood: "the Coalition and Iraqi governments' statements that during the first three years of occupation, Iraq's violent-death rate was lower than those of Russia, Estonia, Latvia, South Africa and Kazakhstan remain an implausible contrast with our findings."
As usual, they give no specific citation for whatever "statements" they're constructing these factoids with. Maybe they cherry-picked some "statement" from somewhere to make it work, who knows, but it's not true of the figures I've seen from the Iraqi gov.
And who can make sense of that last paragraph about "local officials"? That's not what was asked or answered according to Giles. This looks like just equivocation again.
The response from Nature at the end seems to be about what this letter deserves.
Josh, presumably a "cluster form" is the form used to record results from each cluster and replicated the study in a sample of the clusters would allow you to check the information provided on the original forms.
Ian, I guess I'll take your word for it. But in their letter to Science they write:
Those who work in conflict situations know that checkpoints often scrutinize written materials carried by those stopped, and their purpose may be questioned. Unique identifiers, such as neighborhoods, streets, and houses, would pose a risk not only to those in survey locations, but also to the survey teams. Not including unique identifiers was specified in the approval the study received from the Johns Hopkins Bloomberg School of Public Health Committee on Human Research. At no time did the teams "destroy" details, as Bohannon contends. Not recording unique identifiers does not compromise the validity of our results.
So of what use are their "cluster forms" for finding out if anyone "fabricated the data" if those forms don't identify the neighborhoods, streets or houses sampled?
Can you source or verify the claim that Prof. Riyadh Lafta was denied a U.S. visa? Him being one of the Lancet Study authors, this should be taken as an outrageous case of political censorship, and is highly newsworthy - less-interesting visa denials have made headlines before, if only briefly. There's nothing on google news - is this possibly a false rumor, or are we just very early in the story?
The original source (following the links) is purportedly an official email, but posted on an anonymous blog; can anyone validate it?
http://newstons.wordpress.com/2007/04/07/over-one-half-million-deaths-i…
-a scienceblogs lurker
Good question Thomas - one I should have asked before posting the link. I'm checking it out and will post again if I learn anything further. This link confirms the event itself but says nothing about refusal of a visa:
http://apps.sphcm.washington.edu/epi/cal/cal.asp
The fact that he is collaborating with U Washington people but giving his talk at U Washington by video link does seem to confirm the visa story.
Certainly the circumstantial evidence fits. But then again Dr Lafta might have heard Senator McCain saying that Baghdad market reminds him of a farmers' market in Indiana and decided that the USA is a dangerous place.
The only named source so far is a Kate Armstrong who "holds a position in the School of Interactive Arts and Technology and the Faculty of Applied Science at Simon Fraser University" and apparently published the story here. The internets being what they are, you wouldn't want to take anything for granted.
[quote]
As usual, they give no specific citation for whatever "statements" they're constructing these factoids with.
[/quote]
Are you criticizing the study or your own post?
I have to say I was a bit disappointed in the way the authors dealt with the MSB issue here. In this case, I think that Josh is correct in his criticism.
It *does* look like there is a discrepancy in the number of violent deaths during the period that the studies overlap. What is unclear is how much that has to do with MSB or any other issues, for example confidence intervals over these point estimates. Anyway, it's just not as cut and dried as they are trying to make out, to my mind.
Incidentally, I also didn't really understand the business about local officials... anyone have any answers?
"After being asked by Nature whether even this system allowed enough time, author Les Roberts of Johns Hopkins said that the four individuals in a team often worked independently. But an Iraqi researcher involved in the data collection, who asked not to be named because he fears that press attention could make him the target of attacks, told Nature this never happened. Roberts later said that he had been referring to the procedure used in a 2004 mortality survey carried out in Iraq with the same team (L. Roberts et al. Lancet 364, 1857-1864; 2004)."
So here is what Nature is sticking to. They asked Roberts if even going in pairs would work; he said they went singly and upon being told he had been contradicted, retracted his claim and said he was talking about 2004. Now Burnham has said they "sometimes" went singly. If that is the case, why did Roberts change his story to Nature upon being apparently contradicted and why is Nature sticking to their guns?
"The Iraqi interviewer told Nature that in bigger towns or neighbourhoods, rather than taking the main street, the team picked a business street at random and chose a residential street leading off that, so that peripheral parts of the area would be included. But again, details are unclear. Roberts and Gilbert Burnham, also at Johns Hopkins, say local people were asked to identify pockets of homes away from the centre; the Iraqi interviewer says the team never worked with locals on this issue."
This issue remains a blatant contradiction between the authors and Nature and is apparently a claim Nature still supports.
Further, looking at the claim made in their reply to Nature:
"On the first point, the 1,849 interviews in 49 days described in our study suggest that 38 interviews had to be conducted each day by our eight interviewers. Although introducing themselves and explaining the confidentiality agreement might have taken interviewers several minutes, the five-question interview would take only a couple of minutes for most households that reported no deaths. The idea that eight interviewers could not conduct a total of 38 interviews in a day is not credible."
Why are they talking about what is "suggested" by the implications of their claims rather than simply stating what happened? Shouldn't they know? Why the circuitous language? Why only talk about how long the interviewing would take in households reporting no deaths and in such indefinite language? Their interviewers didn't record the length of the interviews?
And for all of the fans of the recently invented phrase 'disproof by incredulity,' let's hear the ringing condemnations of the Lancet authors use of same in rebutting the attacks on their timeline.
I can verify that I received that same email on two different University of Washington Health Sciences mailing lists.