Tim Lambert (deltoidblog AT gmail.com) is a computer scientist at the University of New South Wales.
Back in November 2001 Neil Munro was an advocate of war with Iraq and predicted:
The painful images of starving Iraqi children will be replaced by alluring Baghdad city lights, smiling wages-earners and Palestinian job seekers.
Iraq war advocates like Munro don't like the results of the Lancet study that suggest that about 600,000 Iraqis have died as a result of the war they championed. So Munro has written a piece that throws every piece of mud he can find at the study, and to their discredit, the National Journal has published it. And if you think I'm being unfair by stressing Munro's obvious bias, Munro has 1400 words about how the Lancet researchers were ideologically biased against the Iraq war and were funded by the Great Satan himself, George Soros. Presumably Munro thinks that studies on cancer are brought into question because the researchers are anti-cancer and funded by anti-cancer foundations.
And of course Munro's fellow war supporters amongst bloggers such as Glenn Reynolds, Charles Johnson and Gateway Pundit seized on his article as proof that the study was a fraud or at least had serious problems. The serious problems are, however, with Munro's article with fabricated claims, misrepresentation of sources, a misleading graph, and reliance on discredited arguments.
Munro writes:
Even Garfield, a co-author of the first Lancet article, is backing away from his previous defense of his fellow authors. In December, Garfield told National Journal that he guesses that 250,000 Iraqis had died by late 2007. That total requires an underlying casualty rate only one-quarter of that offered by Lancet II.
I contacted Garfield and this is a misrepresentation of his views. He told me:
I seem to have a special ability to make statements that lend themselves to misinterpretation.
I could not believe that 100,000 had died in 2004, but the best evidence made me believe it.
As a guess, out of the blue, I feel confident that at least a quarter million Iraqis have died due to violence since the 2003 invasion. But that is just a guess.
An estimate, based on field data, collected via good methods, is far better than a guess, even if there are some biases along with imprecision in it.
He is not backing way from his statement in 2006 that:
I am shocked that it is so high, it is hard to believe, and I do believe it. There is no reasonable way to not conclude that this study is by far the most accurate information now available.
Munro claims:
The survey teams failed to collect the fraud-preventing demographic data that pollsters routinely gather.
This is an obvious fabrication. Munro even provides a copy of the survey instrument, which tells the surveyor to record demographic data:
Who lives in this household? (Resident means spent most of the past 3 months sleeping in this household.) (only record M/F and the age, if less than 4 years, record age in months)
Correction: I was wrong. While the plan was to collect ages (hence the instructions above), during the survey this was dropped to speed things up. Les Roberts explains here. My apologies to Neil Munro.
Munro presents a highly misleading graph (shown on right) that compares the Lancet estimates of violent deaths with the IBC count of the civilian deaths reported by the media. Not all deaths are reported by the media, but the graph deceptively presents the IBC number as an estimate of violent deaths.
Munro presents a list of what he claims are "potential problems". Long-time readers will be familiar with most of these alleged problems, but for the sake of completeness I'll go through the entire list.
Munro:
Still, the authors have declined to provide the surveyors' reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent -- and repeatable -- to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers.
This is deceitful. Munro is well aware that IRB rules require that the identity of respondents be kept confidential and that breaking such rules are not "customary scientific practice". We know this because he has a sidebar alleging that the Lancet team broke IRB rules by recording names of respondents.
Munro:
Sample size. The design for Lancet II committed eight surveyors to visit 50 regional clusters (the number ended up being 47) with each cluster consisting of 40 households. By contrast, in a 2004 survey, the United Nations Development Program used many more questioners to visit 2,200 clusters of 10 houses each. This gave the U.N. investigators greater geographical variety and 10 times as many interviews, and produced a figure of about 24,000 excess deaths -- one-quarter the number in the first Lancet study. The Lancet II sample is so small that each violent death recorded translated to 2,000 dead Iraqis overall.
This is just a rehash of Steven E Moore's innumerate criticism. When sampling, you do not need a larger sample when the population is larger. Steve Simon explains
The best analogy I have heard about sampling goes something like: "Every cook knows that it only takes a single sip from a well-stirred soup to determine the taste." It's a nice analogy because you can visualize what happens when the soup is poorly stirred.
And you just need a single sip whether it's a large vat or a small pot of soup. Munro does not understand the most basic thing about sampling and despite apparently spending months on his story, failed to learn it. Rebecca Goldin of stats.org is also unimpressed with Munro's understanding of statistics.
Munro's comparison of the 2004 UNDP survey with Lancet I is misleading. He failed to mention that that the UN survey covered a different time frame and measured something different. If you look at their measurements of the same thing, the UNDP and Lancet 1 agree.
Munro:
"Main street" bias? According to the Lancet II article, surveyors randomly selected a main street within a randomly picked district; "a residential street was then randomly selected from a list of residential streets crossing the main street." This method pulled the survey teams away from side streets and toward main streets, where car bombs can kill the most people, thus boosting the apparent death rate, according to a critique of the study by Michael Spagat, an economics professor at the Royal Holloway, University of London, and Sean Gourley and Neil Johnson of the physics department at Oxford University.
Even if this "main street bias" exists (and it's likely that it's just a poorly worded sentence in the paper), it makes no significant difference.
Munro:
Oversight. To undertake the first Lancet study, Roberts went into Iraq concealed on the floor of an SUV with $20,000 in cash stuffed into his money belt and shoes. Daring stuff, to be sure, but just eight days after arriving, Roberts witnessed the police detaining two surveyors who had questioned the governor's household in a Sadr-dominated town. Roberts subsequently remained in a hotel until the survey was completed. Thus, most of the oversight for Lancet I -- and all of it for Lancet II -- was done long-distance.
Roberts did not need to go to Iraq for Lancet II, since he was satisfied that Lafta could properly conduct the survey.
Munro:
To Kane, the study's reported response rate of more than 98 percent "makes no sense," if only because many male heads of households would be at work or elsewhere during the day and Iraqi women would likely refuse to participate.
To my knowledge, David Kane has never conducted a door-to-door survey in Iraq or anywhere else. Why is his uninformed opinion presented by Munro? Oh right, he said something that suited Munro's agenda.
Lack of supporting data. The survey teams failed to collect the fraud-preventing demographic data that pollsters routinely gather.
As noted earlier, this is an outright fabrication by Munro.
Munro:
Death certificates. The survey teams said they confirmed most deaths by examining government-issued death certificates, but they took no photographs of those certificates.
I can't help but be impressed with the why Munro feigns ignorance of the IRB rules he put tin the sidebar.
Under pressure from critics, the authors did release a disk of the surveyors' collated data, including tables showing how often the survey teams said they requested to see, and saw, the death certificates. But those tables are suspicious, in part, because they show data-heaping, critics said. For example, the database reveals that 22 death certificates for victims of violence and 23 certificates for other deaths were declared by surveyors and households to be missing or lost. That similarity looks reasonable, but Spagat noticed that the 23 missing certificates for nonviolent deaths were distributed throughout eight of the 16 surveyed provinces, while all 22 missing certificates for violent deaths were inexplicably heaped in the single province of Nineveh. That means the surveyors reported zero missing or lost certificates for 180 violent deaths in 15 provinces outside Nineveh. The odds against such perfection are at least 10,000 to 1, Spagat told NJ.
Well, he may have told NJ that, but the only he could do such a calculation is if he made some unwarranted assumptions that missing certificates would have the same distribution for violent and non-violent deaths. Furthermore, you can always find low probability patterns in any data. For example, I just rolled a die four times and got the sequence 6426. Notice how the number goes down by two each time (wrapping around when it goes to 0), The odds of this happening are 215 to 1 against. But while this particular pattern is unlikely, there are many many patterns, so it almost certain that I can find some pattern to fit any sequence.
Munro:
Suspicious cluster. Lafta's team reported 24 car bomb deaths in early July, as well as one nonviolent death, in "Cluster 33" in Baghdad. The authors do not say where the cluster was, but the only major car bomb in the city during that period, according to Iraq Body Count's database, was in Sadr City. It was detonated in a marketplace on July 1, likely by Al Qaeda, and killed at least 60 people, according to press reports.
The authors should not have included the July data in their report because the survey was scheduled to end on June 30, according to Debarati Guha-Sapir, director of the World Health Organization's Collaborating Center for Research on the Epidemiology of Disasters at the University of Louvain in Belgium.
This is ridiculous. The survey was designed for 50 clusters. Even if the survey was scheduled to end by June 30, it makes no sense to stop then if some provinces were unsampled.
The Cluster 33 data is curious for other reasons as well. The 24 Iraqis who died violently were neatly divided among 18 houses -- 12 houses reported one death, and six houses reported two deaths, according to the authors' data. This means, Spagat said, that the survey team found a line of 40 households that neatly shared almost half of the deaths suffered when a marketplace bomb exploded among a crowd of people drawn from throughout the broader neighborhood.
Is it possible that a group of people from the same street travelled to the marketplace together for protection? Iraq is a dangerous place, you know.
After this, Munro goes on at length about the "ideological bias" against the Iraq war of the people connected with the survey. He states:
Whether this affected the authors' scientific judgments and led them to turn a blind eye to flaws is up for debate.
Do you think that perhaps Munro's spirited advocacy of the war affected his journalistic judgments and led him to turn a blind eye to flaws in the criticisms he printed?
I asked Munro to explain how someone such as himself, with an evident bias towards presenting the war as a success was chosen to write the piece, and why his war advocacy was not disclosed. He evaded the question, writing:
We're a journal of fact and politics, not opinion, and that's why we printed the facts about Soros' money, Lafta's Saddam-era articles, the claim of 15,000 dead from U.S. vehicles, Roberts' description of Lafta's views, Spagat & Kane's claims about data-heaping, etc.
It is rather telling that, to Munro, the most damning "fact" is "Soros' money". And if you wondering about "the claim of 15,000 deaths from U.S. vehicles", neither Lancet study makes any such claim. Perhaps Munro's eyes were dazzled by those alluring Baghdad city lights.
Update: Burnham and Roberts reply to Munro:
The overwhelming confirmatory evidence of the Lancet study findings, the conventional nature of our survey procedures, and the abundance of internal consistencies in the data of which Mr. Munro was informed and chose not to report, suggests that National Journal's critique of our work should itself be examined for political motivations.
Politics
Comments
"Lafta's team reported 24 car bomb deaths in early July, as well as one nonviolent death, in "Cluster 33" in Baghdad. The authors do not say where the cluster was, but the only major car bomb in the city during that period, according to Iraq Body Count's database, was in Sadr City."
Alternately, the fatalities might have come from several smaller incidents.
Munro (unless this is one of the instances where he's claiming fraud) is claiming that of the 60 people killed July 1, 24 or almost half happened to live in the cluster selected for the study.
Posted by: Ian Gould | January 9, 2008 2:34 PM
While most of Munro's points do in fact seem to be complete crap, the point about funding from Soros is reasonable (if for example, the study had found the opposite and had been funded by some right wing billionaire I suspect we'd all be quick to all that out)
Posted by: Joshua Zelinsky | January 9, 2008 3:08 PM
I wondered whether you would bother with that article. It's such a load of shit I thought it might be better to ignore it. It really looks like a kind of swift-boating exercise to me - the whole idea is to put the targets in the position of having to deny that they copulate with pigs.
One particularly nauseating detail is the way they try to portray Lafta's researchers as being first Baathists, on the strength of his having been (they say) "a child-health official in Saddam Hussein's ministry of health", then Sadrists on the strength of the Mahdi Army controlling the health ministry, "which employed some of Lafta's researchers." (Oddly enough, this is the same Ministry of Health which becomes such a trustworthy source when it produces suitably modest death totals.) Who the hell do they think Lafta is, some latter-day Talleyrand, equally at home with theocrats and revolutionaries? But then to the likes of Glenn Reynolds and Charles Johnson all these Iraqis look much the same.
Posted by: Kevin Donoghue | January 9, 2008 3:09 PM
There is a lot to discuss here, but let us start with the most important issue. Tim claims that Munro is guilty of a "fabrication" when Munro writes that "The survey teams failed to collect the fraud-preventing demographic data that pollsters routinely gather."
But this is true! I have examined the data. Have you, Tim?
For background, it is standard in a survey to collect demographic data so that you can check that your survey sample matches up with the population, especially in a case in which you are worried about inexperienced interviewers. So, for each household, you should tabulate something like:
1) male, age 46 2) male, age 20 3) female, age 10 ...
Sex and age are the minimum background demographic variables that you should collect. Whatever the survey document says, the Lancet interviewers did not do this.
I think that Tim owes an apology to Munro.
Posted by: David Kane | January 9, 2008 3:48 PM
David Kane -- I would prefer to think that you are simply mistaken, and thinking of something different to what is being discussed here. The alternative is that you are deliberately lying.
The data collected for the Lancet II study includes age, sex, and household size. Explicitly. There in words and numbers. So what on earth are you talking about?
Posted by: Luna_the_cat | January 9, 2008 3:59 PM
With the Democratic candidates talking about ending the war I wonder if we will get a chance to see what that county is actually like, sans war? Dave Briggs :~)
Posted by: Dave Briggs | January 9, 2008 4:23 PM
1) male, age 46 2) male, age 20 3) female, age 10 ...
dear David, telephone surveys have become all too popular. i get a call at least every other month. i can not remember, that i was EVER asked to provide the information about my household, that you describe above. actually i am pretty sure that i would NOT provide exact age of all household members to a survey.
i can remember the last phone survey. i was asked whether i smoke. what age CATEGORY i belong to. how many household memebrs are living with me and how many of those smoke.
looks pretty close to the survey sheet of the lancet, doesn t it?
Posted by: sod | January 9, 2008 4:29 PM
David Kane, in Munro's article: "The authors refuse to provide anyone with the underlying data"
David Kane, in comment #4: "I have examined the data. "
Posted by: Tim Lambert | January 9, 2008 4:35 PM
Er, yes. Indeed.
David Kane, you are a seriously confused individual.
Posted by: Luna_the_cat | January 9, 2008 4:39 PM
David Kane: "I have examined the data."
Now why do I not find that reassuring?
Posted by: Kevin Donoghue | January 9, 2008 4:41 PM
Tim Lambert, I do not envy you your labors.
We need to constantly make our cross-references and debunking of the denialism network more sytematic. The day may come when they can't make lies up more quickly than we can refute them.
Posted by: Marion Delgado | January 9, 2008 5:11 PM
1) I agree that the quote Munro used from me about "underlying data" was somewhat out of context. Details here. Do you really want to go into the details here? I am happy to! No one, outside the authors, has seen the underlying data (not cluster summaries) for L1. And, as of now, the data is "unavailable." No one, outside the authors, has seen all the (non-person identifying) underlying data for L2. (I have seen some of the data. Spagat and others have not. For example, we have yet to see the data, which Roberts has been promising to Fritz Scheuren for months, which allows us to compare data across interviewers.) I agree that Munro's quote of me is not clear because the words "the data" are not very specific, but the facts are still damning.
2) Again, Tim, you claim that something is a "fabrication" that is simply true. Do you claim that age/sex information for the households was collected? If not, what is wrong with Munro's statement?
There are, of course, other topics to debate here. But if we can't even agree as to whether or not age/sex information was collected for the 1849 households, we aren't going to make much progress on the more complex issues.
Posted by: David Kane | January 9, 2008 6:32 PM
I'm guessing that "unavailable" is probably simply unavailable to you and organisations like the Cato Institute, based on one of the conditions of release:
"1. The data will be provided to organizations or groups without publicly stated views that would cause doubt about their objectivity in analyzing the data."
That doesn't mean "unavailable to everyone", though. Truly...
And, explicitly from the first Lancet study (quoted without permission, but under terms of fair use):
"If the household agreed to be interviewed, the interviewees were asked for the age and sex of every current household member. Respondents were also asked to describe the composition of their household on Jan 1, 2002, and asked about any births, deaths, or visitors who stayed in the household for more than 2 months. Periods of visitation, and individual periods of residence since a birth or before a death, were recorded to the nearest month."
This was precisely the methodology used for the second study, as well. Ages and gender information were collected.
Posted by: Luna_the_cat | January 9, 2008 6:56 PM
That Soros funded the Lancet study is relevant while the cancer analogy is not. His ideological leanings could lead him to endorse a sloppy study that grossly overstates the number of deaths.
The Lancet death numbers are so unbelievably high that it appears the only people who believe them are those with an ideological agenda. They want to believe the study, so no matter how unreliable it is, they do.
Posted by: Charles H. | January 9, 2008 6:58 PM
I have to ask, Charles H. (post #14), what criteria do you use to state that "high numbers" are unbelievable?
Posted by: SpotWeld | January 9, 2008 7:01 PM
SpotWeld, let's phrase the question another way,
"Why do you find the Lancet study believable when not a single other study confirms its findings?"
Posted by: Charles H. | January 9, 2008 7:09 PM
Charles H.> I have not made any assertions about the Lancet study, only asked you to state the reasoning for yours. Is there an upper bound by the numbers of deaths would have to be limited to, to be believable. And if so, how did you arrive at that value?
Posted by: SpotWeld | January 9, 2008 7:14 PM
I have no idea if demographic data was collected for L1. I know that it was not for L2, and so Tim owes Munro an apology over the word "fabrication."
And, not to derail the thread further, but please note the way that I ended my TCS Daily article (after praising Tim).
A fair observer, whatever his opinions of the accuracy of L2, needs to be prepared to update his estimates when new information, like this, comes to light. I used to think that a good estimate was somewhere between Jon Pedersen's 100,000 and L2's 600,000. I am now revising that estimate downward. What are the L2 supporters doing?
Posted by: David Kane | January 9, 2008 7:15 PM
Uh, what about the ORB survey (http://www.opinion.co.uk/Newsroom_details.aspx?NewsId=78)? Or the work by Zogby International? They confirm that the numbers are credible.
Posted by: Luna_the_cat | January 9, 2008 7:24 PM
Thanks for commenting on this Tim. Hadn't realised the UNDP & Lancet were comparing apples and oranges.
David, what is your evidence that demographic data was not collected, when the paper (as quoted in post 5 & 13 by luna the cat) specifically says that it was?
Posted by: James | January 9, 2008 7:34 PM
David Kane,
Per here and here, how is the publication of your critic of the Johns Hopkin study published in the Lancet coming? It has been six months since you wrote telling everyone that it is under review. When can we expect to see it in print?
Posted by: hardindr![[TypeKey Profile Page]](http://scienceblogs.com/deltoid/nav-commenters.gif)
|
January 9, 2008 8:01 PM
David Kane still doesn't have any proof that No one, outside the authors, has seen all the (non-person identifying) underlying data for L2, because he still has no proof that the cluster-level data cannot be used to identify individuals. Only the authors and data-collecting agencies have that proof, and naturally they can't share it.
I have worked on national surveys in a developed country and I am pretty confident that there is no way David Kane (or anyone else) can get access to that raw data without being connected to the study. Yet DK still makes this claim about much more sensitive data. It's extremely dishonest, and nasty too for coming as it does with the implied fraud allegations.
Posted by: SG | January 9, 2008 9:07 PM
SG,
What exactly are we disagreeing about?
1) It is a fact that the Lancet authors share data with some critics (like me) but not others (like Spagat). I can not find a single example of similar behavior relating to a peer-reviewed publication. Can you?
2) It is a fact that the authors related data at a certain level for L2 while refusing to release data at the same level for L1.
3) It is a fact that Les Roberts promised Fritz Scheuren at the JSM that he (Roberts) would produce data on which interviewers provided which data. (Not with names! Just interviewer A did interviews 1, 2, 3; interviewer B did interviews 4, 5, 6; and so on.) I believe that Roberts has not done so.
Those are the problems. If you disagree with these facts, please explain why.
Posted by: David Kane | January 9, 2008 9:16 PM
17 by SpotWeld
"Is there an upper bound by the numbers of deaths would have to be limited to, to be believable."
Probably not, the Lancet study could have said 500,000,000 have died in the war in Iraq and there would be posters here defending the study.
Posted by: Charles H. | January 9, 2008 9:36 PM
Tim writes: "This is an obvious fabrication. Munro even provides a copy of the survey instrument, which tells the surveyor to record demographic data."
But the Iraq Mortality Survey Template that Tim links to includes the ungrammatical instruction: "only record M/F and the age, if less than 4 years, record age in months," which could be interpreted as an instruction to ask the age of children only.
If we look at the "Iraq Mortality Survey Questionnaire (actual)," for which Tim neglects to provide a link, we find that age is only collected for children and the dead, but not most household members.
So it seems Tim owes Munro an apology, and owes the rest of us a correction.
Posted by: Ragout | January 9, 2008 10:05 PM
I'm looking forward to Tim's comments on the new NEJM study which finds "that the 2006 study by Burnham et al. considerably overestimated the number of violent deaths." The new study, which uses considerably better methods than did Roberts et al (such as best-practice methods of selection a random sample) finds 150,000 violent deaths after the US invasion of Iraq.
Posted by: Ragout | January 9, 2008 10:11 PM
Further to the admittedly secondary issue of car bomb deaths in Baghdad, during the last two weeks of June 2006 a minimum of 80 deaths by car bomb were reported by IBC in Baghdad.
Posted by: Ian Gould | January 9, 2008 10:21 PM
"The new estimate of 151,000 violent deaths from March 2003 to June 2006 is three times higher than the number of casualties recorded for the same period by Iraq Body Count, a nongovernmental organization in London, which reports only the violent deaths of civilians and only documented deaths.
The WHO survey teams were unable to visit 115, or 11 percent, of the "clusters" or areas they had originally targeted because of insecurity. Moreover, the movement of families away from their homes because of violence can also lead to underreporting of deaths, they said. As a result of such uncertainties, researchers calculated that the number of Iraqis who died violently in the survey period could range from 104,000 to 223,000."
The new study focuses on violent deatsh not total excess deaths, the measure used by the Lancet study.
Past critics of the Lancet study who relied on the IBC who now want to rely on the NEJM study have to start by admitting that the IBC was a serious undercount. The question is whether it undercounted deaths by a factor of three or a factor of ten.
Can anyone recall offhand the lower confidence interval of the Lancet study?
I look forward to ferocious forensic analysis of the current study including deamnds for death certificates, accusations of fraud; claims that all them ay-rabs is liars et cetera.
Posted by: Ian Gould | January 9, 2008 10:30 PM
Ragout,
The study to which you refer estimates "violent" deaths. The Lancet/John Hopkins studies estimate "excess" deaths from all causes. You are comparing a subset to a whole set.
Your 'could be interpreted' version of the survey template might be a plausible inference only if one selectively quotes the entry as you did by omitting the prefacing question to which those notations refer:
"Who lives in this household? (Resident means spent most of the past 3 months sleeping in this household.)"
How does it feel to be so intellectually dishonest?
Posted by: luminous beauty | January 9, 2008 10:44 PM
The study to which you refer estimates "violent" deaths. The Lancet/John Hopkins studies estimate "excess" deaths from all causes. You are comparing a subset to a whole set.
Actually, it was the NEJM researchers -- not me -- who said that Lancet II "considerably overestimated the number of violent deaths." So, take it up with them
Before you write a letter to the NEJM, though, you might want to read the Lancet II study, which reports 600,000 violent deaths, a number considerably higher than the 150,000 found in the NEJM study.
Posted by: Ragout | January 9, 2008 10:56 PM
luminous beauty,
Read the "actual" questionnaire not the "template." You'll see that the actual questionnaire has no space to record the ages of household members (except for children and the dead).
No doubt their failure to collect age data explains why Roberts et al -- unlike the higher-quality NEJM and ILCS studies -- haven't published the age distribution of their sample.
Posted by: Ragout | January 9, 2008 11:00 PM
How does it feel to be so intellectually dishonest?
You could always look in the mirror or simply ask Gouldiechops to supply a pic. That ought to help.
Hey gouldiechops,
So you think the Iraq death toll is now approaching the 2 million mark on the Roberts' "analysis". You there are 750 people a day dying in Iraq as a result of the war according to Roberts.
Ragout:
The number would be approaching 2 million at the present time if we used the top end of Roberts scholerly work. I never knew Roberts' hired the equivilent of a Dr. Mengels assistant to help him figure it all out. LOl
Posted by: jc | January 9, 2008 11:01 PM
David Kane
1) yes, this sort of thing happens a lot. I, personally, have seen it happen in a variety of situations.
2)You can keep repeating this, but until you address the possibility that the authors cannot release the data because of possible identification of individuals, it remains a deliberately disingenuous criticism. If I sample 100 people in 20 clusters about a sensitive topic in 2003, then 1000 people in 20 clusters about the same topic in 2005, I am perfetly within my rights to release different levels of data from the two surveys. You know this, why keep acting like it's fraud?
3) Not only is what you believe often irrelevant to the truth, but the researcher is under no obligation to provide any information about the reviewers to a researcher whose first and continuing criticism of the paper was "fraud".
Now, David, so that we can be sure you aren't pushing an agenda, you need to do the following post-haste:
1) demand the NEJM data immediately, at its most detailed level 2) demand the NEJM interviewers' records immediately, at their most detailed level 3) if they aren't provided immediately, claim fraud 4) check every RR and CI in that paper using the method you used for the lancet paper, so that you can confirm with your (wrong) first year psych stats that there is no conflict between normally-distributed estimates and ratio estimates 5) check every other calculation 6) if you find ANY errors, claim fraud
Have you done this yet? I recall that when the Lancet survey came out you were making fraud accusations before you had taken any of these steps.
Posted by: SG | January 9, 2008 11:52 PM
Ragout, I think you are misunderstanding the word "template". The "template" Tim linked to describes the protocol for conducting the study. It is not the tool for recording final information about deaths, which is the "actual" instrument you linked to. The interviewer will follow the template, starting by introducing the consent form and then checking household composition, etc. Finally they will use the "actual" data sheet to record the details of those who died. The data sheet is linked to the household (whose descriptive data is recorded in the consent form, I suspect) by the variables "governate", "cluster number", "household number".
Quite often in surveys like this the information on the person contacted, the household composition, might be recorded on, for example, a clipboard (where empty households and refusals are also recorded), and then the detailed data for the households who agree to participate is recorded in the "actual" sheet. Or did you think that every householder who refused to participate has one of those "actual" sheets filled out, but blank?
(I have myself recruited for surveys in this way, have instructed others working with me to do this, and have seen and analysed data collected in this way. It's hardly a mystery).
I suspect this is a deliberate misrepresentation by that magazine, especially the use of the words "template" and "actual".
Posted by: SG | January 10, 2008 12:05 AM
"I have worked on national surveys in a developed country and I am pretty confident that there is no way David Kane (or anyone else) can get access to that raw data without being connected to the study. Yet DK still makes this claim about much more sensitive data. It's extremely dishonest, and nasty too for coming as it does with the implied fraud allegations."
How many of those surveys were funded by Soros? How many came out right before the American elections? How many could influence the outcome of the American elections? How many of those surveys were conducted by researchers who had an axe to grind with the current American administration?
I'm thinking none.
Posted by: ben | January 10, 2008 12:56 AM
Ben, if the survey I worked on wasn't politically charged and wasn't funded by a political enemy of the government, but my survey had some of the same methods or policies as the one Soros funded, doesn't that suggest something to you about the Soros-funded survey? I don't think your logic or your insinuation or whatever it is is entirely clear...
Posted by: SG | January 10, 2008 1:33 AM
SG, no other group or organization has a death tally for Iraq approaching anywhere near the total the Lancet study does.
Interestingly, those with less of a political ax to grind also have death figures that are at least in the realm of plausibility.
Posted by: Charles H. | January 10, 2008 1:51 AM
Yes Charles H, that's right. And the correct approach is not to immediately dismiss the highest number by claiming fraud (as David Kane does) or ignoring the understanding of survey sampling you previously claimed to have (as Ragout does) so you can blatantly misrepresent a survey instrument. The correct approach is to compare the studies and ask why the difference arose.
And why exactly is it that supporters of the invasion of Iraq have "less of a political ax to grind" than those who oppose it? Exactly who is it gets to decide which axe is bluntest, or being sharpened the most? Do you? Do I?
Posted by: SG | January 10, 2008 1:58 AM
"SG, no other group or organization has a death tally for Iraq approaching anywhere near the total the Lancet study does."
This is technically correct seeing as the ORB survey came up with an even higher estimate.
http://www.npr.org/templates/story/story.php?storyId=14501232
Posted by: Ian Gould | January 10, 2008 2:06 AM
"This is technically correct seeing as the ORB survey came up with an even higher estimate."
ORB is what? A polling agency? From Britain? Their figures say 1,200,000 Iraqis murdered? And I'm sure you find their poll accurate too, no?
ORB partnered with an Iraq company, www.iiacss.org, which appears to be a phantom company, existing on the web only. Only a fool would accept the polling data done by an apparently non-existent company.
The only thing more comical thing then the ORB poll would be to find out that Soros paid for that too. I wonder if he did.
Thanks for the laugh Ian, I needed one today.
Posted by: Charles H. | January 10, 2008 2:48 AM
Ah, clearly Charles H gets to decide whose axe is sharpest. No biasses at all, eh, Charles?
Posted by: SG | January 10, 2008 3:30 AM
Everyone has biases SG, I at least try to be aware of my own instead of parading them here for all to see.
Posted by: Charles H. | January 10, 2008 3:35 AM
you certainly aren't parading them for all to see...
Posted by: SG | January 10, 2008 3:57 AM
I don't understand what Ragout is on about "The new study, which uses considerably better methods than did Roberts et al (such as best-practice methods of selection a random sample)". The NEJM study has 10% of its clusters censored, probably informatively (as in, unable to be surveyed because too dangerous), and the death rates cobbled together from comparisons with the IBC data. That's not exactly a trivial problem.
On the other hand, the NEJM study does look pretty well done, and like the Lancet study, it rejects the null that the violent death rate didn't increase with a very high degree of confidence and rejects the null that the death count is only what the IBC say it is at a fairly high degree of confidence. So I'm prepared to accept it as, like the Lancet study, a reasonable qualitative assessment of what things have been like in Iraq (ie, a disaster).
Posted by: dsquared | January 10, 2008 4:21 AM
Ben continues with his hilarity when he asks:
"How many [of the studies critical of the Iraq war] came out right before the American elections? How many could influence the outcome of the American elections?
For heavens sake man! Wake up! What difference would it have made if Kerry had won over Bush? Both are imperial war hawks; in the US you don't have any real opposition to the current corporatocracy because both parties are utterly pro-establishment. Like the mainstreram media, they are totally beholden to the agenda of those with concentrated wealth and power. US elections stopped being waged on policy years ago; nowadays its personality and nothing more. The current crop of 'Democrats' is no exception. There's one party in the US: the Property Party, with two right wings.
What is most remarkabale are those idiots out there who are using whatever means they can to defend the indefensible; an illegal war (aggression) that has largely reduced a country to rubble, led to mass death and generated a humanitarian catastrophe. A war fought entirely on economic and political grounds using the same old chestnut - western 'security' - as an excuse. A war that gives complete creedence to Naomi Klein's hypothesis of a 'disaster capitalism' complex that is alive and well.
Let me ask the dimwits out there this: how would 150,000 civilian deaths from an illegal invasion of the United States by a foreign power over a 3-4 year span be received by the punditocracy in the US?
Posted by: Jeff Harvey | January 10, 2008 4:51 AM
I also don't understand what Ragout is on about with "only record M/F and the age, if less than 4 years, record age in months," which could be interpreted as an instruction to ask the age of children only. ...this could be interpreted in this fashion only through deliberate stupidity, as far as I can tell.
The instruction is to record age and gender for the household members, and if a household member is under the age of 4, to record age in months rather than years. This is immediately clear to anyone with experience of surveys, however ungrammatical Ragout finds the comma.
I also wonder why Charles H doubts the existance of the Independent Institute for Administration and Civil Society Studies. They've been around since 2003; they frequently work with universities (including the University of Michigan and a couple of German universities), and it is possible to contact both employees and former employees now living elsewhere, such as http://www.lightstalkers.org/bashar_ali. I can also tell you that IIACSS holds a fair number of software licences, including Cisco, .Net, and SPSS (I know this because it is possible to access software profiles of Microsoft Registered Partners online). They also have landline phone numbers registered. Charles H can't have looked very hard for IIACSS existance.
Posted by: Luna_the_cat | January 10, 2008 6:32 AM
Let me ask the dimwits out there this: how would 150,000 civilian deaths from an illegal invasion of the United States by a foreign power over a 3-4 year span be received by the punditocracy in the US?
Jeff,
They'd be brave fraydum fighters opposing the imperial aggressors, dying their valiant deaths to pertekt our way of life. We'd be looking for ever more creative ways to blow up the invaders, opposing the puppets put in power, many would be leaving the country for Canada...well, kinda just like what's happening in Eye-rack today.
Best,
D
Posted by: Dano | January 10, 2008 8:21 AM
Ragout, I think you are misunderstanding the word "template". The "template" Tim linked to describes the protocol for conducting the study. It is not the tool for recording final information about deaths, which is the "actual" instrument you linked to.
But the actual instrument has no space for recording the ages of household members (except for children and the dead)! Anyway, you can easily show that I'm wrong by pointing me to someplace where Roberts et al reported the age distribution of the population. Where is it?
SG & Luna the Cat claim to have seen survey instruments like this before. In what context? Market research? A poll for a college class? I don't recall ever seeing such an informal and sketchy survey instrument. I've only seen professional survey instruments like those provided in the NEJM article. In what field is the Lancet II survey instrument the norm?
Posted by: Ragout | January 10, 2008 8:30 AM
Ragout, are you having a "deliberately misread things" day? I said I had recruited for surveys "in this way", not with an instrument that looks like the one provided. i.e. using 2 pieces of paper - one to record who refused and who agreed to the survey (with demographic details) and one for the actual information of those who agreed. Sometimes the surveys can fit both on the same piece of paper, but this isn't always done. In this case the document called "Template" clearly tells the recruiters to collect details on the age and sex of the household members. They obviously record this information separately to the information on deaths. You may recall talk about the interviewers getting in trouble for being seen walking around with clipboards... what do you think was on those clipboards?
You are again treating the words "Template" and "actual" as if they have the same meaning for these documents as if the documents were a draft and a final copy. And where did you pull the word "actual" from anyway? It isn't written on the questionnaire and it isn't in the magazine's URL. Did you invent that word yourself to confuse the meaning of the Template, perchance?
Posted by: SG | January 10, 2008 8:51 AM
Tim still needs to correct his "fabrication" claim and issue an apology to Munro. I have e-mailed with Lance author Shannon Doocy on this topic. No demographic data on the households was collected. If it had been, any fraud would be much easier to detect.
SG,
Normally arguing with anonymous people isn't that useful, but here goes:
1) You claim to know of examples in which the data behind a peer reviewed article was given to one set of critics but not another. Great! Cite it. Tell us the article and the names of the two sets of critics. I have been issuing this challenge for over a year and no one has provided an example. Roberts behavior is unprecendented.
2) You write:
I did not say this was fraud. It is just ridiculous. And, even better, I have e-mailed with the authors on this. Have you? They do not cite any of these concerns. More or less, the underlying data for L1 and L2 are the same. The clusters are similar in size. There is no reason why privacy concerns would make you not release L1 while allowing you to release L2. Don't believe me? Ask the authors yourself! They will confirm that this is not their excuse. They just claim that the data for L1 is "unavailable" and decline to elaborate further.
3) Fritz Scheuren has never accused the authors of fraud. Are you even paying attention to what I wrote? How do you defend Roberts after he promised to such Scheuren that data and then failed to do so?
Posted by: David Kane | January 10, 2008 9:02 AM
David Kane,
Methinks you blather on too much. Hounding Roberts et al. seems to be your current reason for being; your 'life's call'. Here's some real advice for you:
First, read Dahr Jamail's latest Iraq despatch:
http://www.zcommunications.org/znet/viewArticle/16133
Then, given the horrific damage inflicted upon Iraq by the war party, you might start to expend some of your precious energy in condemining the morons who carried out this illegal act of aggression and of supporting efforts to end the illegal occupation, instead of trying to condone this abomination.
Posted by: Jeff Harvey | January 10, 2008 9:41 AM
Charles H> In response to your post #24: you seem to not understand what I mean by "upper bound". By this I mean a limit on the maximum value a survey could arrive and still be believable. You seem to assert that the value in the Lancet study is too large, but you aren't telling us what method you use to conclude that. Instead you tell me:
"Probably not, the Lancet study could have said 500,000,000 have died in the war in Iraq and there would be posters here defending the study."
I'm sorry, but since the population of Iraq is about 27,000,000. If the value you suggest was reported it would not be defensible.
Now please, what is your criterion for "believable" and why do you think it must be less than the value the Lancet study arrived at?
Posted by: SpotWeld | January 10, 2008 9:53 AM
David
1) I'm not interested in becoming non-anonymous in a hurry, thank you very much. However I don't need to. As far as I know every journal one ever publishes in includes in its review process the right for authors to contest reviews and to demand that certain individuals not review their paper. Perhaps you have never seen that right exercised, but I have. And you can be very confident that when one exercises that right, one also exercises the right to refuse to divulge data to the same individuals. But even beyond this, institutions with sensitive data choose who to send that data to, because they need to be confident it can be used wisely. I think you have never worked with sensitive data, or you would know this.
2) I am very interested to hear from you why you think it is ridiculous that the Lancet authors will not reveal their data to a non-epidemiologist, with a clear axe to grind, who cannot do basic CMR calculations, accused them of fraud at first blush, and publishes his work via Michelle Malkin. Please do tell.
3) I never intended to accuse Fritz Scheuren of anything. I wasn't paying attention and numbered a point which was really just another random insult aimed at you. Sorry about the confusion.
Posted by: SG | January 10, 2008 9:59 AM
1) If you don't have a specific cite for the your claim, then my point stands.
2) The L1 authors refuse to give the L1 data (household-level, comparable to their release for L2) to anyone, not just me.
3) No worries! Although I am not in a rush to argue with anonymous commentators, you have demonstrated good faith on a number of occasions in this discussion.
Posted by: David Kane | January 10, 2008 10:33 AM