Back in November 2001 Neil Munro was an advocate of war with Iraq and predicted:

The painful images of starving Iraqi children will be replaced by alluring Baghdad city lights, smiling wages-earners and Palestinian job seekers.

Iraq war advocates like Munro don’t like the results of the Lancet study that suggest that about 600,000 Iraqis have died as a result of the war they championed. So Munro has written a piece that throws every piece of mud he can find at the study, and to their discredit, the National Journal has published it. And if you think I’m being unfair by stressing Munro’s obvious bias, Munro has 1400 words about how the Lancet researchers were ideologically biased against the Iraq war and were funded by the Great Satan himself, George Soros. Presumably Munro thinks that studies on cancer are brought into question because the researchers are anti-cancer and funded by anti-cancer foundations.

And of course Munro’s fellow war supporters amongst bloggers such as Glenn Reynolds, Charles Johnson and Gateway Pundit seized on his article as proof that the study was a fraud or at least had serious problems. The serious problems are, however, with Munro’s article with fabricated claims, misrepresentation of sources, a misleading graph, and reliance on discredited arguments.

Munro writes:

Even Garfield, a co-author of the first Lancet article, is backing away from his previous defense of his fellow authors. In December, Garfield told National Journal that he guesses that 250,000 Iraqis had died by late 2007. That total requires an underlying casualty rate only one-quarter of that offered by Lancet II.

I contacted Garfield and this is a misrepresentation of his views. He told me:

I seem to have a special ability to make statements that lend themselves to misinterpretation.

I could not believe that 100,000 had died in 2004, but the best evidence made me believe it.

As a guess, out of the blue, I feel confident that at least a quarter million Iraqis have died due to violence since the 2003 invasion. But that is just a guess.

An estimate, based on field data, collected via good methods, is far better than a guess, even if there are some biases along with imprecision in it.

He is not backing way from his statement in 2006 that:

I am shocked that it is so high, it is hard to believe, and I do believe it. There is no reasonable way to not conclude that this study is by far the most accurate information now available.

Munro claims:

The survey teams failed to collect the fraud-preventing demographic data that pollsters routinely gather.

This is an obvious fabrication. Munro even provides a copy of the survey instrument, which tells the surveyor to record demographic data:

Who lives in this household? (Resident means spent most of the past 3 months sleeping in this household.) (only record M/F and the age, if less than 4 years, record age in months)

Correction: I was wrong. While the plan was to collect ages (hence the instructions above), during the survey this was dropped to speed things up. Les Roberts explains here. My apologies to Neil Munro.

i-a21a98ebb647833c38b8d565d6060338-munro.gif
Munro presents a highly misleading graph (shown on right) that compares the Lancet estimates of violent deaths with the IBC count of the civilian deaths reported by the media. Not all deaths are reported by the media, but the graph deceptively presents the IBC number as an estimate of violent deaths.

Munro presents a list of what he claims are “potential problems”. Long-time readers will be familiar with most of these alleged problems, but for the sake of completeness I’ll go through the entire list.

Munro:

Still, the authors have declined to provide the surveyors’ reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent — and repeatable — to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers.

This is deceitful. Munro is well aware that IRB rules require that the identity of respondents be kept confidential and that breaking such rules are not “customary scientific practice”. We know this because he has a sidebar alleging that the Lancet team broke IRB rules by recording names of respondents.

Munro:

Sample size. The design for Lancet II committed eight surveyors to visit 50 regional clusters (the number ended up being 47) with each cluster consisting of 40 households. By contrast, in a 2004 survey, the United Nations Development Program used many more questioners to visit 2,200 clusters of 10 houses each. This gave the U.N. investigators greater geographical variety and 10 times as many interviews, and produced a figure of about 24,000 excess deaths — one-quarter the number in the first Lancet study. The Lancet II sample is so small that each violent death recorded translated to 2,000 dead Iraqis overall.

This is just a rehash of Steven E Moore’s innumerate criticism. When sampling, you do not need a larger sample when the population is larger. Steve Simon explains

The best analogy I have heard about sampling goes something like: “Every cook knows that it only takes a single sip from a well-stirred soup to determine the taste.” It’s a nice analogy because you can visualize what happens when the soup is poorly stirred.

And you just need a single sip whether it’s a large vat or a small pot of soup. Munro does not understand the most basic thing about sampling and despite apparently spending months on his story, failed to learn it. Rebecca Goldin of stats.org is also unimpressed with Munro’s understanding of statistics.

Munro’s comparison of the 2004 UNDP survey with Lancet I is misleading. He failed to mention that that the UN survey covered a different time frame and measured something different. If you look at their measurements of the same thing, the UNDP and Lancet 1 agree.

Munro:

“Main street” bias? According to the Lancet II article, surveyors randomly selected a main street within a randomly picked district; “a residential street was then randomly selected from a list of residential streets crossing the main street.” This method pulled the survey teams away from side streets and toward main streets, where car bombs can kill the most people, thus boosting the apparent death rate, according to a critique of the study by Michael Spagat, an economics professor at the Royal Holloway, University of London, and Sean Gourley and Neil Johnson of the physics department at Oxford University.

Even if this “main street bias” exists (and it’s likely that it’s just a poorly worded sentence in the paper), it makes no significant difference.

Munro:

Oversight. To undertake the first Lancet study, Roberts went into Iraq concealed on the floor of an SUV with $20,000 in cash stuffed into his money belt and shoes. Daring stuff, to be sure, but just eight days after arriving, Roberts witnessed the police detaining two surveyors who had questioned the governor’s household in a Sadr-dominated town. Roberts subsequently remained in a hotel until the survey was completed. Thus, most of the oversight for Lancet I — and all of it for Lancet II — was done long-distance.

Roberts did not need to go to Iraq for Lancet II, since he was satisfied that Lafta could properly conduct the survey.

Munro:

To Kane, the study’s reported response rate of more than 98 percent “makes no sense,” if only because many male heads of households would be at work or elsewhere during the day and Iraqi women would likely refuse to participate.

To my knowledge, David Kane has never conducted a door-to-door survey in Iraq or anywhere else. Why is his uninformed opinion presented by Munro? Oh right, he said something that suited Munro’s agenda.

Lack of supporting data. The survey teams failed to collect the fraud-preventing demographic data that pollsters routinely gather.

As noted earlier, this is an outright fabrication by Munro.

Munro:

Death certificates. The survey teams said they confirmed most deaths by examining government-issued death certificates, but they took no photographs of those certificates.

I can’t help but be impressed with the why Munro feigns ignorance of the IRB rules he put tin the sidebar.

Under pressure from critics, the authors did release a disk of the surveyors’ collated data, including tables showing how often the survey teams said they requested to see, and saw, the death certificates. But those tables are suspicious, in part, because they show data-heaping, critics said. For example, the database reveals that 22 death certificates for victims of violence and 23 certificates for other deaths were declared by surveyors and households to be missing or lost. That similarity looks reasonable, but Spagat noticed that the 23 missing certificates for nonviolent deaths were distributed throughout eight of the 16 surveyed provinces, while all 22 missing certificates for violent deaths were inexplicably heaped in the single province of Nineveh. That means the surveyors reported zero missing or lost certificates for 180 violent deaths in 15 provinces outside Nineveh. The odds against such perfection are at least 10,000 to 1, Spagat told NJ.

Well, he may have told NJ that, but the only he could do such a calculation is if he made some unwarranted assumptions that missing certificates would have the same distribution for violent and non-violent deaths. Furthermore, you can always find low probability patterns in any data. For example, I just rolled a die four times and got the sequence 6426. Notice how the number goes down by two each time (wrapping around when it goes to 0), The odds of this happening are 215 to 1 against. But while this particular pattern is unlikely, there are many many patterns, so it almost certain that I can find some pattern to fit any sequence.

Munro:

Suspicious cluster. Lafta’s team reported 24 car bomb deaths in early July, as well as one nonviolent death, in “Cluster 33″ in Baghdad. The authors do not say where the cluster was, but the only major car bomb in the city during that period, according to Iraq Body Count’s database, was in Sadr City. It was detonated in a marketplace on July 1, likely by Al Qaeda, and killed at least 60 people, according to press reports.

The authors should not have included the July data in their report because the survey was scheduled to end on June 30, according to Debarati Guha-Sapir, director of the World Health Organization’s Collaborating Center for Research on the Epidemiology of Disasters at the University of Louvain in Belgium.

This is ridiculous. The survey was designed for 50 clusters. Even if the survey was scheduled to end by June 30, it makes no sense to stop then if some provinces were unsampled.

The Cluster 33 data is curious for other reasons as well. The 24 Iraqis who died violently were neatly divided among 18 houses — 12 houses reported one death, and six houses reported two deaths, according to the authors’ data. This means, Spagat said, that the survey team found a line of 40 households that neatly shared almost half of the deaths suffered when a marketplace bomb exploded among a crowd of people drawn from throughout the broader neighborhood.

Is it possible that a group of people from the same street travelled to the marketplace together for protection? Iraq is a dangerous place, you know.

After this, Munro goes on at length about the “ideological bias” against the Iraq war of the people connected with the survey. He states:

Whether this affected the authors’ scientific judgments and led them to turn a blind eye to flaws is up for debate.

Do you think that perhaps Munro’s spirited advocacy of the war affected his journalistic judgments and led him to turn a blind eye to flaws in the criticisms he printed?

I asked Munro to explain how someone such as himself, with an evident bias towards presenting the war as a success was chosen to write the piece, and why his war advocacy was not disclosed. He evaded the question, writing:

We’re a journal of fact and politics, not opinion, and that’s why we printed the facts about Soros’ money, Lafta’s Saddam-era articles, the claim of 15,000 dead from U.S. vehicles, Roberts’ description of Lafta’s views, Spagat & Kane’s claims about data-heaping, etc.

It is rather telling that, to Munro, the most damning “fact” is “Soros’ money”. And if you wondering about “the claim of 15,000 deaths from U.S. vehicles”, neither Lancet study makes any such claim. Perhaps Munro’s eyes were dazzled by those alluring Baghdad city lights.

Update: Burnham and Roberts reply to Munro:

The overwhelming confirmatory evidence of the Lancet study findings, the conventional nature of our survey procedures, and the abundance of internal consistencies in the data of which Mr. Munro was informed and chose not to report, suggests that National Journal’s critique of our work should itself be examined for political motivations.

Comments

  1. #1 Ian Gould
    January 9, 2008

    “Lafta’s team reported 24 car bomb deaths in early July, as well as one nonviolent death, in “Cluster 33″ in Baghdad. The authors do not say where the cluster was, but the only major car bomb in the city during that period, according to Iraq Body Count’s database, was in Sadr City.”

    Alternately, the fatalities might have come from several smaller incidents.

    Munro (unless this is one of the instances where he’s claiming fraud) is claiming that of the 60 people killed July 1, 24 or almost half happened to live in the cluster selected for the study.

  2. #2 Joshua Zelinsky
    January 9, 2008

    While most of Munro’s points do in fact seem to be complete crap, the point about funding from Soros is reasonable (if for example, the study had found the opposite and had been funded by some right wing billionaire I suspect we’d all be quick to all that out)

  3. #3 Kevin Donoghue
    January 9, 2008

    I wondered whether you would bother with that article. It’s such a load of shit I thought it might be better to ignore it. It really looks like a kind of swift-boating exercise to me – the whole idea is to put the targets in the position of having to deny that they copulate with pigs.

    One particularly nauseating detail is the way they try to portray Lafta’s researchers as being first Baathists, on the strength of his having been (they say) “a child-health official in Saddam Hussein’s ministry of health”, then Sadrists on the strength of the Mahdi Army controlling the health ministry, “which employed some of Lafta’s researchers.” (Oddly enough, this is the same Ministry of Health which becomes such a trustworthy source when it produces suitably modest death totals.) Who the hell do they think Lafta is, some latter-day Talleyrand, equally at home with theocrats and revolutionaries? But then to the likes of Glenn Reynolds and Charles Johnson all these Iraqis look much the same.

  4. #4 David Kane
    January 9, 2008

    There is a lot to discuss here, but let us start with the most important issue. Tim claims that Munro is guilty of a “fabrication” when Munro writes that “The survey teams failed to collect the fraud-preventing demographic data that pollsters routinely gather.”

    But this is true! I have examined the data. Have you, Tim?

    For background, it is standard in a survey to collect demographic data so that you can check that your survey sample matches up with the population, especially in a case in which you are worried about inexperienced interviewers. So, for each household, you should tabulate something like:

    1) male, age 46
    2) male, age 20
    3) female, age 10

    Sex and age are the minimum background demographic variables that you should collect. Whatever the survey document says, the Lancet interviewers did not do this.

    I think that Tim owes an apology to Munro.

  5. #5 Luna_the_cat
    January 9, 2008

    David Kane — I would prefer to think that you are simply mistaken, and thinking of something different to what is being discussed here. The alternative is that you are deliberately lying.

    The data collected for the Lancet II study includes age, sex, and household size. Explicitly. There in words and numbers. So what on earth are you talking about?

  6. #6 Dave Briggs
    January 9, 2008

    With the Democratic candidates talking about ending the war I wonder if we will get a chance to see what that county is actually like, sans war?
    Dave Briggs :~)

  7. #7 sod
    January 9, 2008

    1) male, age 46 2) male, age 20 3) female, age 10 …

    dear David,
    telephone surveys have become all too popular. i get a call at least every other month. i can not remember, that i was EVER asked to provide the information about my household, that you describe above.
    actually i am pretty sure that i would NOT provide exact age of all household members to a survey.

    i can remember the last phone survey. i was asked whether i smoke. what age CATEGORY i belong to. how many household memebrs are living with me and how many of those smoke.

    looks pretty close to the survey sheet of the lancet, doesn t it?

  8. #8 Tim Lambert
    January 9, 2008

    David Kane, in Munro’s article: “The authors refuse to provide anyone with the underlying data”

    David Kane, in comment #4: “I have examined the data. ”

  9. #9 Luna_the_cat
    January 9, 2008

    Er, yes. Indeed.

    David Kane, you are a seriously confused individual.

  10. #10 Kevin Donoghue
    January 9, 2008

    David Kane: “I have examined the data.”

    Now why do I not find that reassuring?

  11. #11 Marion Delgado
    January 9, 2008

    Tim Lambert, I do not envy you your labors.

    We need to constantly make our cross-references and debunking of the denialism network more sytematic. The day may come when they can’t make lies up more quickly than we can refute them.

  12. #12 David Kane
    January 9, 2008

    1) I agree that the quote Munro used from me about “underlying data” was somewhat out of context. Details here. Do you really want to go into the details here? I am happy to! No one, outside the authors, has seen the underlying data (not cluster summaries) for L1. And, as of now, the data is “unavailable.” No one, outside the authors, has seen all the (non-person identifying) underlying data for L2. (I have seen some of the data. Spagat and others have not. For example, we have yet to see the data, which Roberts has been promising to Fritz Scheuren for months, which allows us to compare data across interviewers.) I agree that Munro’s quote of me is not clear because the words “the data” are not very specific, but the facts are still damning.

    2) Again, Tim, you claim that something is a “fabrication” that is simply true. Do you claim that age/sex information for the households was collected? If not, what is wrong with Munro’s statement?

    There are, of course, other topics to debate here. But if we can’t even agree as to whether or not age/sex information was collected for the 1849 households, we aren’t going to make much progress on the more complex issues.

  13. #13 Luna_the_cat
    January 9, 2008

    I’m guessing that “unavailable” is probably simply unavailable to you and organisations like the Cato Institute, based on one of the conditions of release:

    “1. The data will be provided to organizations or groups without publicly stated views that would cause doubt about their objectivity in analyzing the data.”

    That doesn’t mean “unavailable to everyone”, though. Truly…

    And, explicitly from the first Lancet study (quoted without permission, but under terms of fair use):

    “If the household agreed to be interviewed, the
    interviewees were asked for the age and sex of every
    current household member. Respondents were also
    asked to describe the composition of their household on
    Jan 1, 2002, and asked about any births, deaths, or
    visitors who stayed in the household for more than
    2 months. Periods of visitation, and individual periods
    of residence since a birth or before a death, were
    recorded to the nearest month.”

    This was precisely the methodology used for the second study, as well. Ages and gender information were collected.

  14. #14 Charles H.
    January 9, 2008

    That Soros funded the Lancet study is relevant while the cancer analogy is not. His ideological leanings could lead him to endorse a sloppy study that grossly overstates the number of deaths.

    The Lancet death numbers are so unbelievably high that it appears the only people who believe them are those with an ideological agenda. They want to believe the study, so no matter how unreliable it is, they do.

  15. #15 SpotWeld
    January 9, 2008

    I have to ask, Charles H. (post #14), what criteria do you use to state that “high numbers” are unbelievable?

  16. #16 Charles H.
    January 9, 2008

    SpotWeld, let’s phrase the question another way,

    “Why do you find the Lancet study believable when not a single other study confirms its findings?”

  17. #17 SpotWeld
    January 9, 2008

    Charles H.> I have not made any assertions about the Lancet study, only asked you to state the reasoning for yours. Is there an upper bound by the numbers of deaths would have to be limited to, to be believable. And if so, how did you arrive at that value?

  18. #18 David Kane
    January 9, 2008

    I have no idea if demographic data was collected for L1. I know that it was not for L2, and so Tim owes Munro an apology over the word “fabrication.”

    And, not to derail the thread further, but please note the way that I ended my TCS Daily article (after praising Tim).

    Where is the debate going? I sometimes worry that, like so many other left/right disputes, this will never be resolved, that we will never be sure whether or not the Lancet articles were fraudulent. Will these estimates be the Chambers/Hiss debate of the 21st century? I hope not. Fortunately, other scientists are hard at work on the topic, reanalyzing the data produced in L2 and conducting new surveys. Both critics and supporters of the Lancet results should be prepared to update their estimates in the face of this new evidence. If independent scientists publish results that are similar to those of the Lancet authors, then I will recant my criticism. Will Lancet supporters do the same when the results go against their beliefs? I have my doubts.

    A fair observer, whatever his opinions of the accuracy of L2, needs to be prepared to update his estimates when new information, like this, comes to light. I used to think that a good estimate was somewhere between Jon Pedersen’s 100,000 and L2′s 600,000. I am now revising that estimate downward. What are the L2 supporters doing?

  19. #19 Luna_the_cat
    January 9, 2008

    Uh, what about the ORB survey (http://www.opinion.co.uk/Newsroom_details.aspx?NewsId=78)? Or the work by Zogby International? They confirm that the numbers are credible.

  20. #20 James
    January 9, 2008

    Thanks for commenting on this Tim. Hadn’t realised the UNDP & Lancet were comparing apples and oranges.

    David, what is your evidence that demographic data was not collected, when the paper (as quoted in post 5 & 13 by luna the cat) specifically says that it was?

  21. #21 hardindr
    January 9, 2008

    David Kane,

    Per here and here, how is the publication of your critic of the Johns Hopkin study published in the Lancet coming? It has been six months since you wrote telling everyone that it is under review. When can we expect to see it in print?

  22. #22 SG
    January 9, 2008

    David Kane still doesn’t have any proof that No one, outside the authors, has seen all the (non-person identifying) underlying data for L2, because he still has no proof that the cluster-level data cannot be used to identify individuals. Only the authors and data-collecting agencies have that proof, and naturally they can’t share it.

    I have worked on national surveys in a developed country and I am pretty confident that there is no way David Kane (or anyone else) can get access to that raw data without being connected to the study. Yet DK still makes this claim about much more sensitive data. It’s extremely dishonest, and nasty too for coming as it does with the implied fraud allegations.

  23. #23 David Kane
    January 9, 2008

    SG,

    What exactly are we disagreeing about?

    1) It is a fact that the Lancet authors share data with some critics (like me) but not others (like Spagat). I can not find a single example of similar behavior relating to a peer-reviewed publication. Can you?

    2) It is a fact that the authors related data at a certain level for L2 while refusing to release data at the same level for L1.

    3) It is a fact that Les Roberts promised Fritz Scheuren at the JSM that he (Roberts) would produce data on which interviewers provided which data. (Not with names! Just interviewer A did interviews 1, 2, 3; interviewer B did interviews 4, 5, 6; and so on.) I believe that Roberts has not done so.

    Those are the problems. If you disagree with these facts, please explain why.

  24. #24 Charles H.
    January 9, 2008

    # 17 by SpotWeld
    “Is there an upper bound by the numbers of deaths would have to be limited to, to be believable.”

    Probably not, the Lancet study could have said 500,000,000 have died in the war in Iraq and there would be posters here defending the study.

  25. #25 Ragout
    January 9, 2008

    Tim writes: “This is an obvious fabrication. Munro even provides a copy of the survey instrument, which tells the surveyor to record demographic data.”

    But the Iraq Mortality Survey Template that Tim links to includes the ungrammatical instruction: “only record M/F and the age, if less than 4 years, record age in months,” which could be interpreted as an instruction to ask the age of children only.

    If we look at the “Iraq Mortality Survey Questionnaire (actual),” for which Tim neglects to provide a link, we find that age is only collected for children and the dead, but not most household members.

    So it seems Tim owes Munro an apology, and owes the rest of us a correction.

  26. #26 Ragout
    January 9, 2008

    I’m looking forward to Tim’s comments on the new NEJM study which finds “that the 2006 study by Burnham et al. considerably overestimated the number of violent deaths.” The new study, which uses considerably better methods than did Roberts et al (such as best-practice methods of selection a random sample) finds 150,000 violent deaths after the US invasion of Iraq.

  27. #27 Ian Gould
    January 9, 2008

    Further to the admittedly secondary issue of car bomb deaths in Baghdad, during the last two weeks of June 2006 a minimum of 80 deaths by car bomb were reported by IBC in Baghdad.

  28. #28 Ian Gould
    January 9, 2008

    “The new estimate of 151,000 violent deaths from March 2003 to June 2006 is three times higher than the number of casualties recorded for the same period by Iraq Body Count, a nongovernmental organization in London, which reports only the violent deaths of civilians and only documented deaths.

    The WHO survey teams were unable to visit 115, or 11 percent, of the “clusters” or areas they had originally targeted because of insecurity. Moreover, the movement of families away from their homes because of violence can also lead to underreporting of deaths, they said. As a result of such uncertainties, researchers calculated that the number of Iraqis who died violently in the survey period could range from 104,000 to 223,000.”

    1. The new study focuses on violent deatsh not total excess deaths, the measure used by the Lancet study.

    2. Past critics of the Lancet study who relied on the IBC who now want to rely on the NEJM study have to start by admitting that the IBC was a serious undercount. The question is whether it undercounted deaths by a factor of
    three or a factor of ten.

    3. Can anyone recall offhand the lower confidence interval of the Lancet study?

    4. I look forward to ferocious forensic analysis of the current study including deamnds for death certificates, accusations of fraud; claims that all them ay-rabs is liars et cetera.

  29. #29 luminous beauty
    January 9, 2008

    Ragout,

    The study to which you refer estimates “violent” deaths. The Lancet/John Hopkins studies estimate “excess” deaths from all causes. You are comparing a subset to a whole set.

    Your ‘could be interpreted’ version of the survey template might be a plausible inference only if one selectively quotes the entry as you did by omitting the prefacing question to which those notations refer:

    “Who lives in this household? (Resident means spent most of the past 3 months sleeping in this household.)”

    How does it feel to be so intellectually dishonest?

  30. #30 Ragout
    January 9, 2008

    The study to which you refer estimates “violent” deaths. The Lancet/John Hopkins studies estimate “excess” deaths from all causes. You are comparing a subset to a whole set.

    Actually, it was the NEJM researchers — not me — who said that Lancet II “considerably overestimated the number of violent deaths.” So, take it up with them

    Before you write a letter to the NEJM, though, you might want to read the Lancet II study, which reports 600,000 violent deaths, a number considerably higher than the 150,000 found in the NEJM study.

  31. #31 Ragout
    January 9, 2008

    luminous beauty,

    Read the “actual” questionnaire not the “template.” You’ll see that the actual questionnaire has no space to record the ages of household members (except for children and the dead).

    No doubt their failure to collect age data explains why Roberts et al — unlike the higher-quality NEJM and ILCS studies — haven’t published the age distribution of their sample.

  32. #32 jc
    January 9, 2008

    How does it feel to be so intellectually dishonest?

    You could always look in the mirror or simply ask Gouldiechops to supply a pic. That ought to help.

    Hey gouldiechops,

    So you think the Iraq death toll is now approaching the 2 million mark on the Roberts’
    “analysis”. You there are 750 people a day dying in Iraq as a result of the war according to Roberts.

    Ragout:

    The number would be approaching 2 million at the present time if we used the top end of Roberts scholerly work.
    I never knew Roberts’ hired the equivilent of a Dr. Mengels assistant to help him figure it all out. LOl

  33. #33 SG
    January 9, 2008

    David Kane

    1) yes, this sort of thing happens a lot. I, personally, have seen it happen in a variety of situations.

    2)You can keep repeating this, but until you address the possibility that the authors cannot release the data because of possible identification of individuals, it remains a deliberately disingenuous criticism. If I sample 100 people in 20 clusters about a sensitive topic in 2003, then 1000 people in 20 clusters about the same topic in 2005, I am perfetly within my rights to release different levels of data from the two surveys. You know this, why keep acting like it’s fraud?

    3) Not only is what you believe often irrelevant to the truth, but the researcher is under no obligation to provide any information about the reviewers to a researcher whose first and continuing criticism of the paper was “fraud”.

    Now, David, so that we can be sure you aren’t pushing an agenda, you need to do the following post-haste:

    1) demand the NEJM data immediately, at its most detailed level
    2) demand the NEJM interviewers’ records immediately, at their most detailed level
    3) if they aren’t provided immediately, claim fraud
    4) check every RR and CI in that paper using the method you used for the lancet paper, so that you can confirm with your (wrong) first year psych stats that there is no conflict between normally-distributed estimates and ratio estimates
    5) check every other calculation
    6) if you find ANY errors, claim fraud

    Have you done this yet? I recall that when the Lancet survey came out you were making fraud accusations before you had taken any of these steps.

  34. #34 SG
    January 10, 2008

    Ragout, I think you are misunderstanding the word “template”. The “template” Tim linked to describes the protocol for conducting the study. It is not the tool for recording final information about deaths, which is the “actual” instrument you linked to. The interviewer will follow the template, starting by introducing the consent form and then checking household composition, etc. Finally they will use the “actual” data sheet to record the details of those who died. The data sheet is linked to the household (whose descriptive data is recorded in the consent form, I suspect) by the variables “governate”, “cluster number”, “household number”.

    Quite often in surveys like this the information on the person contacted, the household composition, might be recorded on, for example, a clipboard (where empty households and refusals are also recorded), and then the detailed data for the households who agree to participate is recorded in the “actual” sheet. Or did you think that every householder who refused to participate has one of those “actual” sheets filled out, but blank?

    (I have myself recruited for surveys in this way, have instructed others working with me to do this, and have seen and analysed data collected in this way. It’s hardly a mystery).

    I suspect this is a deliberate misrepresentation by that magazine, especially the use of the words “template” and “actual”.

  35. #35 ben
    January 10, 2008

    “I have worked on national surveys in a developed country and I am pretty confident that there is no way David Kane (or anyone else) can get access to that raw data without being connected to the study. Yet DK still makes this claim about much more sensitive data. It’s extremely dishonest, and nasty too for coming as it does with the implied fraud allegations.”

    How many of those surveys were funded by Soros? How many came out right before the American elections? How many could influence the outcome of the American elections? How many of those surveys were conducted by researchers who had an axe to grind with the current American administration?

    I’m thinking none.

  36. #36 SG
    January 10, 2008

    Ben, if the survey I worked on wasn’t politically charged and wasn’t funded by a political enemy of the government, but my survey had some of the same methods or policies as the one Soros funded, doesn’t that suggest something to you about the Soros-funded survey? I don’t think your logic or your insinuation or whatever it is is entirely clear…

  37. #37 Charles H.
    January 10, 2008

    SG, no other group or organization has a death tally for Iraq approaching anywhere near the total the Lancet study does.

    Interestingly, those with less of a political ax to grind also have death figures that are at least in the realm of plausibility.

  38. #38 SG
    January 10, 2008

    Yes Charles H, that’s right. And the correct approach is not to immediately dismiss the highest number by claiming fraud (as David Kane does) or ignoring the understanding of survey sampling you previously claimed to have (as Ragout does) so you can blatantly misrepresent a survey instrument. The correct approach is to compare the studies and ask why the difference arose.

    And why exactly is it that supporters of the invasion of Iraq have “less of a political ax to grind” than those who oppose it? Exactly who is it gets to decide which axe is bluntest, or being sharpened the most? Do you? Do I?

  39. #39 Ian Gould
    January 10, 2008

    “SG, no other group or organization has a death tally for Iraq approaching anywhere near the total the Lancet study does.”

    This is technically correct seeing as the ORB survey came up with an even higher estimate.

    http://www.npr.org/templates/story/story.php?storyId=14501232

  40. #40 Charles H.
    January 10, 2008

    “This is technically correct seeing as the ORB survey came up with an even higher estimate.”

    ORB is what? A polling agency? From Britain? Their figures say 1,200,000 Iraqis murdered? And I’m sure you find their poll accurate too, no?

    ORB partnered with an Iraq company, http://www.iiacss.org, which appears to be a phantom company, existing on the web only. Only a fool would accept the polling data done by an apparently non-existent company.

    The only thing more comical thing then the ORB poll would be to find out that Soros paid for that too. I wonder if he did.

    Thanks for the laugh Ian, I needed one today.

  41. #41 SG
    January 10, 2008

    Ah, clearly Charles H gets to decide whose axe is sharpest. No biasses at all, eh, Charles?

  42. #42 Charles H.
    January 10, 2008

    Everyone has biases SG, I at least try to be aware of my own instead of parading them here for all to see.

  43. #43 SG
    January 10, 2008

    you certainly aren’t parading them for all to see…

  44. #44 dsquared
    January 10, 2008

    I don’t understand what Ragout is on about “The new study, which uses considerably better methods than did Roberts et al (such as best-practice methods of selection a random sample)”. The NEJM study has 10% of its clusters censored, probably informatively (as in, unable to be surveyed because too dangerous), and the death rates cobbled together from comparisons with the IBC data. That’s not exactly a trivial problem.

    On the other hand, the NEJM study does look pretty well done, and like the Lancet study, it rejects the null that the violent death rate didn’t increase with a very high degree of confidence and rejects the null that the death count is only what the IBC say it is at a fairly high degree of confidence. So I’m prepared to accept it as, like the Lancet study, a reasonable qualitative assessment of what things have been like in Iraq (ie, a disaster).

  45. #45 Jeff Harvey
    January 10, 2008

    Ben continues with his hilarity when he asks:

    “How many [of the studies critical of the Iraq war] came out right before the American elections? How many could influence the outcome of the American elections?

    For heavens sake man! Wake up! What difference would it have made if Kerry had won over Bush? Both are imperial war hawks; in the US you don’t have any real opposition to the current corporatocracy because both parties are utterly pro-establishment. Like the mainstreram media, they are totally beholden to the agenda of those with concentrated wealth and power. US elections stopped being waged on policy years ago; nowadays its personality and nothing more. The current crop of ‘Democrats’ is no exception. There’s one party in the US: the Property Party, with two right wings.

    What is most remarkabale are those idiots out there who are using whatever means they can to defend the indefensible; an illegal war (aggression) that has largely reduced a country to rubble, led to mass death and generated a humanitarian catastrophe. A war fought entirely on economic and political grounds using the same old chestnut – western ‘security’ – as an excuse. A war that gives complete creedence to Naomi Klein’s hypothesis of a ‘disaster capitalism’ complex that is alive and well.

    Let me ask the dimwits out there this: how would 150,000 civilian deaths from an illegal invasion of the United States by a foreign power over a 3-4 year span be received by the punditocracy in the US?

  46. #46 Luna_the_cat
    January 10, 2008

    I also don’t understand what Ragout is on about with
    “only record M/F and the age, if less than 4 years, record age in months,” which could be interpreted as an instruction to ask the age of children only.
    …this could be interpreted in this fashion only through deliberate stupidity, as far as I can tell.

    The instruction is to record age and gender for the household members, and if a household member is under the age of 4, to record age in months rather than years. This is immediately clear to anyone with experience of surveys, however ungrammatical Ragout finds the comma.

    I also wonder why Charles H doubts the existance of the Independent Institute for Administration and Civil Society Studies. They’ve been around since 2003; they frequently work with universities (including the University of Michigan and a couple of German universities), and it is possible to contact both employees and former employees now living elsewhere, such as http://www.lightstalkers.org/bashar_ali. I can also tell you that IIACSS holds a fair number of software licences, including Cisco, .Net, and SPSS (I know this because it is possible to access software profiles of Microsoft Registered Partners online). They also have landline phone numbers registered. Charles H can’t have looked very hard for IIACSS existance.

  47. #47 Dano
    January 10, 2008

    Let me ask the dimwits out there this: how would 150,000 civilian deaths from an illegal invasion of the United States by a foreign power over a 3-4 year span be received by the punditocracy in the US?

    Jeff,

    They’d be brave fraydum fighters opposing the imperial aggressors, dying their valiant deaths to pertekt our way of life. We’d be looking for ever more creative ways to blow up the invaders, opposing the puppets put in power, many would be leaving the country for Canada…well, kinda just like what’s happening in Eye-rack today.

    Best,

    D

  48. #48 Ragout
    January 10, 2008

    Ragout, I think you are misunderstanding the word “template”. The “template” Tim linked to describes the protocol for conducting the study. It is not the tool for recording final information about deaths, which is the “actual” instrument you linked to.

    But the actual instrument has no space for recording the ages of household members (except for children and the dead)! Anyway, you can easily show that I’m wrong by pointing me to someplace where Roberts et al reported the age distribution of the population. Where is it?

    SG & Luna the Cat claim to have seen survey instruments like this before. In what context? Market research? A poll for a college class? I don’t recall ever seeing such an informal and sketchy survey instrument. I’ve only seen professional survey instruments like those provided in the NEJM article. In what field is the Lancet II survey instrument the norm?

  49. #49 SG
    January 10, 2008

    Ragout, are you having a “deliberately misread things” day? I said I had recruited for surveys “in this way”, not with an instrument that looks like the one provided. i.e. using 2 pieces of paper – one to record who refused and who agreed to the survey (with demographic details) and one for the actual information of those who agreed. Sometimes the surveys can fit both on the same piece of paper, but this isn’t always done. In this case the document called “Template” clearly tells the recruiters to collect details on the age and sex of the household members. They obviously record this information separately to the information on deaths. You may recall talk about the interviewers getting in trouble for being seen walking around with clipboards… what do you think was on those clipboards?

    You are again treating the words “Template” and “actual” as if they have the same meaning for these documents as if the documents were a draft and a final copy. And where did you pull the word “actual” from anyway? It isn’t written on the questionnaire and it isn’t in the magazine’s URL. Did you invent that word yourself to confuse the meaning of the Template, perchance?

  50. #50 David Kane
    January 10, 2008

    Tim still needs to correct his “fabrication” claim and issue an apology to Munro. I have e-mailed with Lance author Shannon Doocy on this topic. No demographic data on the households was collected. If it had been, any fraud would be much easier to detect.

    SG,

    Normally arguing with anonymous people isn’t that useful, but here goes:

    1) You claim to know of examples in which the data behind a peer reviewed article was given to one set of critics but not another. Great! Cite it. Tell us the article and the names of the two sets of critics. I have been issuing this challenge for over a year and no one has provided an example. Roberts behavior is unprecendented.

    2) You write:

    You can keep repeating this, but until you address the possibility that the authors cannot release the data because of possible identification of individuals, it remains a deliberately disingenuous criticism. If I sample 100 people in 20 clusters about a sensitive topic in 2003, then 1000 people in 20 clusters about the same topic in 2005, I am perfetly within my rights to release different levels of data from the two surveys. You know this, why keep acting like it’s fraud?

    I did not say this was fraud. It is just ridiculous. And, even better, I have e-mailed with the authors on this. Have you? They do not cite any of these concerns. More or less, the underlying data for L1 and L2 are the same. The clusters are similar in size. There is no reason why privacy concerns would make you not release L1 while allowing you to release L2. Don’t believe me? Ask the authors yourself! They will confirm that this is not their excuse. They just claim that the data for L1 is “unavailable” and decline to elaborate further.

    3) Fritz Scheuren has never accused the authors of fraud. Are you even paying attention to what I wrote? How do you defend Roberts after he promised to such Scheuren that data and then failed to do so?

  51. #51 Jeff Harvey
    January 10, 2008

    David Kane,

    Methinks you blather on too much. Hounding Roberts et al. seems to be your current reason for being; your ‘life’s call’. Here’s some real advice for you:

    First, read Dahr Jamail’s latest Iraq despatch:

    http://www.zcommunications.org/znet/viewArticle/16133

    Then, given the horrific damage inflicted upon Iraq by the war party, you might start to expend some of your precious energy in condemining the morons who carried out this illegal act of aggression and of supporting efforts to end the illegal occupation, instead of trying to condone this abomination.

  52. #52 SpotWeld
    January 10, 2008

    Charles H> In response to your post #24: you seem to not understand what I mean by “upper bound”. By this I mean a limit on the maximum value a survey could arrive and still be believable. You seem to assert that the value in the Lancet study is too large, but you aren’t telling us what method you use to conclude that. Instead you tell me:

    “Probably not, the Lancet study could have said 500,000,000 have died in the war in Iraq and there would be posters here defending the study.”

    I’m sorry, but since the population of Iraq is about 27,000,000. If the value you suggest was reported it would not be defensible.

    Now please, what is your criterion for “believable” and why do you think it must be less than the value the Lancet study arrived at?

  53. #53 SG
    January 10, 2008

    David

    1) I’m not interested in becoming non-anonymous in a hurry, thank you very much. However I don’t need to. As far as I know every journal one ever publishes in includes in its review process the right for authors to contest reviews and to demand that certain individuals not review their paper. Perhaps you have never seen that right exercised, but I have. And you can be very confident that when one exercises that right, one also exercises the right to refuse to divulge data to the same individuals. But even beyond this, institutions with sensitive data choose who to send that data to, because they need to be confident it can be used wisely. I think you have never worked with sensitive data, or you would know this.

    2) I am very interested to hear from you why you think it is ridiculous that the Lancet authors will not reveal their data to a non-epidemiologist, with a clear axe to grind, who cannot do basic CMR calculations, accused them of fraud at first blush, and publishes his work via Michelle Malkin. Please do tell.

    3) I never intended to accuse Fritz Scheuren of anything. I wasn’t paying attention and numbered a point which was really just another random insult aimed at you. Sorry about the confusion.

  54. #54 David Kane
    January 10, 2008

    1) If you don’t have a specific cite for the your claim, then my point stands.

    2) The L1 authors refuse to give the L1 data (household-level, comparable to their release for L2) to anyone, not just me.

    3) No worries! Although I am not in a rush to argue with anonymous commentators, you have demonstrated good faith on a number of occasions in this discussion.

  55. #55 Tim Lambert
    January 10, 2008

    Les Roberts on whether they collected demographic data:

    >What is striking is Mr. Moore’s statement that we did not collect any demographic data, and his implication that this makes the report suspect. This is curious because, not only did I tell him that we asked about the age and gender of the living residents in the houses we visited, but Mr. Moore and I discussed, verbally and by e-mail, his need to contact the first author of the paper, Gilbert Burnham, in order to acquire this information as I did not have the raw data. I would assume that this was simply a case of multiple misunderstandings except our first report in the Lancet in 2004 referenced in our article as describing the methods states, “…interviewees were asked for the age and sex or every current household member.” Thus, it appears Mr. Moore had not read the description of the methods in our reports. It is not important whether this fabrication that “no demographic data was collected” is the result of subconscious need to reject the results or whether it was intentional deception.

    Will David Kane now call for Neil Munro to correct his article? I’m not holding my breath.

  56. #56 Luna_the_cat
    January 10, 2008

    David Kane claims:

    2) The L1 authors refuse to give the L1 data (household-level, comparable to their release for L2) to anyone, not just me.

    And I say, David Kane, you are a liar. I know of at least one person who has this dataset, by requesting it from Dr. Burnham, who sent it to them.

    Do you understand that? Is that clear enough and simple enough?

    It *IS* possible to get the dataset, if you are a legitimate researcher associated with a legitimate research body, subject to appropriate protections for sensitive data. The authors of the study will not release the dataset to anyone who is not legally bound by some very strict rules for the protection of sensitive data, due to their worry for the safety of both interviewers and respondents in a still-volatile situation. If it were to get given to some schmuck who decides to put up respondent data on the internet, although they would be able to pursue said schmuck legally, there is no guarantee that they would be able to get potentially-identifying data OFF the internet fast enough and completely enough to protect all those who might be seen to be “cooperating” with Westerners by anti-Western militias. If anyone gets killed because they even potentially helped an outside organisation with demographic data-gathering, it jeopardizes all future cooperation with such efforts. Therefore, the authors have a legitimate interest to only release the data to people who are used to abiding by the protection of sensitive data, and who can reasonably be trusted to do so. i.e., not everyone who asks.

    I say this once again, in case the above use of more than five words at a time has confused you.

    At least one person not associated with the authors has this data.

    And, as the authors have repeatedly explicitly stated, it includes basic demographic data such as gender and ages.

    When you claim that:
    (a) The data is not available to anyone
    and
    (b) that it does not contain basic demographics

    You. ARE. LYING.

    I’m not able to put it in words shorter than that.

  57. #57 dsquared
    January 10, 2008

    if you are a legitimate researcher associated with a legitimate research body, subject to appropriate protections for sensitive data

    in respect of which I note that the National Journal article certainly implies that Kane shared the data with Spagat – I thought you had promised not to do that, David?

  58. #58 David Kane
    January 10, 2008

    Good stuff!

    1) Can you provide a link to original source of that Roberts statement? I am sure that it is accurate, but I need to provide a reference when I reach out to the other authors.

    2) What’s the answer? My guess is that Roberts mispoke above (or is being purposely misleading). First, the interviewers may very well have “asked” such questions without recording the data. Of course, this would be incredibly misleading but at least not an out-right lie. Second, Roberts could have confused data about all the residents of each household with data about the deaths. Age/sex was recorded for all deaths.

    But, it could be that I am wrong! Age/sex information on all the household residents might have been colllected but not shared in the data set that was distributed. But, Shannon Doocy told me that this was not the case, that she did not have the data. I will check with her.

    If I am wrong, I will write to Munro/NJ and request a correction. If I am right, I assume that Tim will issue an apology.

    I love getting to the bottom of empirical questions!

  59. #59 Luna_the_cat
    January 10, 2008

    Sorry, let me just clarify: You said there, “L1″ data — I assume you mean L2 data. I am personally referring to the dataset for the 2006 Iraqi mortality study, just to avoid any ambiguity.

  60. #60 David Kane
    January 10, 2008

    Luna,

    I said L1 and meant L1. No one has seen the L1 household data. The L2 household data has been shared with some researchers (like me) but not others (like Spagat).

    dsquared,

    I have not shared the data with Spagat or anyone else. The agreement with Hopkins does not allow doing so and I have honored that agreement. I am allowed, however, to provide summaries of the data and answer questions (which do not identify specific respondants) about the data. I have done so for Spagat, Munro and many others. Indeed, my efforts uncovered several errors in the initial data release which were then corrected. Feel free to check all this with author Shannon Doocy.

  61. #61 Luna_the_cat
    January 10, 2008

    Then, I apologise for misunderstanding. The summary data is unacceptable? Why?

    Dr. Burnham is out of the country at the moment, and only intermittently in communication with anyone. He is due back at the end of January. At that time, I will call him, and clarify the situation with L1 data.

  62. #62 Tim Lambert
    January 10, 2008

    The Les Roberts quote comes from [some blog called Deltoid](http://scienceblogs.com/deltoid/2006/10/les_roberts_responds_to_steven.php).

  63. #63 JB
    January 10, 2008

    Kane said: ‘Tim still needs to correct his “fabrication” claim and issue an apology to Munro.”

    This from the guy who speculated about fraud in the Lancet study based on little more than his incredulity at the response rate. Has Kane ever apologized to Roberts et al?

    And after Tim Lambert quotes Roberts (above) as saying that they collected demographic data (HINT: the Roberts quote “contact the first author of the paper, Gilbert Burnham, in order to acquire this information” implies the data was indeed collected), Kane comes back with

    “My guess is that Roberts mispoke above (or is being purposely misleading).”

    In other words, the “lawyerly” way of implying Roberts may be dishonest without actually coming out and saying as much.

    I wonder if other members of harvard IQSS would approve. Somehow I doubt it, based on the fact that member Amy Perfors already chastized Kane once [for his original fraud speculation], saying that the “tone is unacceptable, the facts are shoddy, and the ideas are not endorsed by myself, the other authors on the sidebar, or the Harvard IQSS.”

  64. #64 sod
    January 10, 2008

    2) What’s the answer? My guess is that Roberts mispoke above (or is being purposely misleading). First, the interviewers may very well have “asked” such questions without recording the data. Of course, this would be incredibly misleading but at least not an out-right lie. Second, Roberts could have confused data about all the residents of each household with data about the deaths. Age/sex was recorded for all deaths.

    the relevant part is at the bottom of page 2 of the 2004 paper:

    If the household agreed to be interviewed, the
    interviewees were asked for the age and sex of every
    current household member.

    http://web.mit.edu/humancostiraq/reports/lancet04.pdf

  65. #65 Brownie
    January 10, 2008

    the interviewers may very well have “asked” such questions without recording the data.

    That’s where my money is.

  66. #66 hardindr
    January 10, 2008

    David Kane,

    You seem to have ignored my earlier question (#22). When can we expect to see your critique of the Johns Hopkins study published in the Lancet appear in a reputable, peer-reviewed journal?

  67. #67 SG
    January 10, 2008

    David Kane

    even if I identified myself online, any particular instances of people not sharing data would still have to be confidential, right? I’m hardly going to go blathering about the practices of my workplace on a blog, am I? I merely restate my point that you know this situation occurs, even if you haven’t observed it personally, and if you are surprised it might occur we can all take this as evidence you have zero experience with sensitive data. Why don’t you try writing to the US NIMH and asking them for household-level data on drug use and sexual activity, or ask the CDC for individual patient level data on HIV or gonorrhea diagnoses. It’s really cool data, and you’re a serious chap who can calculate a CMR – I’m sure they’ll send it to you post haste.

    Anyhow, I know you aren’t going to change your mind about this, because it would involve attributing good faith intentions to Burnham et al, and that would undermine your entire raison d’etre. I insist on making the point so that anyone silly enough to read this far in these Threads of Doom can see that others disagree with your motivating example (even if those others are anonymous and unserious).

    And, wtf do you mean by “a specific cite for your claim”? Do you think that there is some kind of published article out there detailing all the occasions that researchers had a bit of a blue? That’s ridiculous. You’re making deliberate attempts to muddy the waters, as if this debate were the same as a debate about the pre-war Iraqi death rate. It’s quite disingenuous.

  68. #68 Ragout
    January 10, 2008

    And where did you pull the word “actual” from anyway? It isn’t written on the questionnaire and it isn’t in the magazine’s URL.

    That’s how the link was labeled in the National Journal: “Iraq Mortality Survey Questionnaire (actual).”

    They obviously record this information separately to the information on deaths.

    So where’s the evidence that this alleged separate survey form ever existed? Maybe it’s stored with the alleged tabulations of age data.

  69. #69 David Kane
    January 10, 2008

    How many times do we need to go through this? First, there are lots of examples of researchers not sharing their data, although this is becoming much less common giving changing standards in publication. Second, there are lots of examples of researches sharing their data with everyone, including all of their critics. The latest example that I skimmed yesterday was the American Economic Review smackdown involving C. Hoxby. Third, I also believe that there are examples of researchers who share their data with just a few pals, those they are sure won’t be critical of their work (or compete with it). I don’t know of any such examples personally, but I wouldn’t be surprised about them.

    These are facts and we can all agree with them.

    But the behavior of the L2 authors is unlike any of this. They share the data with some critics (like me) but not others (like Spagat). I have never heard of a scientific paper, something published in the peer-reviewed literature, in which the authors behaved like this. If you know of such a paper, then cite that paper. Tell us the specific paper. You don’t need to tell me your name. You don’t need to tell me where you work. If we are going to have a scientific discussion, then you need to cite the literature.

  70. #70 SG
    January 10, 2008

    David, even if I cite you a paper we still won’t be having a scientific discussion. Very few papers come with a regularly updated list of who asked for and accessed the data, so the issue would remain hearsay. I don’t see how citing a paper is going to help with this. The discussion can be no more scientific than your claims of “trust me, it’s fraud”. The L2 authors no more have to share their data with some bunch of unskilled physicists than Einstein had to share his calculations with RA Fisher. That kind of “peer review” is just mendacious self-aggrandizement, such as might be seen when a researcher dumps a half-completed, highly erroneous article in Michelle Malkin’s lap.

    Ragout, you are saying that you took the National Journal’s titles as reflective of the true nature of the instruments? You believed that the “template” was a draft and the “actual” a final copy? This seems a remarkable sudden suspension of your usual cynicism.

    It’s likely that the information on age and gender of the household was associated with the cluster identifier data. What are the chances that data will be released? It’s essentially an identifying key, given the detail contained therein. At the very least, anyone whose household was in a smaller town would be identifiable, since within clusters in that town it would be quite easy to find a specific house based on its detailed age/gender and death structure. This means that the data won’t be released any time soon, I suspect.

  71. #71 Ragout
    January 10, 2008

    SG writes: You believed that the “template” was a draft and the “actual” a final copy?

    Huh? I share your interpretation:

    The “template” Tim linked to describes the protocol for conducting the study. It is not the tool for recording final information about deaths, which is the “actual” instrument you linked to. The interviewer will follow the template, starting by introducing the consent form and then checking household composition, etc. Finally they will use the “actual” data sheet to record the details of those who died.

    As you note, the “actual” data sheet has space to record the ages of those who died, but no space to record the ages of living adults.

    This is where SG gets ridiculous:

    The data sheet is linked to the household (whose descriptive data is recorded in the consent form, I suspect)

    Here you postulate the existence of some other survey form: a consent form. On this consent form, you postulate, they recorded the age data: data they never bothered to tabulate. But not only has a consent form never been produced, it has been specifically reported that there was no written consent form!

  72. #72 David Kane
    January 10, 2008

    David, even if I cite you a paper we still won’t be having a scientific discussion. Very few papers come with a regularly updated list of who asked for and accessed the data, so the issue would remain hearsay. I don’t see how citing a paper is going to help with this.

    Easy! It won’t still be hearsay after I check. I’ll call the authors of the paper and ask if they shared their data with anyone. If they confirm your story, then we have a counterexample. Simple!

  73. #73 Marion Delgado
    January 11, 2008

    Just in case anyone MISSED it, Luna, in particular, gives the precise reasons I use the L word with Kane. Extremely good work, Luna.

    Instead of engaging this pointless little Swiftboating Troll, find his most salient points and refute them en masse for general consumption, please, people. That’s the time-tested remedy for the marching troll hordes. They actually resemble the demons in Phantom Tollbooth – most of them are Terrible Triviums and Census Takers, but all the demons are there somewhere.

  74. #74 SG
    January 11, 2008

    so Ragout, you’re so familiar with the survey-taking process that you can’t imagine the existence of two pieces of paper, one on a clipboard for easily recording household details with cluster number, and another containing the remainign survey data?

  75. #75 Ragout
    January 11, 2008

    SG,

    I can imagine the existence of lots of things, but I wait for evidence before I believe them. I’ll believe that the Lancet team collected age data just as soon as they report the age distribution of their sample.

  76. #76 sod
    January 11, 2008

    Easy! It won’t still be hearsay after I check. I’ll call the authors of the paper and ask if they shared their data with anyone. If they confirm your story, then we have a counterexample. Simple!

    David, you obviously have read at least part of the literature on this subject, before you published that this is the only instance of this type of information sharing?!?
    you know, it is good practise that people SUPPORT their thesis, before they ask others to give a counterexample!

    a simple scholar search with some relevant terms (researchers sharing data collaboration) gives about 20000 entries.

    this is a nice study on the subject:

    http://www.uta.fi/~lisaka/Taljaisic2002_konv.pdf

  77. #77 c.l. ball
    January 11, 2008

    As far as I know every journal one ever publishes in includes in its review process the right for authors to contest reviews and to demand that certain individuals not review their paper.

    Nonsense. Political science journals (APSA, ISQ, IO) have no such procedures. An author can make the request in the submission letter but journals are under no obligation to follow it. If the authors make it conditional, their paper will not be reviewed. The authors never get to know who reviewed their paper (the reviewers learn only if it is published). ISQ and others require that data sets be publicly available.

  78. #78 Luna_the_cat
    January 11, 2008

    You know, I’ve just been reminded of an incident around 1997. My husband, who is also an academic, had a paper in for review at the journal Disability & Society. Not only was he aware of who was reviewing the paper, as the lead author he was in direct communication with some of them about some issues and definitions that they wanted to see addressed further. At one point during the review process he got into a rather acrimonious debate with one of the reviewers, and requested that the journal pass the paper to a different reviewer — which they did, and the paper was eventually published in 1998.

    Obviously, policies differ from journal to journal.

  79. #79 David Kane
    January 11, 2008

    In another thread, Tim writes:

    David Kane: “Author Shannon Doocy told me it was not.”

    Perhaps you could provide a quote from the email?

    Consider this scenario: Ages have been removed from the data set given to David Kane to protect the privacy of the respondents. Kane tells Munro that ages were not collected. When called on this, he invents an email from Doocy.

    Funny! Unfortunately, I do not like to quote from someone’s e-mail without their permission. Fortunately, in this case I don’t have to. I e-mailed Doocy to confirm (after all, I could have been wrong about this) and cc’d Tim. She replied and cc’d Tim. No age information was collected for the households. This means:

    1) Munro is not guilty of “fabrication.” Tim owes him a correction and apology.

    2) Roberts has been lying (on more than one occasion?) about this. Why?

    3) It is very hard to confirm that the L2 sample is representative without age information. Was there fraud? Without age information, it is harder to tell than it should have been.

    Getting to the bottom of these factual issues is one of my favorite parts of the Lancet wars. The last good one was the several hundred posts it took to convince dsquared that the CMR confidence intervals were normally distributed in L1. That was fun!

    Anyway, we now have a wonderful example of Roberts lying to Tim, making him look very foolish because he was too trusting. Perhaps Roberts doesn’t know that bad things happen to people who (try to) mislead to Tim Lambert . . .

  80. #80 John Tirman
    January 11, 2008

    One quick note about the Soros bugaboo. I commissioned L2. It was commissioned in Oct 2005, with internal funds from the Center for International Studies at MIT, of which I am executive director. The funds for public education (not the survey itself) came from the Open Society Institute in the following spring, long after things had started. Burnham did not know this (Roberts was not much involved at this point.) MIT was providing funds, that’s all he knew or needed to know. There were other small donors involved too. I told this to Munro on the telephone and in an email. He nonetheless implied that Soros money had funded the survey from the start, possibly at Soros’ behest. That is a disgraceful lie, and Munro knows it.

    This timing also underscores another Munro falsehood: the attempt to influence 2006 congressional elections. We began in Oct 05 with the intention of getting the survey done in winter and results out in spring. The violence was so severe that the survey could not be conducted until late spring, and then at great peril. About two months for data entry, analysis, writing, peer review, etc. We decided to delay the release if too close to the election, setting our own deadline of Oct 14. It was never intended to influence the congressional election, though there is certainly nothing wrong in a democracy with wanting the public informed.

    Munro also knew this and fabricated a tale to make this sound like a political gambit from the start. These are just two aspects that I know first hand. Munro’s behavior–screaming at me on the telephone, demanding to know if any donors were Muslims, etc.—signalled his intentions from the start. This is a bad actor and is a disgrace to the newsletter where the diatribe appeared.

    The NEJM article is far more important and interesting. This is where debate should be focused, not a blatant hatchet job by a guilty malcontent and one “source.”

  81. #81 David Kane
    January 11, 2008

    I agree with John Tirman that the section about Soros in the NJ article was weak. I am also happy to believe that he did not seek to influence the 2006 elections. But AP reporter Paul Hoy interviewed Les Roberts in August and wrote:

    Roberts organized two surveys of mortality in Iraqi households that were published last October in Britain’s premier the medical journal, The Lancet. He acknowledged that the timing was meant to influence midterm U.S. elections.

    So, Roberts sought to influence the 2006 election, just as he did the 2004 election. (See also this previous Deltoid discussion.) Nothing wrong with that! Democracy is wonderful and, as Tirman notes, there is “certainly nothing wrong in a democracy with wanting the public informed.” But Roberts, at least, did “attempt to influence 2006 congressional elections.” Tirman may be correct that he failed to do so, but that is not the point under dispute.

    And, just for the record, why did it only take a three weeks for L1 to go from survey completion (Sep 20, 2004) to press while it took 3 months for L2? To a casual observer, the key controlling factor seems to be a desire to publish in October . . .

  82. #82 Tim Lambert
    January 11, 2008

    Since I have contradictory information from two of the Lancet authors, I’ve asked Les Roberts for a clarification.

  83. #83 David Kane
    January 11, 2008

    Thanks to Tim for trying to get to the bottom of this. One of the main reasons that I spend so much time at Deltoid is that Tim, although generally not on my “side,” has almost always been an honest broker on these points of empirical fact.

    What will Les say? Good question! a) He might claim to have misspoken, to have been referring to just the procedure for 2004. That’s a great answer since the data for 2004 is no longer “available,” so there is no way for us to tell if this claim is any more accurate than the previous one. (Doocy was not involved with the 2004 paper, so I can’t check with her.) Still, this would require Roberts to apologize to Moore, at least on this point. And then Tim would need to correct this post and apologize to Munro. And this will complete my small effort to set the record straight. Munro is not guilty of “fabrication.” Roberts misled Tim, causing him to make the worst possible accusation against a journalist.

    b) Roberts might also claim that the interviewers “asked” for this data but did not write it down. That would be a ludicrous answer, but might still allow him to claim that his statements to Moore were accurate.

    c) Roberts could claim that he (and/or Lafta?) do have the age data but that they didn’t share it with Doocy. That would be a fun rabbit hole to go down.

    d) He might decline to respond to Tim’s e-mail. I wouldn’t do that, if I were him, since you don’t want Tim Lambert on your bad side. Ask John Lott!

    I guess a).

  84. #84 douglas clark
    January 12, 2008

    David Kane,

    The real question is whether the Johns Hopkins study was accurate or not. All of this peripheral stuff is just so much smoke and mirrors.

  85. #85 sod
    January 12, 2008

    to quote David Kane on David Kane in #83:

    Instead of calling this a lie, could we go with misunderstanding?
    http://tinyurl.com/yrmpdb

    And then Tim would need to correct this post and apologize to Munro. And this will complete my small effort to set the record straight. Munro is not guilty of “fabrication.” Roberts misled Tim, causing him to make the worst possible accusation against a journalist.

    you are continuosly making the same accusations against scientists. the difference is, that Tim has a basis for his claims, while you have not!

    here again the original text from the Munro paper:

    Lack of supporting data. The survey teams failed to collect the fraud-preventing demographic data that pollsters routinely gather. For example, D3 Systems, a polling firm based in Vienna, Va., that has begun working in Iraq, tries to prevent chicanery among its 100-plus Iraqi surveyors by requiring them to ask respondents for such basic demographic data as ages and birthdates.

  86. #86 Tim Lambert
    January 12, 2008

    Les Roberts emails:

    >I was wrong! Shannon cleaned and analyzed the data. I never saw the
    raw forms. We collected age and gender on everyone in 2004. That was the plan in 2006. My understanding is that they did this for some houses in the start but as that was the most lengthy part of the
    interview they just started recording how many people were in the house and the age and gender of the dead. Their focus in the field was on security and Riyadh said that this made the visits to each area a lot shorter….and I cannot second-guess that judgment call. Thus, at the end, we could not calculate an gender or age specific mortality or make a demographic profile because it was missing for many (I think most) households. In 2004, our IMR and <5MR were so imprecise as to be meaningless because children were so few and I know that led to Riyadh's decision.

    >I was mistaken when I spoke to Steven Moore, and I was mistaken when I wrote to you.

    I have corrected my post.

  87. #87 izzy
    January 12, 2008

    Tim castigates Munro for listing sample size as one of the common concerns about Lancet. Munro is therefore an “innumerate”:

    This is just a rehash of Steven E Moore’s innumerate criticism. When sampling, you do not need a larger sample when the population is larger. Steve Simon explains

    The best analogy I have heard about sampling goes something like: “Every cook knows that it only takes a single sip from a well-stirred soup to determine the taste.” It’s a nice analogy because you can visualize what happens when the soup is poorly stirred.

    And you just need a single sip whether it’s a large vat or a small pot of soup. Munro does not understand the most basic thing about sampling and despite apparently spending months on his story, failed to learn it.

    Of course, violence and violent deaths in Iraq is very much not a “well stirred soup”. It’s more like a pot of water with a lot of bouillon cubes floating in it. People claiming this “well stirred soup” analogy as a valid one for Iraq would seem to be the “innumerates” here, as they seem clueless about the phenomenon these studies are trying to measure.

    Another “innumerate” who, like Munro and Steven Moore, supposedly “do not understand the most basic thing about sampling” would be:

    Debarati Guha-Sapir, director of the Centre for Research on Epidemiology of Disasters in Brussels … thinks the WHO’s sample is too small to come up with a definitive number, especially in Iraq where many people have been displaced, and violent deaths tend to be clustered in small areas. “The sample size is not really good enough to represent extremely variable conditions,” she says.

    Another “innumerate” would be:

    Jon Pederson of Fafo, who thinks that “No sample size is big enough, but [around] 10,000 is starting to come into the range where we can do something with the data.”
    http://www.nature.com/news/2008/080109/full/news.2008.426.html

    If the WHO size is still too small (Guha-Sapir) or only “starting” to approach a useable sample size here (Pedersen), where would this leave the vastly smaller L2 sample size?

    Don’t these “innumerates” know that Iraq is a “well stirred soup”? Maybe Tim Lambert needs to teach them about the basics of sampling.

    [btw..in addition to sample size concerns, Steven Moore’s article also criticized (correctly) the lack of demographic data in L2, at which time he was (falsely) smeared for “fabrication” by Les Roberts, as we now know beyond doubt thanks to David Kane pushing Tim to finally, a year later, get a straight answer to this simple question. There’s a good post about this Roberts/Moore episode here: http://www.mediahell.org/community/08010704.htm

  88. #88 David Kane
    January 12, 2008

    Kudos to Tim for getting to the bottom of this, for correcting the post and for apologizing to Munro. This is why I find Tim and Deltoid to be such a useful forum for discussion. We all care about the truth and are working to uncover it.

    Minor quibble: Shouldn’t this quote be corrected in the post as well?

    “As noted earlier, this is an outright fabrication by Munro”.

    A casual reader might not notice the correction several screens of text above.

    Also, don’t you need a correction for this post and an apology to Moore?

    And, finally, are you sure that Roberts is not “wrong” about the demographic information collected in 2004? When last I asked, Roberts told me that any data beyond the cluster summary for L1 was “not available.” Has it vanished? Did the demographic data ever exist? Has anyone outside of the L1 authors seen it?

    I would expect a natural skeptic like Tim to be curious, at least, as to why the data for L2 is available but the exact same data for L1 is not. Why keep that hidden, especially if good demographic data would allow us to compare the L1 sample with the IFHS sample. Cross-checking survey data is one of the hallmarks of good epidemiology.

    By the way, I have heard that the IFHS authors will be interested in taking a look at the L1 data. Roberts should start working now on getting his story straight. He can brush off me all he wants. He can’t ignore them.

  89. #89 sod
    January 12, 2008

    guys, aren t you making slightly too much fuss about this?

    the demographic data is ONLY relevant IF there was fraud. and so far we have seen absolutely ZERO evidence to support this allegations.

    the way in which “David wrong on everything Kane” is demanding apologies freom everyone to everyone, is at best bizarre.

    how far have you spread the news about your inability to calculate a CMR David?

  90. #90 izzy
    January 12, 2008

    I think David is right that Tim should do a little better here to set the record straight. Note that on Tim’s front page, the summary of his posting ends this way:

    “The serious problems are, however, with Munro’s article with fabricated claims, misrepresentation of sources, a misleading graph, and reliance on discredited arguments.”

    Unless someone clicks further to read the full posting and manages to catch the correction buried in there, they will still walk away with Tim’s falsehood that Munro used fabricated claims.

    But I don’t think falsely accusing Munro of fabrication over demographic data is the only misrepresentation in Tim’s conclusions. The “misrepresentation of sources” claim is also rather weak. He accuses Munro of “misrepresenting” Garfield, but the passages he quotes from an email with Garfield to support this claim don’t say or show this. If Garfield hadn’t backed away from earlier statements, such as that he “does believe” L2, as quoted by Tim, it would make no sense for him to tell Munro that he currently believes 250,000. If Garfield still believed L2 his guess would be 650,000 or higher, not 250,000. That a guess is not the same as a study is obvious, but if he believed the study, *then its conclusions would be his guess.*

    Then Tim claims Munro uses a “misleading graph”, but it is just putting the IBC number alongside the Lancet ones. Everyone has discussed or debated the discrepancy between these two sources. All the graph does is illustrate what that discrepancy is. Tim’s claim that this itself is “misleading” is just empty.

    The final claim in Tim’s conclusion is that Munro relies on “discredited arguments”, in that he notes that critics have questioned the sample size as being too small. Yet, as noted above the “discredited argument” is nothing of the sort. People far more knowledgeable on these matters than Tim don’t think it’s a discredited argument. The discredited argument really should be the ‘Iraq as well stirred soup’ one that Tim uses, and most of the ones Tim has used in the past to deflect concerns about small sample size:
    http://scienceblogs.com/deltoid/2008/01/flypaper_for_innumerates_natio.php#comment-708422

    Basically all four of the accusations at the closing of Tim’s summary on this are empty. The fact-to-fiction ratio seems quite a bit better in Munro’s article than in Tim’s knee-jerk ‘rebuttal’.

  91. #91 Tim Lambert
    January 12, 2008

    Thanks for demonstrating that you are as innumerate as Moore or Munro, izzy. Iraq is not a “well stirred soup”. If it was, you could take a convenience sample instead of a random sample. Stirring the soup and taking a sip is analogous to taking a random sample. Pedersen did not say that the L2 sample was too small — why don’t you read what he said about it?

    I don’t need to change my summary because Munro’s “Soros funded it” claim has been now revealed as a fabrication. As Garfield explained in his email, 250,000 is the number he would have guessed without the L2 study, which is why Munro is guilty of misrepresentation.

  92. #92 SG
    January 12, 2008

    Kane,

    I would expect a natural skeptic like Tim to be curious, at least, as to why the data for L2 is available but the exact same data for L1 is not. Why keep that hidden, especially if good demographic data would allow us to compare the L1 sample with the IFHS sample. Cross-checking survey data is one of the hallmarks of good epidemiology.

    why do you keep banging on about this. You have been regularly and consistently refuted. No-one is going to supply their data to a hack like you, and families in the Fallujah cluster (and probably others) can be identified by their demographic profile. Get over it.

  93. #93 izzy
    January 12, 2008

    Tim, Pedersen is quoted by Nature as saying the WHO sample size is “starting” to get to a size where you can do something with the data. Where do you suppose this leaves the vastly smaller L2 sample size?

    “Soros funded it” seems to be true. He did fund it, and this was concealed until Munro revealed it. You posted something from Tirman suggesting that Munro “implied” that Soros was the first funder, and that this is supposedly the misrepresentation, an implied one about who was an initial funder. I didn’t take away that particular ‘implied misrepresentation’ from the Data Bomb article, but then I don’t care much about the “Soros funded it” issue, aside from wondering why this was concealed for over a year.

    But if this implied misrepresentation – if you choose to take Tirman’s word on the events – is sufficient to leave your “misrepresentation” claim, then certainly Pedersen at the very least implies that L2′s sample was too small by what he says about WHO’s in the Nature article. And Guha-Sapir’s statement in the same article certainly can’t be interpreted any other way (another “innumerate” no doubt, in need of your teachings on sampling basics).

    I think it’s time for you to admit that a lot of intelligent and well informed people have this concern regarding sample size, and stop your childish and overly-defensive name-calling.

    “As Garfield explained in his email, 250,000 is the number he would have guessed without the L2 study, which is why Munro is guilty of misrepresentation.”

    Now that is just a bizarre. I don’t see that in his email. Aside from your framing that asserts a misrepresentation, the snippet of email doesn’t say anything that contradicts what he’s presented as saying in the Munro article. As I said before, if he did actually believe the L2 study, that would be his guess. He says L1 convinced him of that number. He does not say the same about L2.

    I think you got the cluster 33 issue wrong too:
    “This is ridiculous. The survey was designed for 50 clusters. Even if the survey was scheduled to end by June 30, it makes no sense to stop then if some provinces were unsampled.”

    The point here is not when they do the interview, but the time frame of deaths the study is supposed to be recording, and the wacky pattern for the deaths when they allowed in stuff past that time frame. I won’t call you a name though. You do enough of that for all of us.

  94. #94 Ian Gould
    January 12, 2008

    Actually there’s no evidence that the survey did continue past June 30th.

    Munro’s claim that they did is based on the assertion that the car bomb fatalities must all relate to the July 1st car-bombing – which is nonsense because a simple examination of the IBC database shows that there were hundreds of casualties from car-bombings during the survey period.

  95. #95 Tim Lambert
    January 13, 2008

    izzy, your attempts to torture the words from Pedersen, Garfield etc to mean something other than what they say are not working. But let’s go back to the beginning. Do you understand the tasting soup analogy yet? Do you see why you don’t need a larger sample for a larger population?

  96. #96 David Kane
    January 13, 2008

    Tim,

    You quote from Roberts include ellipses.

    Their focus in the field was on security and Riyadh said that this made the visits to each area a lot shorter….and I cannot second-guess that judgment call.

    Were those dots in Roberts’ original e-mail or did you edit something out? If the latter, could you tell us what it was? As you may recall, there was a lot of discussion about whether or not there was enough time to do the surveys, so it would be interesting to get more details about interview length.

  97. #97 Tim Lambert
    January 13, 2008

    I did not edit anything of Roberts’ email.

  98. #98 Jason
    January 13, 2008

    Tim Lambert,

    izzy, your attempts to torture the words from Pedersen, Garfield etc to mean something other than what they say are not working.

    The news report from Nature quotes Pedersen as saying:

    “No sample size is big enough, but [around] 10,000 is starting to come into the range where we can do something with the data.”

    What was the sample size in L2? If it was significantly less than 10,000, Pedersen’s statement obviously implies that he thinks it is too small to be useful.

  99. #99 Jason
    January 13, 2008

    Tim Lambert,

    Rebecca Goldin of stats.org is also unimpressed with Munro’s understanding of statistics.

    That’s a pretty absurd characterization of Goldin’s comments. She doesn’t really say whether she thinks the sample size is large enough to be useful, she merely says that “the small sample size is accounted for in the large confidence interval.” In other words, the sample size is adequate in a statistical sense only because the confidence in the “655,000 excess deaths” estimate is so low.

    Goldin clearly thinks there is merit to the “main-street bias” criticism and to the suspicions that the data may have been faked or manipulated, writing that “the Journal made a convincing argument that the data may well have been tweaked.”

  100. #100 BruceR
    January 15, 2008

    Izzy is right that there is something of a change of opinion indicated when one sees these two statements when coming from the same person a year apart:

    2006: “There is no reasonable way to not conclude that [~600,000 violent deaths] is by far the most accurate information now available.” (2006)

    2007: “I feel confident that at least [250,000] have died due to violence since the 2003 invasion…”

    If Garfield still maintains L2 was “the most accurate information now available” as Tim asserts without supporting quotation in this post, then it’s hard to explain why Garfield’s states his personal confidence as lying now instead in a significantly lower number… unless, as the Munro article asserts, his confidence in the 2006 study has eroded somewhat, either in the face of criticism or new data. I fail to see any of the misrepresentation that Garfield and Tim allege.