Tim Lambert (deltoidblog AT gmail.com) is a computer scientist at the University of New South Wales.
Anjana Ahuja has written an extraordinarily one-sided article attacking the Lancet study. She drags out the same criticisms that were covered in the Nature story, but even though she cites the Nature piece, she carefully avoids mentioning the Lancet authors' replies, or the opinions of the researchers supporting the study. Ahuja also makes many factual errors, even going as far as claiming that one of the interviewers contradicted Burnham when, in fact, they supported him. All of Ahuja's errors are in the direction of supportting her case, suggesting that she is biased.
Ahuja begins:
Iraq Body Count, an antiwar web-based charity that monitors news sources, put the civilian death toll for the same period at just under 50,000,
This is untrue. The IBC just counts deaths reported in the media. It is not a count of the total number of deaths.
One critic is Professor Michael Spagat, a statistician from Royal Holloway College, University of London. He and colleagues at Oxford University point to the possibility of "main street bias" -- that people living near major thoroughfares are more at risk from car bombs and other urban menaces. Thus, the figures arrived at were likely to exceed the true number. The Lancet study authors initially told The Times that "there was no main street bias" and later amended their reply to "no evidence of a main street bias".
Spagat is an economist and not a statistician, and has no experience in conducting surveys. His colleagues at Oxford are physicists who also have no experience in conducting surveys. Spagat and co know full well that most of deaths occurred outside the home so it matters little where they live. Spagat et al's contrived analysis was only able to make the alleged "main street bias" matter by making absurd assumptions like that they only sampled 10% of the population, and that 90% of the population of Iraq virtually never used main streets for travel or shopping.
And don't you just love the way Ahuja plays "Gotcha!" with the slight change in wording from the authors?
Professor Spagat says the Lancet paper contains misrepresentations of mortality figures suggested by other organisations, an inaccurate graph, the use of the word "casualties" to mean deaths rather than deaths plus injuries, and the perplexing finding that child deaths have fallen.
The authors acknowledged that their graph labeled "casualties" as "deaths" and erroneously compared rates and accumulated counts. So that's is two errors that Spagat correctly reported, but the next mistakes are Spagat's: The word "casualties" does not even appear in the body of the paper. And the study actually found that child deaths increased. In one paragraph Spagat made as many mistakes as he was able to list from the Lancet study.
"The authors ignore contrary evidence, cherry-pick and manipulate supporting evidence and evade inconvenient questions," contends Professor Spagat, who believes the paper was poorly reviewed. "They published a sampling methodology that can overestimate deaths by a wide margin but respond to criticism by claiming that they did not actually follow the procedures that they stated." The paper had "no scientific standing". Did he rule out the possibility of fraud? "No."
I guess Spagat thinks that they should have got economists and physicists to review the paper instead of statisticians and epidemiologists. I think the reviewers were able to figure out what the sampling methodology was from the paper. Unlike Spagat, who came up with the interpretation that they said that they only sampled 10% of the country and when told that he had misunderstood what they had done accused them of not following the procedures that they has stated. And can we rule out the possibility of fraud in Spagat's work? No.
If you factor in politics, the heat increases. One of The Lancet authors, Dr Les Roberts, campaigned for a Democrat seat in the US House of Representatives and has spoken out against the war. Dr Richard Horton, editor of the The Lancet is also antiwar.
And The Times, like every single Murdoch paper, was stridently pro-war. This might explain why Ahuja's article is biased.
Dr Richard Garfield, an American academic who had collaborated with the authors on an earlier study, declined to join this one because he did not think that the risk to the interviewers was justifiable. Together with Professor Hans Rosling and Dr Johan Von Schreeb at the Karolinska Institute in Stockholm, Dr Garfield wrote to The Lancet to insist there must be a "substantial reporting error" because Burnham et al suggest that child deaths had dropped by two thirds since the invasion. The idea that war prevents children dying, Dr Garfield implies, points to something amiss.
No, Burnham et al do not suggest that child deaths dropped since the invasion (they increased), and Garfield did not say that the study suggested that either. Garfield suggested that because the child mortality rate was much lower than from surveys conducted in the 90s, the study had undercounted child deaths. And Ahuja has cherry picked her quotes from Garfield. Here is what he thinks about the accuracy of the study:
I am shocked that it is so high, it is hard to believe, and I do believe it. There is no reasonable way to not conclude that this study is by far the most accurate information now available.
Back to Ahuja's story:
Professor Rosling told The Times that interviewees may have reported family members as dead to conceal the fact that relatives were in hiding, had fled the country, or had joined the police or militia. Young men can also be associated with several households (as a son, a husband or brother), so the same death might have been reported several times.
However, they were able to produce death certificates, so it is not credible that they invented the deaths. And if you wanted to conceal that someone had joined the militia, why not just not say that they had joined the militia instead of concocting a lie? As for double counting, that is very easy to check against and the researchers made sure that deaths were not counted twice.
Another critic is Dr Madelyn Hsaio-Rei Hicks, of the Institute of Psychiatry in London, who specialises in surveying communities in conflict. In her letter to The Lancet, she pointed out that it was unfeasible for the Iraqi interviewing team to have covered 40 households in a day, as claimed. ...
Professor Burnham says the doctors worked in pairs and that interviews "took about 20 minutes". The journal Nature, however, alleged last week that one of the Iraqi interviewers contradicts this.
Only if by "contradicts" you mean "confirmed". Here's the Nature story:
The US authors subsequently said that each team split into two pairs, a workload that is "doable", says Paul Spiegel, an epidemiologist at the United Nations High Commission for Refugees in Geneva, who carried out similar surveys in Kosovo and Ethiopia. After being asked by Nature whether even this system allowed enough time, author Les Roberts of Johns Hopkins said that the four individuals in a team often worked independently. But an Iraqi researcher involved in the data collection, who asked not to be named because he fears that press attention could make him the target of attacks, told Nature this never happened. Roberts later said that he had been referring to the procedure used in a 2004 mortality survey carried out in Iraq with the same team (L. Roberts et al. Lancet 364, 1857-1864; 2004).
So the Iraqi researcher told Nature that they worked in pairs, which Spiegel says is doable. But Ahuja, after reading that very paragraph, carefully avoids mentioning Spiegel's opinion, presumably because he is an epidemiologist with experience in such surveys as opposed to the psychiatrist Hicks. And she falsely claims that the Iraqi researcher says that they didn't work in pairs when they did.
Ahuja's piece is a disgrace.
Email this entry to a friend
View the Technorati Link Cosmos for this entry
Comments
Extraordinary, isn't it? I liked this bit best:
Several academics have tried to find out how the Lancet study was conducted; none regards their queries as having been addressed satisfactorily
So out of the population of people who have outstanding queries, 100% of them haven't had their queries resolved? When someone puts this in their third paragraph, you pretty much know what level of understanding they are working at when it comes to sampling theory.
The thing that confuses me is that the Garfield letter is printed in the Lancet correspondence, and all of Spagat's objections are (garbled versions of) letters in the correspondence. But the author doesn't mention Burnham's "reply to critics" at all - she actually quite strongly implies that it didn't exist, in claiming that Burnham hasn't responded to Garfield etc on birth rates. Did Spagat not tell her that it existed, did he tell her and she didn't bother to read it, or did she read it and not bother to mention it.
A spot of "Kaplan's Fallacy", btw - Garfield clearly says in the letter that the undercounting of child deaths might account for +/- 30% variance in the estimate, but the article argues as if it's definitely -30%.
Posted by: dsquared | March 5, 2007 2:35 PM
The title says it all "Could 650,000 Iraqis really have died because of the invasion?"
It's called "proof by incredulity" and is typically used when all other efforts to dismiss the evidence have failed (usually miserably).
Here are some titles in a similar vein:
"Could men really have landed on the moon in 1969?"
"Could humans [as opposed to aliens] really have built the pyramids 5000 years ago?"
Posted by: JB | March 5, 2007 3:29 PM
Oh, and let us not forget the most oft-cited "Proof by Incredulity" of all:
"Could human beings ever have evolved from single-celled organisms as Darwin proposed?
Posted by: JB | March 5, 2007 3:51 PM
The IBC is obviously wrong, if they are only counting what gets reported in the media. I don't know if every death in America gets reported in the media.
Much less, the people who die in a country that is going through a civil war and that doesn't appear to have a strong history of media independence.
It just seems obvious.
Posted by: Thom | March 5, 2007 3:54 PM
dsquared writes:
This is vaguely off topic, but I know several academics with concerns about the Lancet studies (especially the survey details), who have sought (unsuccessfully) to get satisfactory answers to their questions. Do you know any such academics who have received satisfactory answers? (There are lots of academics who have always thought (and said) that the Lancet studies are wonderful. I am curious about those with concerns that have had their concerns addressed.)
If you don't know any, wouldn't that suggest that the statement is accurate?
By the way, there is some hope that the data will be available in the future. Kudos to all those who have fought this lonely fight (and to the Lancet authors for agreeing to this reasonable request).
Posted by: David Kane | March 5, 2007 4:09 PM
IBC, I think, would say that their count includes the government and hospital death tolls, so it comes down to whether one thinks those numbers are complete or better than 50 percent complete, as opposed to missing 90 percent or more. (Or some intermediate figure--if the Lancet papers were discredited tomorrow it wouldn't necessarily mean IBC was right.)
Posted by: Donald Johnson | March 5, 2007 4:14 PM
From the IBC's own FAQ
"It is likely that many if not most civilian casualties will go unreported by the media. That is the sad nature of war."
So it's pretty obvious that they are low balling the true number. At best, they can be considered a bottom figure. But any newspaper that reports their number without taking this into account is doing a horrible job of reporting.
If the mainstream media are so interested in accuracy, then where are all the news stories that are as critical of IBC as they are of the Hopkins study?
Posted by: Thom | March 5, 2007 4:38 PM
David Kane wrote:
Perhaps, but I do know at least a couple, so that makes the statement inaccurate.
Posted by: Robert | March 5, 2007 4:42 PM
David Kane asks: If you don't know any [academics who have received satisfactory answers], wouldn't that suggest that the statement is accurate?
Not in any sense of the words "suggest" and "accurate" that a science reporter ought to be using. As for the kudos to people who have "fought this lonely fight", is it seemly to be congratulating yourself thusly? (I take it that accusing people of fraud counts as fighting; it certainly smacks of looking for a fight.) And what do you propose to do with the data? If you are right and the whole thing was cooked, do you suppose Burnham et al made such a hash of it that the data will incriminate them?
It's not hard to guess what will happen when the data is made available. The carpers will say that what we really need is to hear the Iraqis describe in detail just how they implemented the instructions they were given. One of the interviewers is dead and it isn't likely that the others will be in any hurry to identify themselves in public. Certain powerful people have made it very clear that they don't approve of what they were doing. So the MSB squad will still have their Devastating Critique ((C)d-squared), data or no data.
Posted by: Kevin Donoghue | March 5, 2007 4:45 PM
Do you know any such academics who have received satisfactory answers?"
"Satisfactory" is in the eye of the beholder and not all beholders are equally qualified to assess the validity of the results.
If an academic with no understanding of sampling methodology questions the results, it is basically meaningless.
Posted by: JB | March 5, 2007 5:14 PM
Tim,
The headline should read:
"London Times hatchet job on last shreds on Lambert credibility"
There are few people in Australia that are so consistently wrong as you are. Perhaps John Quiggin might have you covered in that regard. From DDT to Global Warming to Iraq death counts you manage to end up on the side that can't discern opinion from truth, an emerging phenomenon in the Internet Age.
Posted by: Jack Lacton | March 5, 2007 6:09 PM
One positive development is that Josh Dougherty's contribution to the Lancet correspondence is a lot more sensible and civilly worded than his occasional contributions to comments threads here.
Posted by: dsquared | March 5, 2007 6:17 PM
Perhaps it's worth pointing out that, according to his CV, Prof Spagat enjoys lucrative consultancy contracts (totalling at least $300,000) with a company called Radiance Technologies. I assume this is the outfit based in Huntsville AL, which supplies sensor systems to the US military in Iraq.
Posted by: abc123 | March 5, 2007 6:40 PM
I knew that "boot" means "trunk" in the UK, but had no idea that "contradict" and "confirm" were synonyms there. The things you learn from the respectable news.
Posted by: QrazyQat | March 5, 2007 6:44 PM
Jack L, you forgot how wrong we got it on evolution, and how the Republican sweep in the 2006 election indicated that the US public has seen through our absurd claims that Iraq is not, in fact, the safest place in the world, except for the possibility of tripping over the sweets and flowers with which US visitors are daily garlanded.
Posted by: John Quiggin | March 5, 2007 7:18 PM
Jack Lacton said
"last shreds on Lambert credibility"
Not to be nitpicky of anything, but shouldn't that be "last shreds of" ?
"Few people in Australia that are so consistently wrong"
and shouldn't that be "Few people in Australia that are as consistently wrong "?
Where do these people learn their English anyway?
Posted by: JB | March 5, 2007 7:31 PM
Tim,
Perhaps you are simply a very subtle reasoner, but I didn't quite get how Les Roberts altering his story after being contradicted by one of his interviewers is implicit support of the integrity of the research. According to the quotation you offered, he made a claim about the interviews in response to being questioned that he retracted after an actual interviewer contradicted him. To me, that somehow doesn't smack of integrity.
And does Hicks being a psychiatrist specializing in surveying communities in conflict dequalify her to make a professional judgement on the Lancet study? If you think psychiatry is irrelevant to the topic, I must say epidemiology doesn't seem that much more [or less] pertinent to war.
Posted by: Kevin | March 5, 2007 7:33 PM
JB - I tried to fix the 'of' but you can't post twice within a short space of time so I didn't worry about it. I don't know where you learned English but your 'so consistently' vs 'as consistently' is flat out wrong. Both are grammatically fine.
Prof Quiggin - Who argues evolution? Only those that made #10 on my list of institutions that ruin the world at http://tinyurl.com/yt8uv5, and who argues that it's difficult in Iraq. If you believe that we weren't greeted to cheers and garlands initially then you're not dealing with truth. The best comment that I've heard about Iraq was from a man that was a janitor (or some such thing) who went to work every day in a very dangerous part of Baghdad and in which he was not of the right religion for the area. When asked whether things were better know than when Saddam was in charge he replied that it absolutely was. When asked why he responded that under Saddam you went about your business, might get killed, and had no hope. Now you go about your business knowing that you might get killed but there's hope for a better future.
Posted by: Jack Lacton | March 5, 2007 8:08 PM
Kevin, here's what Roberts wrote in their reply in the Lancet:
ie the four interviewers worked in two teams of two interviewers. This was confirmed by the interviewer that spoke to Nature. Apparently Roberts also made a statement about the 2004 study that was misunderstood to refer to the 2006 study. Correcting a misunderstanding is not the same as making a retraction.
Posted by: Tim Lambert | March 5, 2007 8:48 PM
"Apparently Roberts also made a statement about the 2004 study that was misunderstood to refer to the 2006 study. Correcting a misunderstanding is not the same as making a retraction."
People questioned the methods used to draw the 2006 sample. In response to this, Roberts says the "teams" all went off individually, so those questions about the 2006 study are "not valid".
It was "misunderstood" because it was a misleading answer (while the only correct answer would have been "I have no idea how to answer these questions").
Posted by: joshd | March 5, 2007 10:41 PM
A nitpick: If you factor in politics, the heat increases. One of The Lancet authors, Dr Les Roberts, campaigned for a Democrat seat in the US House of Representatives and has spoken out against the war. Dr Richard Horton, editor of the The Lancet is also antiwar.
That would be a Democratic seat, the use of the term "Democrat" in this context is nearly universally understood to be a (childish) slur in the United States (members of the Democratic Party are Democrats, but things related to the party are Democratic. It would hvae been correct to say "a seat as a Democrat" but not "a Democrat seat.)
A fairly telling exposition of the author's bias.
Posted by: Lettiuce | March 5, 2007 11:00 PM
Tim,
It still seems mysterious to me how Roberts either mistook a question about the feasibility of the 2006 survey for a question about the 2004 survey such that he either gave a wrong answer or said something so vague it misled the reporter. I do agree retracting and correcting misunderstandings are different things.
I found this in the introduction of the Lancet paper:
"Recently, Iraqi casualty data from the Multi-National Corps-Iraq (MNC-I) Significant Activities database were released.5 These data estimated the civilian casuality [sic] rate at 117 deaths per day between May, 2005, and June, 2006, on the basis of deaths that occurred in events to which the coalition responded."
This seems close enough to be prima facie support of Spagat's critique, regarding the paper containing mistaken references to casualties. Mistakenly pluralizing a term that was plainly in the intro seems significantly less a problem than conflating casualty and death.
And just generally, given the Lancet's rundown of what was asked, the estimate of 20 minutes per interview seems unlikely to me. 20 minutes to enter someone's house, explain your purpose and caveats of your survey, get settled in for note-taking, then discuss an emotionally and politically touchy subject like the death of a loved one and chatter inducing subjects like family births, deaths, migrations and what have you, wait for someone to get a death certificate in 8 of 10 households, read it, discuss it, maybe ask more questions in case of conflict and then take one's leave sounds improbable. The possibility of someone just breaking down and crying alone makes this figure sound low. And if Les Roberts didn't cite the correct number of groups going out and doing interviews or couldn't make himself plain enough to be understood, might he have accidentally misstated the time needed for interviews or misled the reporter by commenting on the time it takes him to shower in the morning rather than answering the question at hand?
Posted by: Kevin | March 5, 2007 11:11 PM
Here's the exact quote
The thing that gives it away that Roberts was talking about the 2004 survey was the fact that he said "in 2004". Twice. And it seems I have to spell out the relevance of the statement about the 2004 study for you. If in 2004, a two person team interviewing individually took about three hours for a 30 house cluster, in 2006, a four person team working in pairs and asking the same questions would be expected to take how many hours for a 40 house cluster?
Oh, and Josh, will the IBC be contacting Ahuja about her misrepresentation of the IBC number in her article?
Posted by: Tim Lambert | March 5, 2007 11:28 PM
Getting back to Tim's own ironically-titled hatchet job, the author for the Times appears to make a couple minor mistakes that Tim tries to make a lot of hay over to serve his own biased agenda.
She confuses the points over which the Lancet authors have been caught fibbing. She may also have confused the child deaths issue with the finding of non-violent deaths going down quite a bit over the first two or three years after the invasion (while L1 finds them going up in the equivalent period - another one of those "striking similarities" with the two studies, providing "strong validation" for both).
Tim can't help making a few errors of his own either:
"the next mistakes are Spagat's: The word "casualties" does not even appear in the body of the paper."
As usual, the mistake is Tim's. Whether the word appears or not is a red-herring of Tim's to begin with, but it appears in the exact way described by Spagat, with "casualties" and "deaths" being interpreted as the same thing: "These [DoD] data estimated the civilian casuality rate at 117 deaths per day between May, 2005, and June, 2006".
(Apparently Tim looked only for the plural in his Expert word search on the document).
Before this, Tim starts his hatchet job off with an absurd complaint over a reference to IBC which he says is "untrue" and supports by referencing a fraudulent analysis he'd written distorting news stories to help support a disgraceful and disinformative smear campaign against IBC in which both he and Expert Les Roberts were up to their eyeballs.
In fact, the Times author's reference is perfectly reasonable and more accurate than the reference written in the 'peer-reviewed scientific paper' that we're all supposed to worship:
"The best known is the Iraq Body Count, which estimated that, up to September 26, 2006, between 43,491 and 48,283 Iraqis have been killed since the invasion."
The Times author gets the fact that IBC refers to civilians, while the peer-reviewed paper from the Experts is, if anything, rather less accurate. It is still more inaccurate if considering this lie the Experts added for good measure: "Estimates from the Iraqi Ministry of the Interior were 75% higher than those based on the Iraq Body Count from the same period."
Perhaps this and many other peer-reviewed lies are what Spagat was referring to when he said their paper contains: "misrepresentations of mortality figures suggested by other organisations".
Tim also issues a bunch of red herrings, misleading claims and speculations to dismiss MSB, like the assertion (from data with Lancet did not collect) that most deaths were outside the home. But of course MSB already considers this. And then Tim speculates that some of the assumptions tested out in the MSB paper (and on which MSB itself does not depend) are "absurd", even as he (like Roberts) has no idea how a "main street" was defined here, or what percentage of Iraqis from which locations might use them, or how often.
The MSB paper I read (and which Tim appears to mostly have not read, if the 'analysis' to which he links is any indication) makes it clear that there is no way to know how much the biased sampling scheme might have biased the results until more is known. Until then, one can only speculate and use various assumptions to try to get some idea of what the effect might be. Tim chooses to speculate that one set of such assumptions tested out in the MSB paper are "absurd". But Tim has no idea if they're "absurd" or not, and neither do the Lancet authors. What is obvious though is that Tim will speculate or say anything at all that will puff up this study and deflect criticisms of it. That seems to be his major role in all this and has been for several years now. In any case, Tim's dissembling about MSB was addressed pretty well in the linked article by an amateur (Robert Shone) who tried in vain to set Expert Tim straight on all the things he got wrong.
The rest of Tim's hatchet job consists mostly of ad hominems, which I don't know are correct or not (judging by them being from Tim it's probably safer to assume not), but are the standard ad hominem diversions from the substance of the arguments.
Posted by: joshd | March 5, 2007 11:32 PM
Tim the passage you quote is not the quote, let alone an "exact quote" of Roberts saying, in response to questions/criticisms about sampling in the 2006 study that the "teams" split off with each member working independently ("teams" of 1).
Furthermore, the passage you quote was addressed by Hicks here, and Roberts' claims there only make matters worse: http://www.hicn.org/research_design/rdn3.pdf
Posted by: joshd | March 5, 2007 11:47 PM
As someone who conducts surveys for a living, I will second Kevin's comments. I find almost every aspect of the fieldwork as described implausible, specifically the response rate, the interview rate (time per interview) and the incidence of death certificates being produced.
The Lancet paper itself refers to the difficulties the field teams encountered including stops at roadlocks and lengthy explanations being required to gain the confidence of respondents.
There was no attempt to validate the interviews (in market research in Western countries typically 10% of interviews are audited). The study authors were not even in the country at the time, so nobody knows what the fieldworkers actually did. I suspect that faced with a difficult and dangerous task, they simply made many of the interviews up.
I know that if I ever got a survey back with a claimed response rate of over 98% and the kind of daily completion rate as claimed, I'd order an audit in a second.
Finally, anticipating Tim telling me that 98% response rates are typical in Iraq, I don't believe it. People decline to be interviewed for all sorts of reasons which are not culturally dependent - they need to go to the toilet, they're about to eat, they don't feel well, they are deaf, blind or dumb, they are infirm, they're about to have sex, they are mentally ill, they have friends over, they are working to a deadline etc etc
98% response rate, dream on.
Posted by: James | March 5, 2007 11:56 PM
Shorter Josh: "Anyone who disagrees with me is a liar".
This time he has added to his list Michael O'Hanlon and Jason Campbell who wrote:
Posted by: Tim Lambert | March 6, 2007 12:10 AM
So James, you think that all surveys ever conducted in Iraq have been fraudulent. Got it.
Posted by: Tim Lambert | March 6, 2007 12:21 AM
I haven't added anyone to the list Tim. Roberts was already on it.
What does "our Iraq Body Count-based estimate" mean.
O'Hanlon and Campbell explain directly prior (and which Roberts must have seen) that this means a number significantly reduced from the IBC figure (with morgue entries and other things removed), and is NOT the "best known" figure you would have seen on the IBC website and which is deceptively quoted and put alongside this claim in the Lancet paper. The actual IBC figures, as opposed to "our" (O'Hanlon/Campbell's) reduced version, were higher than the Iraqi figures being compared to it.
You will not see the explanation given by O'Hanlon/Camp in the Lancet report because Les Roberts is attempting to deceive his readers about IBC by wrenching a factoid out of context, putting it alongside IBC's actual figures, and using it to misreprepresent the "best known" IBC as being lower than everything else, which is right in line with the nonsense he concocted in his ludicrous "sensitivity analysis" of "8 independent studies" from 2005.
Posted by: joshd | March 6, 2007 12:39 AM
Not at all, Tim. But having been in the industry for 30 years, forgive me if I am unsurprised if market research companies operating in the third world exaggerate their response rates (I'm not talking here about the Lancet authors).
I think my point is nobody gets 98% response rates for door-to-door work anywhere in the world, for reasons that I have explained.
Finally, I never mentioned "fraud". I'm saying that nobody knows what the fieldworkers did, but that what they claimed to have done doesn't add up.
Posted by: James | March 6, 2007 12:41 AM
I think my point is nobody gets 98% response rates for door-to-door work anywhere in the world
Really?
Posted by: spartikus | March 6, 2007 1:24 AM
Kevin enumerated a vast sequence of steps that could not possibly be carried out in 20 minutes ...
[And just generally, given the Lancet's rundown of what was asked, the estimate of 20 minutes per interview seems unlikely to me. 20 minutes to enter someone's house, explain your purpose and caveats of your survey, get settled in for note-taking,]
shall we say maybe 5 minutes so far?
[ then discuss an emotionally and politically touchy subject like the death of a loved one and chatter inducing subjects like family births, deaths, migrations and what have you, wait for someone to get a death certificate in 8 of 10 households (sic - dd) , read it, discuss it, maybe ask more questions in case of conflict and then take one's leave sounds improbable.]
certainly does seem improbable that this last bit would only take 15 minutes. However ...
Kevin appears to have missed the fact that not every house in the sample would have had a death in it. In fact there were a total of 629 deaths (547 post-invasion, 82 pre-invastion). Therefore, even in the worst case in which each of those deaths took place in a different household, 1220 of the households surveyed would have reported no deaths (that's 66% of all households).
If 1220 households took an average of 5 minutes to survey (rounding up to make the arithmetic easier), but the total sample of 1849 households to an average 20 minutes to survey, then how long did the (maximum of) 629 households with a death in them take to survey? This is a GCSE maths question. And the answer is [(1849 x20) - (1220 x 5)]/629 = 49.09 minutes.
Posted by: dsquared | March 6, 2007 4:13 AM
The main point that one should make of the Times article is that it rigidly conforms to a pattern that epitomizes our corporate-state 'mainstream' media which must forever downplay or ignore western crimes and forever highlight crimes committed by officially designated enemies, even if the evidence for the latter is fragmentary. This is because part of the foundation of the western 'creed' is that we are, by definition, the 'good guys' who support noble values such as social justice, peace, freedom, and democracy in our foreign policy and thus our media acts as a conduit for this myth of our basic benevolence. The media will occasionally admit that our governments make mistakes in carrying out noble deeds, but the idea that our leaders are calculating liars who have an alternate agenda and who are complicit in mass murder is beyond the pale - never to be acknowledged. When the established order is as divided as they were in the US-UK war of aggression against Iraq, then this makes it increasingly difficult to promote the western 'creed' of benevolence. This explains why the media (including the hatchet job done by the Times) has flexed such muscle in downplaying the carnage in Iraq - in effect it is rallying around the political elites in power who are responsible for the slaughter, because of the damage this has done to their reputation and to myth of western nobility.
Note how the same media only emphasized Saddam Hussein's crimes after he had invaded Kuwait and had 'slipped the leash'. Until then his crimes were largely ignored, because he was 'a man with whom we could do business' in the famous words of Margaret Thatcher, defending Saddam after the Halabja massacre. Similarly, the crimes of other western proxies such as Suharto (whose crimes dwarf even those of Saddam's) were mostly excluded from the western media, at least until he became uppity in 1998 and started challenging IMF rules. Suddenly, the mainstream media shook off its collective amnesia and regained its mental faculty with regard to this world class torturer and mass murderer.
If the study had been conducted using exactly the same methods but had estimated the body count of an aggressive war carried out by an officially designated enemy, I have no doubts that our media would have promoted the study to the hilt, giving it prominent coverage over an extended time. But the utter destruction of Iraq must be downplayed because we are the culprits. Have hundreds of thousands died in Iraq since March 2003? Almost certainly. Did hundreds of thousands of civilians die under the sanctions regime that preceded it? Almost certainly. Was the bombing of Iraq in 1991 aimed at destroying the country's civilian infrastructure? Most definitely. But to acknowledge the fact that our western governments are criminal entities is taboo, hence why the Times and most of the other western MSM sources have either downplayed or attacked the conclusions of the Lancet study. What else do we expect when the main aim of the MSM is to 'support and defend the political, economic and social agenda of the privileged groups that dominate society and the state', which is the propaganda model of Herman and Chomsky (1988).
Posted by: Jeff Harvey | March 6, 2007 5:28 AM
Can anyone explain the point in the white Times piece that the Lancet suurvey actually had the children body count lower than before the war started?
This is an interesting stat. to ponder.
Posted by: JC | March 6, 2007 5:42 AM
JC: the explanation of this point is that someone (almost certainly the Times journalist) has got their facts wrong. The Lancet paper does not find this, as you can see in Table 2. For infant deaths, 11 were recorded before the war (15 months) and 29 after the war (40 months), giving an unchanged rate of 0.73/month in the sample. For total deaths of children under 15, the figures were 14 and 66, meaning that the number of deaths per month rose from 1.3/ month to 1.65/ month. They didn't fall.
von Scheel, Rosling and Garfield note that the overall infant and child death rate (for children under 15) is quite a lot different from other surveys of the under-5 death rate. This is an interesting point, and it is acknowledged in Burnham's response, but the Times has just got the wrong end of the stick.
Posted by: dsquared | March 6, 2007 5:58 AM
Jeff Harvey: "Most definitely. But to acknowledge the fact that our western governments are criminal entities is taboo, hence why the Times and most of the other western MSM sources have either downplayed or attacked the conclusions of the Lancet study."
This has a bit of a conspiracy tone to it, but there is a great deal of truth to some of this, but for different reasons. I don't think that the MSM is somehow trying to prop up our government, but they are complicit for different reasons.
The main reason is simply that most of the sources on a story like this come from government officials, even if they do not appear in the story. We can see that this happened with Weapons of Mass Destruction, in which the media was completely wrong, because it was relying on its cadre of contacts inside the government.
So the media is never going to be too critical of the government policy simply because the nature of the relationship between media and government. There's just very little independence.
There's also the problem of the American media's timidity the need to appear "balanced," and the fear by journalists to not appear unpatriotic.
Posted by: Thom | March 6, 2007 7:47 AM
Thom,
Thanks for some insightful points. However, I don't think what we read in the media has anything to do with a conspiracy at all. I believe that most journalists truly believe what they write, even if it appears to be exceedingly biased. But because the media is often owned by big corporate interests, or depends on these same entities for advertising revenue, then I think what we read is, to partially quote (I think) the Glasgow University Media Group, not a 'natural or neutral phenomenon but a manufactured product of ideology'. In other words, there is a filtering process through which media selects for certain views and perspectives. These views are in line with the underlying political ideology in which we live; those with differing views are marginalized.
There's no doubt that a few excellent journalists do challenge the prevailing wisdom, even in the mainstream 'liberal' press, but the system ensures that the numbers are stacked against them. I suggest a read of the media chapter of Mark Curtis' excellent book on British foreign policy, 'Web of Deceit', or the equally excellent book, 'Guardians of Truth' by David Edwards and David Cromwell of Media Lens. Their analyses provide empirical evidence that the MSM bolsters a specific agenda while marginalizng or ignoring alternate views.. By the way, they also mention the relibaility of the MSM on 'official' sources, but part of the problem they allude to is that the MSM rarely challenges these sources. This at least partially explains why the Iraq war party was able to build up such popular support in the U.S. for its illegal war.
Posted by: Jeff Harvey | March 6, 2007 8:57 AM
Spagat is an economist and not a statistician
You might want to reconsider this point...... Sebb
P.S. To spell it out - economics IS statistics.
Posted by: Sebb | March 6, 2007 9:41 AM
Sebb wrote:
And geography IS climatology.
Posted by: Tim Ball's sock | March 6, 2007 9:53 AM
Most economists use statistical methods (many badly). This is v2.1 of math is not a science.
Posted by: Eli Rabett | March 6, 2007 10:56 AM
Informed verdicts on MSB:
Tim Lambert: [Main street bias is] "bogus".
Stephen Soldz [Main street bias is] "dishonest".
Jon Pedersen: [Main street bias is] "certainly an important problem for many surveys - not only the Iraq Lancet one".
Posted by: Robert Shone | March 6, 2007 2:12 PM
Robert, among the informed verdicts was this one you got from Robert Chung:
"I'm sure that you are correct, and that many people are simply dismissing [MSB] out of hand- but I don't believe many professionals do. I do, however, believe that for this particular situation many professionals dismiss an overall bias factor of 3."
The merit of this comment is that it distinguishes between the problem (which may perhaps be what Pederson had in mind) and the model presented by Johnson et al. Your brief anthology confuses these very different things.
It's a remarkable thing that five people got together to write a paper to tell us something we all knew already: that when you pluck the parameters of a model out of your ass, you can always get the conclusions you want.
Posted by: Kevin Donoghue | March 6, 2007 3:02 PM
Thanks for your insightful and scientific remarks, Kevin. I can now update my "anthology":
Tim Lambert: [Main street bias is] "bogus".
Stephen Soldz: [Main street bias is] "dishonest".
Kevin Donoghue: [Main street bias is] "ass-plucking".
Any more contributors of scientific wisdom to this "science" blog discussion?
Posted by: Robert Shone | March 6, 2007 7:00 PM
JoshD: Thanks for the link to Hicks piece. It was far more carefully considered than my laundry list but I was gratified to see some of my concerns mirrored there.
James: Out of curiousity do you have any experience with average times for obtaining "informed consent" from interviewees? That piece alone sounds like a stumbling block for the proposed length of time in the Lancet interviews.
DD: I would recommend that Hicks article to you. It clarified the points of contention much more accurately than I managed in one run-on sentence. And you're right, I should have writted that death certificates were produced, according to the Lancet, in 8 of 10 houses which claimed a death. Further, I would recommend reading the actual methodology the Lancet published for yourself, or perhaps rereading it, because their explanations of the care and sensitivity with which the interviews were conducted will give you a more accurate grasp of the issue than my vastly curtailed summary. I would submit that even sans a death in a household, the other questions, explanations and assurances would either last significantly longer than the five minutes you've posited or they would have to be taking shortcuts in the stated methodology. After getting settled in, then the questioning starts, which even if there was no death would take a fair amount of time given what they were asking.
Moreover, in the Hicks article, her analysis of Les Roberts explanations leading to a figure of 3 to 6 minutes average per household is interesting. Obtaining informed consent, implying both that the interviewee consents to the interview without feeling rushed or coerced and is adequately informed to make a rational decision, can according to her take more than 6 minutes alone.
So I salute your spirited rebuttal, but I really think the evidence weighs in favor of some skepticism on Burnham et al. 2006, as least as far as the claims and counter claims have been presented on this blog and in various FAQS I've seen cited. Maybe the authors of the study could clear it up and end our armchair speculating.
Posted by: Kevin | March 6, 2007 8:34 PM
So the media is never going to be too critical of the government policy simply because the nature of the relationship between media and government. There's just very little independence.
There's also the problem of the American media's timidity the need to appear "balanced," and the fear by journalists to not appear unpatriotic."
And let us not forget the desire of some within the media (Judith Miller, Bob Woodward) to feel self-important and to rub elbows with those making the life and death decisions.
Posted by: JB | March 6, 2007 9:13 PM
Hicks' paper is flatly wrong in her analysis of Roberts. Here's the quote again:
Roberts says that in 2004 the surveys took "about twice as long" as 7 minutes. That's about 15 minutes. Double check: Each interviewer does 15 houses in three hours, 180 minutes/15 is 12 minutes per interview. Most households did not have a death to report, so the interview can easily be done in 10 minutes.
Posted by: Tim Lambert | March 6, 2007 9:16 PM
Main street bias in and of itself is not "bogus, dishonest, ass-plucking", but claiming that the Lancet results are rendered null and void by main street bias, before one even has a clear idea of the methodology that was used in the study (as Spagat et al did) certainly qualifies as "bogus (if not dishonest) ass-plucking".
And claiming (or even implying) that the IBC count is anything other than an underestimate is certainly "bogus, thoroughly dishonest, ass-plucking."
Posted by: JB | March 6, 2007 10:19 PM
Tim,
Call me slow, I like having implications spelled out; it makes for clear discussion. Your quote of Roberts' comment still doesn't seem relevant to your initial blog entry which was the focus of my comment.
The Nature quotation you provided has Roberts responding to Nature itself that 'sometimes the four individuals worked independently.' I see no more or less implication of this but that Roberts meant they worked singly, I don't know of another way for individuals to work independently of one another, and that Roberts was replying to Nature directly.
Your 'exact quote' to help me clear up my confusion has this provenance: "Joe Emersberger from Canada, who follows this issue closely, collected some of the expert criticisms of the report and a selection was put to Mr Roberts." As far as I can tell Mr. Emersberger is a Canadian engineer and not a writer for Nature, Jim Giles wrote the Nature article [no clue who interviewed Roberts], and Les Roberts was replying directly to Emersberger's question in your 'exact quote.'
Your 'exact quote' doesn't quote Roberts saying the individuals worked singly or independently, which is the only way Hicks would be flatly wrong in her 3 to 6 minute claim re: 2004. It apparently has nothing to do with the Nature question and response at all.
You didn't offer an exact quote of Les Roberts' response to Nature about the teams going it alone and it's irrelevant to the issue at hand. How did Nature's interviewer manage to misunderstand Roberts saying his interviewers went singly in 2006 when he meant they went singly in 2004, given that Nature asked if even two teams of 4 splitting in pairs could possibly do the job in the 2006 study? An exact quote of what Roberts was asked and responded would help settle this; a quote of an irrelevant reply to a different question won't.
The things that "gives away" that you were not offering an exact quote of Roberts to the journal Nature is the Media Lens article you cited stating Paul Reynolds of the BBC was forwarding reader questions [inclu. Emersberger's] to Les Roberts.
Also, I did see what Les Roberts said regarding the length of time it took in the 2004 study.
Here is the stated methodology for that study:
http://www.thelancet.com/journals/lancet/article/PIIS0140673604174412/fulltext [requires a free registration]
Among other things Robert here asserts study teams of three people [team leader and male and female interviewer], a similar methodology of obtaining consent and informing interviewees about the purpose of the survey as in the 2006 study, asking quite a few specific and involved questions that would require a fair amount of recall on the part of the interviewee, even in the event they had no deaths, and an attempt to confirms deaths via death certificate in 2 deaths per cluster.
Now taking Roberts' 3 hour 30 house cluster, this means a team of two did all this and polished off a cluster of 30 houses traveling door to door in 6 minutes per house including all incidentals relating to actually moving from house to house, being greeted, getting inside, asking questions, etc. and every so often waiting for a death cert. Roberts says in many days one team polished off 2 clusters, i.e. 3 minutes per house according to the conjunction of his claims.
You have yet to quote, and I've yet to see, anything else asserting the teams in 2004 worked singly. Why include a male and female interviewer in each team, presumably for the sake of putting the interviewee at ease regardless of gender, if they were going to split up? Moreover, why split them individually in 2004 and not in 2006?
For argument's sake, if they did work singly then, we get 6 minutes per household for those double cluster days in 2004 which falls significantly beneath your estimate of 10 minutes, and which is incredibly optimistic for what they claimed to do and ask. We get this from conjoining your analysis and Robert's in the Media Lens article.
Maybe they didn't perform a sensitive, informed consent style of survey on those double cluster days?
Here's a sample informed consent template for a no-risk [to the respondent] survey:
http://www.umass.edu/research/comply/surveystemplate.doc
I assume one for at-risk respondents would have to be longer and more detailed than this, and take longer than this to complete given the face to face nature [questions from interviewees being possible] of the Lancet interviews.
Maybe the teams did conduct an ethical survey in 2004 but didn't meet Roberts' purported timeframe or target number of houses, i.e. he doesn't understand his own teams' field ops or got bad data. That's about the best face one can put on it without assuming some dissembling by Roberts.
After reading the 2004 and 2006 methodology, the basics of informed consent and Hicks claims about the length of respondent's replies and sensitivity of the topic necessitating some circuitousness, how realistic a timeframe does the most generous interpretation of Roberts' claim at 6 minutes per household on double cluster days in 2004 sound? Not very.
Posted by: Kevin | March 6, 2007 11:51 PM