Science News on Lancet studies

Julie Rehmeyer has a decent article about the Lancet studies in Science News. Unlike Neil Munro and Megan McArdle, she doesn’t have an axe to grind. She talks to experts in the field like Jana Asher instead of non-experts like Michael Spagat:

The conflicting studies in Iraq show just how tricky it is to apply these methods in messy real-life situations. About the Lancet study, Asher says, “I don’t think there was anything obvious in what they did that someone can point to and say this method is flawed. But the WHO study used appropriate methodology too.”

The most suspect part of the Lancet study, Asher says, is that the researchers didn’t supervise the survey workers closely. On the other hand, the World Health Organization relied on government workers to administer the questionnaires. People can be intimidated by government workers and be less inclined to say much, a phenomenon that is particularly common in unstable countries. The only way to resolve the conflict, Asher says, is to do yet another study, with an even more careful design.

Comments

#1 dalazal
March 29, 2008

It’s always good to see at least someone going to talk to specialists, instead of people that have an ax to grind.

But the article still gets some things wrong:

“_This Lancet study estimated that more than 650,000 additional Iraqis died during the invasion than would have at pre-invasion death rates, a vastly higher estimate than any previous. But in January, a World Health Organization study placed the number at about 150,000._”

The NEJM/WHO study found 150,000 additional *violent* deaths, and didn’t directly estimate the overall excess mortality.

L2 found 600,000 additional *violent* deaths. 650,000 was the estimation of overall excess mortality.
#2 Kilo
March 30, 2008

“The only way to resolve the conflict, Asher says, is to do yet another study, with an even more careful design.”

Why bring logic into it at this late stage.
I can’t recall when this blog posted an article titled “Lancet post #135″ but it was a fkn long time ago.
#3 Sortition
March 30, 2008

> “[...] But the WHO study used appropriate methodology too.”

Asher gives the WHO study a free pass. Methodologically speaking, the that study is far inferior to the Lancet studies primarily since it relies heavily on extrapolation using IBC data and since it uses an arbitrary fudge factor to account for under-reporting.
#4 Jeff Harvey
March 31, 2008

When Jana Asher says, “The only way to resolve the conflict, Asher says, is to do yet another study, with an even more careful design” I can only shake my head in disbelief. Who will do the survey? The US and British governments, who are responsible for an illegal invasion that has turned Iraq into a country of wreck and ruin?

This is the bitter irony. Aggressing nations do not tally the numbers of their victims. Ian Gould summed it up in the thread below this: because the real death toll of civilians conflicts with the well-cultivated myth of US benevolence, western crimes are not a part of history because they are never allowed to become a part of history. They thus get sent straight down the memory hole. If an in-depth study was done, providing a very accurate estimate of civilian deaths in Iraq as a result of the invasion, then the myth of western basic benevolence would be obliterated. But as long as no official count is made, the aggressors can forever argue that because the total figure is unknown, that the higher estimates are unreliable and can be downplayed.

How many people died in the Chorillo barrio in Panama when it was shelled by US forces in 1989 by Bush I who invaded in order to capture a regional thug (Manuel Noriega) who had been on the CIA payroll? Does the US care? Estimates run as high as 3,000, but no official count was made. Down the memory hole. A few months ago the Panamanians mourned the anniversary of the bombing. How many western media outlets carried the story?

How many victims were there from the 4 year mass-bombing campaign by the US in Cambodia 1969-73? Hundreds of thousands? Tens? No one really knows. Down the memory hole. How many died in South Vietnam from US bombing? One million? More? Less? No one really knows. Down the memory hole. Korea 1951-53? Another million? More? Less? The Phillipines 1899-1902? Half a million? More? Less? Cuba 1915? Ten thousand? More? Less? Haiti 1918? Five thousand? More? less? Central America 1980-90? Three hundred thousand? More? Less? The list goes on and on and on. No official counts are made of direct or indirect US sponsored aggression or terror, so its all down the memory hole.
#5 David Kane
April 4, 2008

Best part:

These challenges have to be met with very carefully designed protocols. For example, the Lancet study of Iraq, with the shockingly high mortality rates, was initially criticized for not surveying people who lived in back alleys because the areas were too dangerous for surveyors. Les Roberts, who was at Johns Hopkins University in Baltimore at the time but is now at Columbia University, and his collaborators on the study argued that the critics had misunderstood their randomization technique.

Maybe! But, to this day, no one (including Tim) knows whether or not “back alleys” were included in the sample for L2. Were they? Help us out Tim! If you don’t know the sampling plan, it is tough to judge the validity of the survey.
#6 z
April 4, 2008

Yeah, but since we never know nothing for certain, you go with the best you got. and here, as in global warming, the rightwingers violate this common sense rule by refusing to accept an estimate, not in favor of a better estimate, but for wishful thinking that it ought to be something different.
#7 Kevin Donoghue
April 5, 2008

Since David Kane keeps asking the same question there’s no reason why he shouldn’t get the same answer.
#8 sod
April 5, 2008

Maybe! But, to this day, no one (including Tim) knows whether or not “back alleys” were included in the sample for L2. Were they?

simple challenge:

i will do my best to answer this question, as soon as David starts presenting some serious evidence, that people LIVING in roads crossing main roads have SIGNIFICANTLY higher chance to die in Iraq.

while looking into it, you could do some research on how you average poll here compensates for those who only use a mobile or who don t have a phone at all….

BRING IT ON!
#9 David Kane
April 5, 2008

Kevin,

The comment that you link to is a reasonable one but unresponsive to my point. I am not claiming (here) that L2 is wrong or Spagat is wonderful. I am just asking you (and Tim and anyone else) what your understanding is today about the sampling scheme. What do you honestly think happened?

To be specific, were households on back alleys included in the sample? The answer to this question, whatever it is, does not have a direct bearing on whether or not L2 is wrong/wonderful. You can make a reasonable case either way.

The issue is that even people has knowledgeable as you and Tim still do not know what the sampling plan was. You are still in the dark two years later. I think that this is emblematic of larger problems with the survey.

But, again, it is easy for you to refute this! Just answer the question and, if possible, provide a link to your source.
#10 Robert
April 5, 2008

David Kane wrote:

You are still in the dark two years later. I think that this is emblematic of larger problems with the survey.

I think it is emblematic of larger problems that two years into criticism of the Roberts article, David Kane still did not know how to calculate a crude mortality rate.
#11 Kevin Donoghue
April 5, 2008

David Kane: What do you honestly think happened?

I accept Gilbert Burnham’s account at MIT (which you found reasonably satisfactory at the time). Some households in back alleys were included in the sample. In my own neighbourhood, if you start a cluster on a side-street and visit 40 houses, you are going to find yourself in a back alley once in a while. That’s likely to be true in most urban areas.
#12 SG
April 6, 2008

david, get a map of any reasonable city and run a simulation on it yourself. If you can’t be bothered doing this, quit insinuating anything about the sampling scheme.
#13 David Kane
April 6, 2008

[Robert Chung] Make sure to collect all these great contributions to the debate and pass them along to your academic colleagues. I am sure that they will be impressed.

[SG] What do you want me to simulate? I don’t know the procedure that was used. And, unless you are keeping a secret, neither do you.

[Kevin Donoghue] I did find Gilbert Burnham’s account “reasonably satisfactory” at the time. But, one advantage of Spagat’s paper is that it highlights how suspect that account actually is. (Perhaps I should have realized that last year.) Consider this quote from that Burnham presentation.

A group of physicists in Oxford complained that our sampling method favored the principal streets where they interpreted that most of the killings may have occurred. In fact, we went out of our way to try to include all the streets in the sampling frame. And, of course, the other thing we found is most killings occurred away from home anyway. And so that probably didn’t add much to bias, if it had even existed.

So here we have Burnham claiming that “all” the streets in Iraq were included in the sample frame. We now know, from Spagat’s work, that there is no way that this can be true. In fact, other sections of that same talk make it fairly clear that this can’t be true.

Now, no one at the talk really focussed on this discrepancy, including me. Nor do I think that Burnham was lying. I just think that a) He has a sense of what they told the interviewers to do (some version of streets-off-of-main-streets), b) He has some sense that what they actually did may or may not be what they were told to do (goodness knows that they screwed up the governorates) and c) He wants to discredit the Main Street Bias crowd by claiming that “all” streets were in the sampling frame even though this is manifestly impossible.

Surely a reasonable person like you will agree that there is no way that all streets were included? How could they be?

Now, one can certainly argue that, even though not all streets were included, that the streets that were are a reasonably random sample of all streets. That might very well be true. Or it might not. But it is hard/impossible to have that conversation until we know what scheme was used.

And all this matters because Roberts is still spinning this
to authors like Julie Rehmeyer. She writes that Roberts claims that “the critics had misunderstood their randomization technique” without understanding that no one knows what the actual randomization techniques were!
#14 Kevin Donoghue
April 7, 2008

David,

You don’t seem to be able to write even a few sentences without distorting what Burnham asys. He says: “we went out of our way to try to include all the streets in the sampling frame.” (My emphasis.)

On the strength of this you say: “here we have Burnham claiming that ‘all’ the streets in Iraq were included in the sample frame.”

Surely you can see the difference between Burnham’s statement and your paraphrase? It’s not a big deal but your destructive intent is showing here.

Also, what’s so difficult about running a simulation? Have one of your colleagues show a group of student volunteers Burnham’s video and ask them to do what they understand the Iraqi interviewers did, selecting houses on a detailed map. They get no further guidance, they are just to do what they think an intelligent researcher would do in the light of Burnham’s stated intentions. Compare their samples with the selection which would result from your preferred sampling method and see if there is any systematic difference.

If you really want to do it right, use maps of Northern Ireland and see if you get an inflated estimate of the number of people who died in the Troubles – there is a database containing the names of all the victims and researchers can get their last addresses.
#15 sod
April 7, 2008

“we went out of our way to try to include all the streets in the sampling frame.”

thanks for pointing this out. the quote can be founnd on David’s site.

http://lancetiraq.blogspot.com/2007/05/more-quotes-from-burnham.html

but i am not suprised that David is trying to mislead people again.

he continues to da that with the next sentence:

So here we have Burnham claiming that “all” the streets in Iraq were included in the sample frame. We now know, from Spagat’s work, that there is no way that this can be true.

Spagat does NOT provide any evidence, contradicting what Burnham said.

he did neither do any field work, nor a detailed analysis.
instead he is making two wild assumptions:

1. only very big roads are main roads. (this allows him to mark a huge area “unpolled” in his now famous “map”)

2. people LIVING in houses closer to those “mainstreets” have a significantly higher chance of dying violently.

here is Spagat’s argument for point no. 2:

a. Crowded markets, cafes restaurants and other attractions will be on such streets.
b. Military patrols focus on such streets. In fact, many military vehicles can only go down the larger streets.
c. Abductions and mass shootings also tend to be on such streets. For example, Sunnis would not travel deep into Shiite territory, abduct some people and make a long drive to reach safe territory. Rather, they would make a quick foray in and out of enemy territory, perhaps just crossing over a main street that divides the two areas, and continuing only until they were just inside of a residential area.

you might not only notice the total absence of any notion why people LIVING there would make te likely victims but note as well that at least point c is totally absurd.
#16 Robert
April 8, 2008

David Kane whined:

Make sure to collect all these great contributions to the debate and pass them along to your academic colleagues. I am sure that they will be impressed.

I think amused is closer, though we’re mostly obscure academics in an arcane field so almost by definition we’ve demonstrated a low amusement threshold. OTOH, you’ve left quite an impression, judging by the way [this](http://scienceblogs.com/deltoid/2007/08/robert_chung_on_david_kane.php#comment-552074), [this](http://scienceblogs.com/deltoid/2007/08/robert_chung_on_david_kane.php#comment-552101
), and [this](http://scienceblogs.com/deltoid/2007/08/robert_chung_on_david_kane.php#comment-552521) have entered the departmental lexicon.
#17 David Kane
April 8, 2008

I (and others?) stand (somewhat) corrected. I have just listened again to that section of Burnham’s talk. Good stuff. In Burnham’s example, only very big roads are main roads, contra sod’s claim that this is a “wild assumption.” (Check the video.) But, and this is the fun part, he describes how they first pick one of those main roads and then make a list of all the side streets in the vicinity of that main road, including roads that are not directly connected to that side street.

Again, this is just one example that Burnham uses and it might not be that representative of what actually happened and it is hard to see the details on his slides, but it sure looks like streets at the next stage did not actually need to cross/contact that original main street. The exact language that Burnham uses is:

“all the residential streets that crossed it [the randomly selected main street] or were in that immediate area”

And that is what his graphic shows. Indeed, in his example the randomly selected side street does not across the randomly selected main street chosen at stage one.

So, I am guilty of thinking/assuming/remembering that the description that Burnham gave in the talk was consistent with the description given in the paper. Apologies!

Of course, the description here is even worse.

First, different streets have a very different probabilities of being selected depending on how many main streets they are in the “immediate area” of. You are much more likely to survey from the city center than the outskirts. Burnham’s graphic makes that clear, makes it obvious that some side streets are in the “immediate area” or two or more main streets and some are not.

Second, it seems impossible to understand how the surveyors could have done it without full scale maps of every side street since there is no obvious (possible) algorithm for deciding which side streets are in the “immediate area” of a given main street. (There has never been a claim that interviewers had or used detailed maps and, given all the fears about appearing military, I expect that they did not have such maps.) Burnham’s example shows some of these side streets were several blocks away from the main street. How could the interviewers figured out, just by driving around what streets to include?

At least the procedure stated in the paper was plausible. Once you pick a street there are only X number of streets that cross it.

But as soon as you include streets in the “immediate area” — and you haven’t already mapped it all out ahead of time — you have no idea which streets belong in the immediate area of the main street you have chosen and which belong in the immediate area of a different main street that you did not choose.

As always, I do not think that Burnham is a bad guy or that he is intentionally misleading anyone. I think that, when they wrote the paper, they gave their honest opinion of what they thought the interviewers did (or even what they told the interviewers to do). Then, when they got all the complaints about sampling, they revised the story. Nothing wrong with that!

But the real way to settle the dispute is to share with the world the actual sampling plan that they wrote and gave to the interviewers. That piece of paper exists somewhere.
#18 Freud
April 8, 2008

The psychology here is remarkable. Kane is now trying to revise himself. He now claims, and expects us to believe that he believes, that Burnham is “not intentionally misleading anyone.” But, of course, we all remember how all of this started: with Kane saying the Lancet study was a scientific “fraud” – thus getting his own self booted from the Harvard stats site. Is Kane now denying Kane? Has he taken his own concern trolling so seriously that he has double-backed on himself? Who knows, maybe now he has learned how to calculate a crude mortality rate…no, some things are just impossible…
#19 Kevin Donoghue
April 9, 2008

That piece of paper exists somewhere.

WTF??
#20 David Kane
April 9, 2008

1) I think that the underlying data that L2 relies on is fraudulent. That is, I think that the interviewers made (some) stuff up. Burnham did not do the interviews. Indeed, he never went to Iraq. All he got was a pile of interview sheets. I think that he has accurately reported, with some exceptions, what is in those sheets.

2) [Kevin]: My point is that the US based authors including Burnham created paperwork before the interviewers set out in Iraq. They had a survey plan which they wrote down and, among others things, showed to the IRB. They needed to get the plan approved. That “piece of paper” would help clarify this dispute since it would outline, in detail, what the interviewers were supposed to do.
#21 sod
April 9, 2008

I (and others?) stand (somewhat) corrected.

what??? you (ALONE) and AGAIN have been totally WRONG.

no surprise..

In Burnham’s example, only very big roads are main roads, contra sod’s claim that this is a “wild assumption.”

i can t watch the video for unknown reasons. but i d be very much surprised if i d find him massively restricting street size.

a significant part of Spagat s misleading attempt is his aerial picture of Baghdad, that seems to leave people with the impression, that there are only about 10 “main streets” in this city..

As always, I do not think that Burnham is a bad guy or that he is intentionally misleading anyone.

please spare us these disclaimers. you have flooded the internet with claims of fraud.
your claim above is just one example of a litany of totally false accusations, supported by ZERO evidence.
#22 Robert
April 9, 2008

David Kane wrote:

I think that the underlying data that L2 relies on is fraudulent. [...]That “piece of paper” would help clarify this dispute since it would outline, in detail, what the interviewers were supposed to do.

Hmmm. The first time David Kane floated the fraud balloon, he said it was because the response rates were higher than any other national survey anywhere on any topic, ever. The second time David Kane floated the fraud balloon, he said it was because Roberts had manipulated the calculation of excess mortality and no one, anywhere, could replicate it. The third time David Kane floated the fraud balloon, he said it was because Roberts had intentionally excluded Falluja in order to suppress a CI for the estimate of excess mortality that straddled zero. The fourth time David Kane floated the fraud balloon, he said it was because Burnham claimed that all streets were included in the sampling frame.

You know, perhaps it’s appropriate that the name “David Kane” and the term “fraud” be linked. “David Kane” and “crude mortality rate” [already appear to be](http://www.google.com/search?q=%22david+kane%22+%22crude+mortality+rate%22).
#23 sod
April 9, 2008

1) I think that the underlying data that L2 relies on is fraudulent. That is, I think that the interviewers made (some) stuff up.

why accuse Burnham of making false statement then?

why bother with the main street bias at all?

and why don t you simply present sonme evidence of this “fraud”?!?

sorry David, but among the few persons making stuff up here, you are quite a special one!

===

the other rather “special” person is Spagat. while follwing some links about the “main street bias”, i hit this presentation given by him in december 2007:

http://personal.rhul.ac.uk/uhte/014/Households%20in%20Conflict%202007.pdf

my eyes got caught by page 7, showing a Baghdag map:

This map seems to suggest that large attacks
in Baghdad could be biased toward residential cross-streets to main streets … Attacks since May 2003 in which more than 10 people were
killed.

now obviously a rather big color point on an arial map of a city will end up somewhere around a “main street”.
notice how he is playing with the word “residential”, without again knowing ANYTHING about te palces the attacks occur in. (am i the only one who got familiar to the term “MARKET BOMBING” by watching news in iraq?!?)

this is a pretty lame attempt to establish a link between the location of the bombings and people living CLOSE to the place where it occured.

but this sentence is even more absurd:

Note that incidents of this size almost certainly cover over half of all deaths.

note that this sentence obviously is total NONSENSE, as a simple look at any list of violent deaths in iraq will show you:

http://www.iraqbodycount.org/database/incidents/page1

http://icasualties.org/oif/(qb52ieexlryn5f45zlljy0e4)/IraqiDeaths.aspx

actually you willstruggle hard to find a single day, on which events with 10+ casualties cover 50% percents of the daily deathtoll.

Spagat could have known this, by simply taking a look at his map: if the Spagat map did really represent over 50% of the deaths in Baghdad since 2003, the place would be a paradise!!!
#24 David Kane
April 10, 2008

sod asks “why accuse Burnham of making false statement then?”

Well, because he made a false statement and has not corrected it. (Note that I responded to your (reasonable) comment in that thread.) Again, this is not the world’s biggest sin and I continue to think (judging from the e-mail back-and-forth) that Roberts is the bad actor here, but the fact remains that Burnham continues to allow his name to be used with an false attack. He either does not know what the data is or, more likely, is happy to allow Les Roberts to say whatever it takes to discredit Munro/Spagat/me.

Oh wait! I was just updating the link to the Burnham/Roberts letter to the National Journal (it seems like things got moved around the Hopkins server) and, what do you know? They changed the text in the letter without telling anyone. The paragraph now reads:

The statement on missing certificates is wrong. There were 83 deaths (13%) in which the interviewers neglected to note their presence and these deaths were distributed across 20 clusters.

Classy!
#25 David Kane
April 16, 2008

Roberts and Burnham are pulling the kind of stunts that Tim would normally have great fun unmasking.
#26 Jana Asher
July 16, 2008

I just found this thread – interesting. I do feel the need to make one comment in response to Sortition’s post:

“Asher gives the WHO study a free pass. Methodologically speaking, the that study is far inferior to the Lancet studies primarily since it relies heavily on extrapolation using IBC data and since it uses an arbitrary fudge factor to account for under-reporting.”

Sortition is correct in his/her(?) statements about how the WHO study created its final numbers, and (in my opinion) is correct to express concern. My original statement was not intended as a free pass, but rather a “broad stroke” picture – meaning, taken as a whole, the WHO study made reasonable assumptions/methodological choices. If I had personally been doing the work, I might not have made all of the same choices regarding the extrapolation and the under-reporting “fudge factor.” But making the under-reporting adjustment, for example, shows at least a good faith attempt to account for potential sources of bias.

The problem with having to sum up one’s opinion in a sentence is that there will invariably be details that are left out. So thank you to Sortition for bringing up these points.

David Kane organized a session, related to the death count estimates for Iraq, for the Joint Statistical Meetings in Denver, Colorado this year – It will be on Wednesday, August 6th at 10:30am. I’m supposed to be chairing the session. It should be very interesting – somewhat like this thread. I hope some of you will be there.