# Journal of Peace Research publishes badly flawed paper, part 2

This post is some more notes on a reply to the badly flawed “Main Street Bias” paper.

The authors claim that it is plausible that the Lancet paper’s sampling scheme could have missed 91% of the houses in Iraq. (That is, their parameter n, the number of households in the unsampled area divided by the the number in the sampled area could plausibly be 10 or more.) The only support they offer for this is a reference to this analysis of Iraqi maps.

To the right is a detail from their map. The red lines are main streets and the yellow are secondary streets. They assert that the blue areas are not samplable using the Lancet scheme, and yes, the blue area covers 90%+ of households. But there are two things wrong with their map.

Look at this larger scale version of their map:

First have a closer look at their yellow road that they say should not be counted as a main street and compare it with the one they count as a main street. (Scroll around on Google Maps if you want to see more of each street.) It’s just as wide and has a similar amount of traffic and a similar number of large buildings. Clearly it should have been classified as a main street and it is not even slightly plausible that someone who was trying to get every house in the sample frame would leave it out.

Second, the random house on the secondary street is only the start point for the cluster, which also includes 39 houses neighbouring the start point. This means that you can sample houses on tertiary streets that are a few houses from the secondary street. I added two obvious main streets to the ones they chose (the one across the bottom of the map above and one that it is off the map above), and redrew the blue areas to taken this factor into account.

What’s that? You don’t see any blue areas in my map? That’s because there aren’t any. Make just these two corrections to their map and the unsampled area is 0. In their model, that means n=0 and there is no main street bias.

1. #1 sod
February 19, 2009

nice work Tim.

they also prefer to chose Baghdad when talking about “mainstreet bias”. choose a random village and use the lancet approach, and you will nearly always find nearly everything covered…

2. #2 Eli Rabett
February 19, 2009

Another thing is that you don’t tend to find markets and such on wide main streets cause you can’t get goods in and out and customers cannot cross easily.

3. #3 Robert Shone
February 19, 2009

Tim Lambert writes:

They assert that the blue areas are not samplable using the Lancet scheme…

No, they don’t assert this. Nobody knows what’s samplable using the Lancet scheme, because nobody knows how the Lancet scheme worked in reality (since the details haven’t been released).

The MSB authors make it clear that Burnham et al’s definition of main street “is too vague to answer these questions definitively”. They continue: “For starters, we cannot be sure of what is a major commercial street or avenue. But it is possible to explore some scenarios and we do this below. We encourage everyone to do their own explorations based using (the miraculous) Google Earth.”

If that’s not clear enough for Tim, they also write the following [my emphasis]:

We highlight the roads that we consider major commercial streets or avenues in red. We highlight the crossroads to the major roads in yellow. The blue shaded area contains residential building that could not be sampled under the authors’ stated scheme and accepting our classification of major commercial streets or avenues. If these were indeed the major commercial streets chosen in The Lancet paper, then only a tiny sliver of Kirkuk could have been sampled. A more liberal rendering of major commercial streets or avenues would enable deeper penetration.

One can see more clearly from the full map (rather than from the particular corner of it that Tim has chosen to focus on) why the road marked yellow might be considered a cross street – it cuts straight through the residential area, in contrast to the red-marked roads.

I’ve no idea whether n=10 in the real study. It seems infinitely more plausible than n=0.

4. #4 sod
February 19, 2009

why the road marked yellow might be considered a cross street – it cuts straight through the residential area, in contrast to the red-marked roads.

so they assume that only roads not leading through residential areas are “mainstreets”, and then conclude some bias of residents not being sampled?

A more liberal rendering of major commercial streets or avenues would enable deeper penetration.

for some weird reason, they didn t provide too much data, on such a “deeper penetration”.

and instead went for n=10 in their number works. just by chance?

5. #5 Robert Shone
February 19, 2009

Incidentally, if you take Tim Lambert’s “yellow road” scenario for producing n=0, and apply it to the whole of Iraq, you get interesting (but not remotely credible) implications for how the Lancet sampling scheme “must” have worked.

6. #6 sod
February 19, 2009

i am seriously puzzled by the Tikrit map on the Spagat page.
How did they chose those red “mainstreets”

7. #7 Eli Rabett
February 19, 2009

There is relevant information from a BBC intervies

c) most urban clusters spanned 2-3 blocks as we moved in a chain from house to house so that the initial selected street usually did not provide the majority of the 40 households in a cluster and d) people being shot was by far the main mechanism of death, and we believe this usually happened away from home.

especially d

8. #8 Tim Lambert
February 19, 2009

I couldn’t make any sense of their Tikrit map. These are the only two maps they offer in support of their n=10 theory.

9. #9 David Kane
February 19, 2009

1) Thanks to Tim for taking the time to do this analysis and start a new thread. This is the way open-minded scientists search for the truth.

2) Tim: Why don’t you ask Les Roberts whether your “two corrections” are plausible? After all, surely Roberts knows what the survey teams considered to be “main streets” . . . right?

3) I believe that I have listened to more publicly available presentation by the Lancet authors than almost anyone. I have never heard them give a description that is consistent with a methodology that would lead to a map like the one that Tim describes above. They weren’t using Google maps after all! That doesn’t mean that Tim’s description is false. We just have no idea what they actually did. (Pointers to the contrary are welcome.)

4) I hope this mean that Tim has given up on his “proof by contradiction” that the formula is wrong from here. If so, it would be nice of Tim to confirm that he no longer maintains that there is a mistake in the maths.

10. #10 Tim Lambert
February 19, 2009

DK:

3) I disagree. I just followed the methodology they described in their paper. Of course they didn’t use Google maps, but you can use Google maps to determine what someone at street level would do.

4) No. Their formula is wrong as I demonstrated. I never said that there was a mistake in their maths. Their formula is wrong because their model is wrong. Stay tuned for part 3.

11. #11 David Kane
February 19, 2009

Here is an overview of what we know and what we don’t know about the sampling for L2. Summary:

L2 authors have given different (and conflicting) accounts of exactly what the interviewers did and/or were supposed to do. They have made no final statement about what sampling plan was followed. Anyone who claims to “know” what the sampling plan in L2 was is lying.

Tim is correct that his description of L2 sampling could be correct. Alas, he has no real evidence for it since there is no clear, final statement by the authors themselves. How to make progress? Tim should ask Les Roberts to clarify, once and for all, the exact methodology followed by the Iraqi interviewers. For example, did they have maps?

12. #12 David Kane
February 19, 2009

Tim:

We cross-posted. You write: “I just followed the methodology they described in their paper.”

But we already know that the methodology described in the paper is wrong. Burnham/Roberts have told us that! Do you disagree?

Can you provide a citation to your claim that Roberts “already answered this?” I am not doubting that you have such a citation, but collecting the various contradictory things that Roberts has said about the sampling plan is a hobby of mine.

I look forward to part 3.

13. #13 Tim Lambert
February 20, 2009

>”As far as selection of the start houses, in areas where there were residential streets that did not cross the main avenues in the area selected, these were included in the random street selection process, in an effort to reduce the selection bias that more busy streets would have.”

Clearly this makes no difference to my result above. Adding streets to the set of main streets will only improve the coverage.

And note that I didn’t go out a search for a map where the scheme described in the paper worked, this was the one they offered to show that n=10 was plausible. It isn’t.

14. #14 Jody Aberdein
February 20, 2009

Well thus far with my map of london centred on where I live, a bit of string, a ruler and a random number generator I’m getting pretty wide dispersal with only officially designated A roads as main streets even. Will hopefully get this process automated so as to do some proper analysis.

15. #15 David Kane
February 20, 2009

Tim:

In the link you provide (thanks), Burnham/Roberts claim that “Sampling for our study was designed to give all households an equal chance of being included.” This is a contradiction of the sampling scheme you describe above. For example, consider all the houses that are near the intersections of two (or more) cross streets. Those houses are much more likely to be included in the sample since picking either cross street still gives them a chance. Other houses are near (meaning within a possible 40 house sample) only one cross street. Your picture illustrates that perfectly.

So, either Burnham/Roberts were wrong in their letter to Science or you are wrong to claim that your picture illustrates the sampling plan they undertook. Which is it?

16. #16 Robert Shone
February 20, 2009

Tim Lambert should read the following comment from Les Roberts (in a 2007 radio interview, via David Kane’s blog) as it makes Tim’s latest “deep, thorough penetration” scenario questionable:

LR: So they were just not under any circumstances going to use the GPS system, so they did more or less everything the same except they… when they got to a neighborhood and sort of drove around and saw the outline of the neighborhood they sort of counted – they picked off the main streets and then picked one and then picked the side streets of them, or picking one at random, counted all the houses on that street and picked one at random. So they had a slightly different randomization technique.

And then there’s this comment from Burnham (MIT, 2007):

Burnham: [they made a list of] “all the residential streets that either crossed it or were in that immediate area.”

Huh?

17. #17 Eli Rabett
February 20, 2009

Jody is doing what Spagat and Co should have done.

18. #18 David Kane
February 20, 2009

Eli: Instead of telling Spagat and Co what they should have done, why don’t you explain to us what Burnham and Co actually did? You have followed this debate closely. Do you think that this is a true statement by Burnham/Lambert?

Sampling for our study was designed to give all households an equal chance of being included.

If so, then isn’t Tim’s example wrong?

19. #19 sod
February 20, 2009

For example, consider all the houses that are near the intersections of two (or more) cross streets. Those houses are much more likely to be included in the sample since picking either cross street still gives them a chance. Other houses are near (meaning within a possible 40 house sample) only one cross street. Your picture illustrates that perfectly.

David, this is getting stupid (again). you have exactly the same problems if you randomize GPS coordinates (single house will get higher chance of getting picked) or telephone interviews (people with multiple numbers or people only using mobiles or people who travel a lot..).

all of this is “as best as circumstances allow”.

i doubt that anyone including Burnham takes

“Sampling for our study was designed to give all households an equal chance of being included. “

to mean EXACTLY the same chance. you do understand, that populations move a lot in Iraq?

20. #20 Tim Lambert
February 20, 2009

David Kane:
>If so, then isn’t Tim’s example wrong?

No.

I take it that no-one disputes that if you follow the sampling scheme described in the Lancet paper and apply it to the map that the MSB authors chose, you get almost complete coverage, n at most 0.1 rather than n=10 as they claim. Don’t you think that this is a serious flaw in their paper? Is that why you are trying to change the subject?

21. #21 Robert Shone
February 20, 2009

Tim Lambert writes:

I take it that no-one disputes that if you follow the sampling scheme described in the Lancet paper and apply it to the map that the MSB authors chose, you get almost complete coverage.

Tim is confusing the “Lancet sampling scheme” with the “Lambert sampling scheme”.

We actually know more about the latter than the former, which is a pretty sad reflection on the Lancet authors.

22. #22 Robert Shone
February 20, 2009

Or, to spell it out for the slow-learners, Lambert’s analysis is disputed by those who realise that since Lambert doesn’t know what L2 took to be a main street, his own guess wrt “main” street vs cross street is irrelevant.

His choice (suggesting that nearly every street in Iraq was in the samplable space) is at odds with arguments made by the L2 researchers about limitations due to safety and shortage of time, etc.

The Lancet authors claim “all households” had equal chance of inclusion. The burden of evidence is clearly upon them.

23. #23 Eli Rabett
February 20, 2009

Shone and Kane are claiming perfect ignorance, which in their cases is an acceptable starting point. However, down here on earth it is clear that you should be able to look at any city, impose a condition for what a main street is and look at what the results are as that condition is varied. That is what Jody is doing. It is also clear that Spagat, et al (Hi there et, or is it al) imposed an unreasonable condition a la McIntrye and McKitrick, e.g. they picked a set of “main streets” that yielded the effect they wanted.

Now this may not be an easy problem. For example, when Eli looked at the maps, he saw that one reasonable way to pick main streets was to see where the mosques are (domed structures), the markets (lots of sheds and people) and a few other tells. In the main these were NOT the streets picked out by Spagat, Johnson, et al.

So, let us ask Spagat, Johnson and co. what condition THEY used for picking main streets and see if it is reasonable.

Further, a point which appears to be missed, is that most of the deaths by violence were from gunshot wounds, something not as likely to be concentrated on main streets as bombings might be (depending on your definition of a main street).

24. #24 Robert Shone
February 20, 2009

Eli Rabett writes:

However, down here on earth it is clear that you should be able to look at any city, impose a condition for what a main street is and look at what the results are as that condition is varied.

Have you asked the Lancet authors which “conditions” they “imposed” to establish whether a street is a “main” street? I thought not. Why don’t you try asking them, and then share their response with us.

25. #25 Kevin Donoghue
February 20, 2009

Suppose, for the sake of argument, that Gilbert Burnham is such a shameless fraudster that he merits a place on Tim Radfordâ€™s list. Would that excuse what Johnson et al. did with that map? In my view it would not. Is anybody prepared to defend them? So far in this thread nobody has even tried. All weâ€™ve seen from Robert Shone and David Kane is attempts to change the subject.

26. #26 Lee
February 20, 2009

Shone, Kane:

Why on earth are you two not responding on target to THIS criticism of THIS paper, which is what THIS thread is about?

Spagat et al claim that n=10 is a reasonable estimate. Use of a value near that – at least much greater than 1 – is necessary to their conclusion. They support their estimate of n=10 by the maps to which Tim links. One of those maps is incomprehensible.

This map, the one Tim analyzes, shows that an estimate of n=10 can only be arrived at by making facially absurd exclusions of major streets from the category of ‘main streets,’ which in turn excludes obviously secondary streets from the category of “secondary streets.”

This is a criticism of Spagat’s argument, which is based on an absurd classification of streets. Rather than divert, would the two of you please deal directly with that point, and its consequences to Spagat’s argument?

Or you can continue being irrelevant, if you prefer.

27. #27 Eli Rabett
February 20, 2009

Robert Shone doesn’t get it. By examining the result of a sequence of choices for main streets you can get a very good idea of how resilient the result is to the specific method of choice without asking anyone what their choice was.

Frankly the question that I have about main street choice is based on experience. Many places have small commercial zones on long streets/roads. This is especially true about small towns and from the maps also appears to be true about Iraqi cities. If this is so, and the length of the street is much longer than the commercial zone, Spagat, Johnson and Cos argument about main streets vanishes.

28. #28 Robert Shone
February 20, 2009

Lee writes:

Shone, Kane:

Why on earth are you two not responding on target to THIS criticism of THIS paper, which is what THIS thread is about?

Try reading my comments #3, #5, #16, etc – they address Tim’s points directly (in fact they address little else). There’s nothing in Tim’s new post that’s not already covered in my blog entry:

http://dissident93.wordpress.com/2009/02/15/pope-of-debunkers/

29. #29 Jody Aberdein
February 20, 2009

Slightly related, gives you an idea at least perhaps of the challenge in surveying this kind of thing. Unfortunately behind a paywall:

30. #30 Bruce Sharp
February 20, 2009

More Lancet Stuff

Robert, I’ll admit that I fall into your category of “slow-learners.” I have only a high school education, I have no particular math skills, no background in statistics, and no special knowledge of Iraq. I am, however, trying to keep an open mind, and I’m trying to grasp the whether or not the main street bias argument makes sense.

You’ve repeatedly said that Tim doesn’t know what the Lancet team actually did. How is Tim’s guess about what the L2 authors did any less valid than the Johnson et al’s guess? I think Tim makes a convincing case that the methods Johnson et al are using to determine the extent of the coverage are not plausible.

Do you have a comment on Sod’s observation that the criticisms here have focused almost exclusively on Baghdad? And how would main street bias affect the results in smaller towns? Was this what you were referring to in comment #5? I’m sorry if this is tedious for you, but I can’t understand what you are saying, and I think I’m probably not alone. What do you think Tim’s “yellow road” scenario implies when applied to the whole of Iraq, and what is it about these implications that is implausible?

Earlier, in the previous JPR thread, I asked you to clarify what information the L2 authors could have provided that would allow Johnson et al to come up with a reasonable value for q. (You conceded that the parameter value Johnson et al chose might not be correct.) You didn’t reply, but Ozzy/Ron/LancetDebunker/Tell suggested that BBC maps of violent incidents could be used to determine the relative frequency inside and outside of the survey space. Kevin, Sod and I pointed out that this doesn’t help at all: Sod noted that the maps show where the killing happened, and not where the people involved lived, and Kevin and I pointed out that virtually everyone agrees that the media is only reporting a fraction of the deaths, and that the pattern and scope of unreported deaths is exactly what we need to determine.

You say in #22 that the burden of evidence is on the Lancet authors to show that all households had an equal chance of inclusion. Why do you believe that they didn’t have a (practically) equal chance of inclusion? (As Sod noted in responce to David’s comment, yes, there are some minor differences of probability… but that would be true of ANY sampling scheme.) Do you agree that if the Lancet team used “the Lambert sampling scheme,” they would have had a good chance of including nearly any household? As far as I can tell, your only reason for arguing that the burden of proof rests with the Lancet authors is that you have a prior assumption that they were either dishonest or incompetent. Maybe they are, but it seems to me that the burden of proof rests with you, particularly when Tim’s example shows that, depending on whether or not one sensibly defines a “main street,” the scheme is not necessarily subject to “main street bias.”

You’ve admitted that you don’t know what “q” should be, and you don’t know what “n” is. As Kevin noted in the last thread: if we can plug whatever parameters we want into the model, there doesn’t seem to be a whole lot of point to this exercise.

Regards,
Bruce

31. #31 sod
February 20, 2009

Bruce did sum up those points much better than i ever could. thanks!

32. #32 David Kane
February 20, 2009

Bruce writes:

You’ve repeatedly said that Tim doesn’t know what the Lancet team actually did. How is Tim’s guess about what the L2 authors did any less valid than the Johnson et al’s guess?

Tim’s “guess” suffers from two problems.

First, he refuses to admit that it is a guess. He acts like he knows (and everyone else should know) exactly what the Lancet sampling plan was. But he doesn’t know! (See here for extensive documentation of that claim.)

Second, he refuses to help us figure out what happened. Tim could easily e-mail Les Roberts and seek clarification. (Roberts has replied to Tim’s e-mails in the past.)

So, the first step in making progress is for Tim to admit that he is guessing and then to work on turning his guesses into facts. Why won’t he?

But, to answer your question, I think that Tim’s guesses about what the Lancet team did are not unreasonable. I think that Johnson et al’s guesses are not unreasonable. Which ones are better? Tough to say! And, to be honest, not that interesting (to me).

I think Tim makes a convincing case that the methods Johnson et al are using to determine the extent of the coverage are not plausible.

Really? What evidence does Tim cite for this? Start with a specific example. How many main streets are there in a city like the example that Tim creates? One? Five? 50? 500? The number of main streets is, obviously, a crucial input in deciding what the coverage is and, therefore, what n is. But Tim, you, I and everyone else has no idea how many main streets there are in this cluster or in any cluster because the authors refuse to tell us (or anyone else). From my link above:

Seppo Laaksonen, a professor of survey methodology in Helsinki, requested and was denied any information on main streets, even the average number of main streets per cluster (Laaksonen, 2008).

Again, it could be that Tim’s guess about n is more accurate than Johnson et al’s guess. Or maybe your guess is better still! But no one in the debate has any good evidence for why there guess is good because the Lancet authors refuse to provide any useful information for estimating n (or any other parameter).

33. #33 Lee
February 20, 2009

Shone, having read this and the previous thread, I find I must bite my tongue, err on the side of politeness from what I think deserves to be said, and simply say “bullpucky.”

In 3, you dont respond AT ALL to Lamberts demonstration that the Spagat street selection procedure is absurd. You simply quote the paper’s authors to say that other selection procedures might give a different answer. Well, duh! That isn’t a response, it is a deflection.

Your 5 is content free, as far as I can see.

In 16, you again fail to respond to any point Lambert is making, instead quoting the L2 authors on selection procedure. How do those quotes address Lambert’s point that THIS PAPER’s authors use an absurd criteria for distinguishing main streets form secondary streets, and that if one uses more reasonable criteria, their argument fails?

34. #34 Kevin Donoghue
February 20, 2009

David Kane: Seppo Laaksonen, a professor of survey methodology in Helsinki, requested and was denied any information on main streets, even the average number of main streets per cluster.

The way I read it, Laakonsen wasnâ€™t â€śdeniedâ€ť information :

Burnham told Science, however, that the Johns Hopkins team does not have such detailed information. “Our goal was to reduce any type of risk to the community and the participants,” says Burnham. “While we have much of the raw data, we requested that anything designating the interviewers or the location of the neighborhoods visited not be sent to us.” Laaksonen responds that he would not have published “any figures for the country” if he didn’t have direct access to such raw information from surveyors.

A man canâ€™t be said to deny things to people if he doesnâ€™t have them to give. Do you have any reason to believe Burnham has the information Laakonsen requires?

35. #35 Lee
February 20, 2009

Kane, Roberts has said that their selection procedure was designed to attempt to give every house a chance of being selected.

Unless you are going in with the assumption that they are lying and committing fraud – and you clearly are, Kane, and it is IMO despicable behavior – then that adequately answers the question about the numbers of main streets. Their procedure is designed to attempt to give every house an equal chance of selection, and that means that the map Lambert is criticizing in this thread very obviously and clearly does not match the Roberts sampling scheme.

36. #36 Lee
February 20, 2009

Donoghue at 34:

I suspect you’ve just hit on the motivation for much of this.

Roberts et al were designing a survey under very edgy ethical conditions. There was a credible very-non-zero risk of death to the surveyors, and to the people being surveyed for answering the questions. He was and is ethically constrained to to do everything possible to minimize the risk to the people being surveyed, and he designed the procedure with that in mind.

Information that was even borderline related to ways to identify surveyed neighborhoods or people was simply not transmitted to him, in order to maintain the confidentiality of people who were potentially at risk of their lives for answering the survey.

Now these guys come along and invent scenarios which allow them to demand that Roberts release information that he by design did not retain, for ethical reasons – and then use his ethically-constrained survey design and retention policy to lambaste his ethics, because he is not releasing the information that he for ethical reasons CAN NOT release and did not retain.

I suspect that this is intentional on the part of Robert’s critics. I know that it is despicable behavior.

37. #37 Robert Shone
February 20, 2009

Bruce Sharp writes:

You say in #22 that the burden of evidence is on the Lancet authors to show that all households had an equal chance of inclusion. Why do you believe that they didn’t have a (practically) equal chance of inclusion?

I may be wrong, but your questions suggest to me that you haven’t read the MSB paper or its associated explanatory web pages. It’s pretty clear, for example, on the above point. The sampling scheme as published in the Lancet journal wouldn’t give all households an equal chance of being included (not even a “roughly” equal chance). Even Burnham/Roberts seemed to acknowledge this – eg they stated that they omitted from the published account the procedures which would enable them to “reduce the selection bias that more busy streets would have” (see comment #13 for link).

That comment from Burnham/Roberts makes no sense if they believed that the published sampling procedures by themselves “reduced” the bias from “busy streets”.

They’ve never released the details of these necessary “additional” procedures. They simply made assertions that they somehow reduced this bias to zero (or near zero). Hence the burden of evidence.

Forgive me if I don’t answer all your other questions. I’ve made the point repeatedly that people are free to suggest their own parameter values based on their own assumptions, and that until we hear from the Lancet authors these are speculative as far as the Lancet study is concerned. Based on what I’ve read about the study (eg see my comment #16) I do think that the MSB example parameter values are plausible – certainly far more plausible than Tim’s n=0 suggestion. That’s just my opinion based on the limited information available. It may turn out to be wrong. The MSB authors suggested parameter values which they found plausible based on the limited information available. You may disagree with them, but they’ve committed no sin by stating what they find plausible. And if you don’t like their suggested values, they’ve provided an exploration of the parameter space.

The bottom line is that it’s down to the Lancet authors to support their own assertions wrt how they “reduced” the bias to zero (or near zero).

The tail end of these threads tend to get repetitive (and abusive and pointless). I’m not going to repeat myself further – this is my final post in this thread.

A summary of my take on Lambert’s feeble attempt to debunk MSB is available at my blog:

http://dissident93.wordpress.com/2009/02/15/pope-of-debunkers/

38. #38 Raymond
February 20, 2009

Not to answer for Shone, who’s done a good job on his own, but Bruce Sharp writes:

How is Tim’s guess about what the L2 authors did any less valid than the Johnson et al’s guess?

It’s not really. They are just different guesses. Johnson et al’s presentation of this is more valid than Tim’s though because they say they do not know which streets or type of streets the L2 authors (field team actually, not the authors) called main streets in practice. Tim claims to be “correcting” their guess by replacing it with his guess. So he is making invalid claims about his guess while Johnson is not.

I think Tim makes a convincing case that the methods Johnson et al are using to determine the extent of the coverage are not plausible.

I don’t. The issue here is basically that the wider the definition of main street in practice, the more coverage and the less bias. The narrower the definition, the less coverage and the more bias. Tim guesses a wider definition was used than the guess in Johnson’s example. Is this more plausible? I don’t think so. To the extent we want to debate these guesses I think Tim’s guesses are less plausible in practice than Johnson’s for a number of reasons.

There doesn’t seem to be a formal definition of what a main street was for L2. So this definition would have to be a post hoc one, simply the outcome of whatever the field team did. At least three factors work against the wide definition hypothesis: the extreme time constraint on the field teams, the fact that they had to discover the streets in question in each neighborhood, and the security issue.

The time burden here is large as they already have to do 40 interviews in a day. In the case of this example, the teams would have had to drive to Kirkuk (from where?), then select the area to work with (how big is the area?), then travel around that area trying to find every “main street” in the area to put in their list. If a very wide definition of main street is used, their job of discovering and enumerating the main streets becomes harder, more time consuming and more dangerous. If a narrow definition is used the job is easier and takes less time, and they can get right on to cramming those 40 interviews into whatever is left of the day. Then there is security. The wider the definition, the more time they have to spend driving or walking around these strange neighborhoods looking for all the main streets while exposed to whatever dangers there might be.

There appears to have been little in the way of any specific criteria or conditions about what had to be called a main street. This was up to them to decide. Did they choose to make their job as difficult and time consuming and dangerous as possible, or did they choose to make it less? Which is more plausible?

Let’s bend over backwards for the Lancet study as Tim is doing and assume the field teams chose to make their job as hard as possible and used a wide definition of main street. They would then have to travel around the selected area (again how big is this area?) looking for all the main streets to list them. In the best case scenario, they find them all, but that is only the best case. In every other case they don’t discover all these widely defined main streets to begin with. Or, they get a couple of the most obvious ones down on the list and then stop traveling around looking for more main streets that they don’t even really know are out there to find. The same problems would apply to enumerating the cross streets.

More plausible, I think, is that in practice the definition would be narrow. They’d grab the most obvious main streets and get to work.

Do you have a comment on Sod’s observation that the criticisms here have focused almost exclusively on Baghdad?

Yes. The observation is wrong. The map in question is Kirkuk, for example. There are three maps shown on the MSB page linked above and none are for Baghdad.

Why do you believe that they didn’t have a (practically) equal chance of inclusion?

Because the sampling methodology does not establish an equal chance, practically or otherwise. In the best possible scenario it would give a very highly unequal chance of inclusion for each house, but some chance for all. One of Tim’s claims is that his guess about the yellow lines would get every house into the sample frame for the piece of the map he chose because of possible spillover onto tertiary streets when progressing through the 40 houses. But even if this were to get every house some chance of selection it would be nothing like an equal chance of selection.

This issue of progressing through the 40 houses also raises another can of worms. It is again a problem of vague sampling protocol and lots of discretion left to the team in the field to do whatever they want. Tim uses the phrase “neighbouring the start house” to describe how the teams progress from the start house to the rest. The Lancet authors have used the words “adjacent” or “nearest”. All are vague and allow discretion (and therefore bias) to enter in. For example, the team does it’s start house, there’s a house on either side that are both about as “near” to the first, they are both “neighboring” or “adjacent”. The team looks to one side and see a row of bombed or battered houses. They look to the other side and don’t see anything like this. Which way do they go? They could go either way and still be following the (vague) protocol. This allows the team to both follow their instructions and (consciously or unconsciously) seek out high mortality houses.

The way to progress from house to house is wide open. Do they always stay on the same side of the street and travel in a line from one house to the next? What do they do when they hit a street corner? Do they instead go in a kind of concentric circle out from the start house, where maybe the house across the street in front and behind and the two on either side are the next four “nearest” houses? They could conceivably do any of these or something else and still be within the vague method. This problem worsens if the locals are involved (such as neighborhood children used in the survey) who might try to direct the teams to houses they know have deaths to report. Since there’s discretion in how teams progress from the start house, how or why would they refuse a local who’s telling them to go to those houses over there? They can’t tell them it’s against the rules to go to them because it’s not. Nor would they want to do something that might insult the locals or make some of them start to question the “benign intent” of the survey. They can just choose to progress in the direction of those houses from the start house. In each cluster they’d have 38 chances to employ this discretion (and introduce bias).

As Kevin noted in the last thread: if we can plug whatever parameters we want into the model, there doesn’t seem to be a whole lot of point to this exercise.

If you mean trying to determine the extent of this bias in L2, you’re probably right that there is not much point in the exercise. The necessary information isn’t there and isn’t likely to ever be there. So almost any of the parameters could be the right ones. This of course means directly that there is not much point in deriving an estimate of nationwide deaths from L2 data because it hinges on choosing (knowingly or unknowingly) whatever of these parameters we want. The L2 authors (unknowingly) chose to plug in the parameters that gives them zero bias. These parameters make the number come out to 600,000. Another perhaps more plausible guess made (knowingly) by Johnson is the parameters that make the number come out to about 200,000. Since there’s no solid case for either parameter set, or others besides, there’s not much point in the exercise of making an estimate based on L2 data if the aim is to discover the level of deaths in Iraq. We may as well cut out the middle man and just guess the number, rather than guess the parameters that produce the number.

If you mean trying to explain or explore the potential for main street bias (what Johnson is doing) in a survey such as L2, then being able to plug a lot of different parameters into a model and showing the wide range of these and their outcomes has a useful point that, unlike L2, advances our knowledge about the topic it is addressing.

39. #39 Lee
February 20, 2009

Raymond says:

“Johnson et al’s presentation of this is more valid than Tim’s though because they say they do not know which streets or type of streets the L2 authors (field team actually, not the authors) called main streets in practice. Tim claims to be “correcting” their guess by replacing it with his guess. So he is making invalid claims about his guess while Johnson is not.”

One more time – BULLPUCKY!!!

Roberts has said that their main street – secondary street selection procedure was designed to give all houses an equal chance of inclusion. Lambert has shown that it is very easy to make a selection of main streets – secondary streets such that all houses have an approximately equal chance of inclusion. Johnson’s scheme is at odds with Roberts description – Lambert’s is not.

Johnson’s scheme is preferable only if you assume that Roberts is intentionally lying and L2 is fraudulent. You are adding to the despicable behavior here, Raymond.

40. #40 Raymond
February 20, 2009

“Roberts has said that their main street – secondary street selection procedure was designed to give all houses an equal chance of inclusion.”

All houses having an equal chance of inclusion is a desired outcome. A design intended to achieve this may succeed or not. The design the L2 authors have described can not claim to achieve this. If there is some hidden elements of their design that do achieve this they should let other people know what they are. That they don’t makes such statements seem like so much wishful thinking.

“Lambert has shown that it is very easy to make a selection of main streets – secondary streets such that all houses have an approximately equal chance of inclusion.”

No he hasn’t. He’s shown you can make a guess about the definition of main streets in one corner of a particular neighborhood map that might give all those households a highly unequal chance of getting into the sample.

“Johnson’s scheme is preferable only if you assume that Roberts is intentionally lying and L2 is fraudulent.”

I don’t do “faith” Lee. Nor should science. If you want me to believe a sample design achieved equal probability for all households then, like the maths teacher says, show your work. The design they’ve described doesn’t do it.

Dave Kane has a good posting up on his blog right now with a lot of comments from Roberts about the sampling which contradict each other and can’t all be true at the same time. You should check it out, and leave your faith for Sunday.

41. #41 Bruce Sharp
February 21, 2009

Raymond and David, thanks for taking the time to reply. The first point which you both raised — that Tim doesn’t describe his guess as a guess — seems like a non-issue to me. I’m not interested in how he describes his argument. I’m interested in his argument. Suppose you said to me, “Hey, I’ve got the funniest joke ever. This guy walks into a bar…”

I’ll either laugh or not laugh based on the joke. The fact that you told me it’s the funniest joke ever isn’t going to make any difference.

When I look at Tim’s example, I’m using a dirt-simple method to evaluate it: I look at the map, and I ask myself, “What would I describe as a ‘main street’?” I think I’d pick the same things Tim picked. I think most people would.

Incidentally, although I read the draft of the Johnson paper, I had not (until now) looked at the web pages Tim linked to, on the introductory pages. You are right: The authors do discuss other locations, and not just Baghdad. The comments there regarding smaller towns, however, seem plainly at odds with common sense: “It is not clear to us how the major commercial street sampling would work when there is no major commercial street available.” I don’t think any reasonable person would use the same criteria for what constitutes a “main street” in both a town of 1,000 people and a city of 1,000,000.

Raymond, in your explanation of why you don’t think the teams would have found all the main streets, it seems to me that your three reasons would lead to a bias in the opposite direction from what you suggest. If the teams were worried about time and safety constraints, and needed to find main streets, a broader definition of “main street” would be easier and take less time. (“Do you think this is a main street?” “No, let’s just wander around aimlessly for a while until we find a different one that looks ‘main-ier.'”)

In your reply to Lee, you said that the survey design can’t claim to have an equal chance of including all houses. I don’t agree: depending on the implementation of the survey, that may or may not have been the case. Again, the outcome depends on whether or not the chosen “main streets” were at all reasonable. I’m not sure why you say the chance of inclusion is highly unequal, but I’m going to guess that this, too, will come back to an unsettled question of implementation.

Although the discussion in this thread has focused primarily on the inclusion (or exclusion) of particular areas, it seems to me that even if this question were settled, we’d still be left with the impossible-to-calculate ratio of violence inside and outside the survey space. As Kevin has previously noted, the L2 scheme was borne of exigency, and it doesn’t seem likely that it will be used again. I don’t see how Johnson et al is useful, if divorced from L2. Its only value is in convincing people that L2 has no value… and I’m inclined to believe that most of the people who think L2 has no value had reached that conclusion long before they ever heard of main street bias.

Regards,
Bruce

42. #42 sod
February 21, 2009

Yes. The observation is wrong. The map in question is Kirkuk, for example. There are three maps shown on the MSB page linked above and none are for Baghdad.

their [main example](http://www.rhul.ac.uk/Economics/Research/conflict-analysis/iraq-mortality/visualsummary.html), the one from which they draw their horrible “visual summary” of “mainstreet bias” is Baghdad.

it is obvious. what Spagat is doing, is choosing only “city highways” with few exits/crossroads. this is, how he keeps the covered area small.

Because the sampling methodology does not establish an equal chance, practically or otherwise. In the best possible scenario it would give a very highly unequal chance of inclusion for each house, but some chance for all.

again: so does a GPS approach. bigger houses or “lonelier” houses have a higher chance than others.

There appears to have been little in the way of any specific criteria or conditions about what had to be called a main street. This was up to them to decide.

funny. i tend to recognise the “mainstreets”, when i enter a town or village. the one you enter the town on, is one. the big ones leaving that road (mostly leading to other other towns..) are.
in very big cities, it gets slightly more complicated, but even there its no rocket science.

For example, the team does it’s start house, there’s a house on either side that are both about as “near” to the first, they are both “neighboring” or “adjacent”. The team looks to one side and see a row of bombed or battered houses. They look to the other side and don’t see anything like this. Which way do they go?

“bombed and battered houses” will often be abandoned, leading to a NEGATIVE bias, reducing (polled) mortality!

how much experience do you have with polling? imagine you do a telephone poll. one voice on the line is a nice young female one, the other a barely understandable elderly man. who will you try harder to keep finishing the poll?

The L2 authors (unknowingly) chose to plug in the parameters that gives them zero bias. These parameters make the number come out to 600,000. Another perhaps more plausible guess made (knowingly) by Johnson is the parameters that make the number come out to about 200,000. Since there’s no solid case for either parameter set, or others besides, there’s not much point in the exercise of making an estimate based on L2 data if the aim is to discover the level of deaths in Iraq.

let me see, whether i got this right: the methodology of the Lancet authors, made with the aim to give an equal chance of cover for houses is “as plausible” as the Spagat guess at what happened, which is based on a choice of streets that contradict the target of the study?

this is insane!

43. #43 James Wimberley
February 21, 2009

In common parlance, a road is a public right of way wide enough for vehicles. A street is a stretch of road with lots of houses, shops and public buildings on it. Industrial estates have roads, not streets. On the map, the red line is a road, the yellow a street.
It beggars belief that any not half-witted survey team with a quota to meet would start from unpopulated stretches of road.

44. #44 LancetStudy
February 21, 2009

James Wimberley must be from UK (or UK-educated) given the way he distinguishes between streets and roads, which highlights a major misunderstanding on the part of all non-US people on this thread:

**The L2 scheme was designed by a US team. In the US, the “main street” of a town is the major highway. It does not mean a small street in the UK sense.**

So the main street for a US person is a “city highway” as Sod says. People do not tend to live on the main street. L2 started their survey scheme from “main street” in the sense that they then choose cross streets. So very few properties, if any, get included from the main street.
Likewise, in the JPR paper, the main street can happily be included in So, i.e. outside the survey space. The survey space includes cross streets to main streets, not necessarily the main streets themselves.
So the criticisms of JPR based on misconceptions (i.e. non-US interpretations) of “main street” are not valid. The L2 team included residences on cross streets, not main streets. Likewise JPR is completely consistent with main streets being outside the samplable space.

45. #45 sod
February 21, 2009

what part of Kirkuk is that on this [map](http://www.lib.utexas.edu/maps/middle_east_and_asia/karkuk_2003.jpg)?

46. #46 Lee
February 21, 2009

Bruce:
“The design the L2 authors have described can not claim to achieve this.”

Dude, Lambert just showed a map that matches what the L2 authors describe, and that does exactly that.

47. #47 LancetStudy
February 21, 2009

Lee wrote: “..what the L2 authors describe…”

Until we all know what the “main streets” were for L2, any claims are debatable.
The problem is that the L2 authors did not describe this is in sufficiently clear detail to avoid these repetitive debates.

Personally I don’t consider either Tim or JPR “right” or “wrong” in their interpretations of the implementation of the scheme. But I do wish I know something more from L2 to narrow down the discussion.

Side note: In terms of applicability of JPR more generally, I personally find it a useful and original contribution since many areas of medical imaging (my field) come across a very similar situation, where the objects which pick up tags (and hence can be seen) may be a biased sample of the population which are actually reacting, and the precise area of the sample being investigated under the microscope may also be biased.

48. #48 Kevin Donoghue
February 21, 2009

sod,

I think the east-west road that Tim is taking as his main street is roughly where you see the words NAHIYAT PIRYADI on your 2003 map. I fact I wouldnâ€™t be surprised if it was just that map which prompted Sean Gourleyâ€™s choice of colours. But the roads in red on that map seem more like major traffic arteries â€“ the sort of routes you would take if you were passing through Kirkuk, or driving from one side of town to the other.

Incidentally, your 2003 map highlights two things: (1) Gourley made lousy choices for his â€śmain streetsâ€ť, picking one which is actually at the eastern edge of the town; and (2) Kirkuk is a very big place. My guess is that, in such a large city, the JHU team would have sampled their â€śmain streetsâ€ť from a map (or maps) of the entire area, narrowing down their target cluster to within a few square miles before they even drove to Kirkuk. Thatâ€™s what Iâ€™d do anyway.

49. #49 Kevin Donoghue
February 21, 2009

James Wimberley is, I am reliably informed, an Englishman raised in the Channel Islands. But I wouldn’t be too quick to assume that “all non-US people on this thread” have got the wrong end of the stick. The Lancet is a UK publication and the authors of the JPR paper are mostly from that side of the pond. Actually, if it comes down to dialects, the crucial question is, what does Riyadh Lafta understand by main street? Given Iraq’s history I think it’s very likely that his English derives from British sources.

None of which takes anything away from Tim’s point: the efforts of Johnson et al. to derive n = 10 from that map should have prompted the editor of the JPR to ask what they were playing at.

50. #50 LancetStudy
February 21, 2009

Kevin Donoghue, you have it wrong. The authors of the L2 paper are mostly from the US, so it is their definition of “main street” which is under debate, since that is what would have formed the basis for the L2 survey plan. Looking at the JPR maps, it seems to me that JPR proceeded based on a US interpretation of “main street”.

The possibility that you raise that Lafta may have interpreted “main street” differently, because of British-biased education (your suggestion, not mine) then makes the whole L2 survey even more questionable. You are therefore suggesting that Lafta is the only one who actually knows what main streets would then have been included!! This would indeed be a serious scientific oversight of the US L2 team, since they would be co-authoring a paper (as lead author etc) based on a study where they do not know exactly what constituted a main street in practice.

You just shot down L2.

JPR team offered a theory of a generic situation of sampling in a population where all objects are not equal in terms of sampling. Very interesting. Whether their theory is general enough, or too specific or just right, depends on what you want to do with it. They then offered some values for a specific case, L2, and invited readers to use their own values. Seems like good science to me….

51. #51 LancetStudy
February 21, 2009

By the way, if you take the JPR interpretation of “main street” from their maps (which they freely admit are just a guide) then I count about 10 lengths of road which is non-samplable, for every given length of road that is samplable. Assuming a constant housing density per length of road, then n=10. So JPR are not unreasonable.

(Note that counting numbers of roads is not the issue, it is lengths of road if we assume constant housing density on residential roads. If we don’t assume that, then we are off onto another debate, and another can of worms for this thread).

52. #52 Jody Aberdein
February 21, 2009

‘good science….’

Wouldn’t that involve doing something like, I don’t know, testing the model?

53. #53 Kevin Donoghue
February 21, 2009

LancetStudy,

Well if you say I have it wrong, thatâ€™s that I suppose. Here is the passage which led me astray â€“ and it still does despite your best efforts to show me the light. I hope youâ€™re not going to ask me where I found it.

G Burnham, as principal investigator, was involved in the study design and ethical approval, took part in the analysis and interpretation of results, and led the writing of the paper. R Lafta managed the field survey in Iraq, participated in the study design and the analysis, interpretation, and preparation of the manuscript. S Doocy managed the study data and was involved in the analysis, interpretation, and the writing of the manuscript. L Roberts instigated the study and assisted with the analysis and interpretation of the data and the writing of the manuscript.

To me, that means that Riyadh Lafta was in the driverâ€™s seat. Iâ€™m not prepared to entertain the notion that my Hiberno-English-biased education (as you would presumably term it) is causing me to misread that paragraph. Iâ€™m really quite sure that what it says to me, is precisely what it says to Richard Horton.

I note that you are shocked, shocked at the suggestion that a bunch of Americans might have got into something where they did not know exactly what they were doing. That makes me wonder where youâ€™ve been for the last six years or thereabouts. Personally I give a bit more credence to the JHU team precisely because they had the good sense to give the crucial task to a group of Iraqis whom they deemed to have the required skills. If that invalidates the whole study in your eyes, so be it. Perhaps the guys who are losing their jobs on Wall Street will team up with former employees of the Justice Department to produce the sort of study you feel you can trust.

54. #54 Lee
February 21, 2009

The attempts to defend this paper have turned grotesque.

Are these guys really arguing that because in the US ‘main streets’ means ‘the main street’ (which is absurd), that street which are clearly among the main streets on the map are to be teated as secondary streets?

Becasue I’m from teh US, and that definition of main street is absurd beyond words. I drive up and down the California central valley frequently. When I do, I often stop at small freeway towns (5,000 – 20,000 people) to find a ‘taco truck’ to get lunch. To find one, I simply get off the freeway and DRIVE AROUND ON FREAKING MAIN STREETS!!! MULTIPLE. PLURAL.

They are easy to recognize, clearly distinct from secondary streets, there are always more than one even in small towns, usually more like 3-5, and that map at th etop fp this thread clearly excludes a main street from its main street category.

And I am from the US.

55. #55 LancetStudy
February 21, 2009

Lee: How many people live on your taco-rich “main street”? Very few. They are main streets/high streets in UK terminology. Few people live there, they are not in the sample space of L2 or JPR. They are just a starting point for the cross street method which L2 used. But knowing what they are is crucial since they are the starting point.

Jody: On this topic, many US citizens would consider Euston Road a “main street”. There are no markets on Euston Road, nor are cars allowed to park. Few people also live there in relative terms compared to rest of London. Instead, the action in terms of markets, cars parked, people walking and hanging around is on streets off of Euston Road. And people who live on streets off of these, sufficiently close, will walk there and tend to spend more time there. People who live way off will not.
Your analysis using London is from the perspective of someone who defines “main street” according to a European definition.

Kevin D: I am afraid you hit the nail on the head, albeit unintentionally, when you raised the possibility that there are “main street” definitions which Lafta might have used which differ from the US L2 team. So how can the US L2 team be so sure that no MSB exists? They cannot. How can they put their names first etc. on a paper where they do not know what the sample space actually was? They should not.
This is clearly why the AAPOR and JH college themselves, are worried enough to start investigations. Your argument validates this action.

56. #56 LancetStudy
February 21, 2009

Pre-empting somebody’s comment that the subject of this thread is JPR’s n=10 value, and hence that users should address the n=10 issue, I have above. Here it is again:

If you take the JPR interpretation of “main street” from their maps (which they freely admit are just a guide) then I count about 10 lengths of road which is non-samplable, for every given length of road that is samplable. Assuming a constant housing density per length of road, then n=10. So JPR are not unreasonable. (Note that counting numbers of roads is not the issue, it is lengths of road if we assume constant housing density on residential roads. If we don’t assume that, then we are off onto another debate, and another can of worms for this thread).

But the most important outcome of this thread, apart from this point, is the bombshell issue that Kevin Donoghue has raised about Lafta and Burnham/Roberts not being on the same page with regards “main street” definition. Should we have a thread on this?

(P.S. What Lee, Tim, Kevin, Jody guess as constituting a “main street” in the L2 survey sense, is irrelevant. What L2 define as a main street was, I had assumed, the main issue. But now Kevin has raised the important point that Lafta’s definition is actually a major issue. What a mess for L2!)

57. #57 Jody Aberdein
February 21, 2009

‘grotesque…’

Burnham et al central estimate of excess mortality 650 000 (390 000 – 940 000), allowing Johnson et al their guessed at R = 3, central estimate becomes 217 000, not sure what it does to the confidence interval.

Just to put this grotesqueness into context.

58. #58 Bruce Sharp
February 21, 2009

Lee, re your comment #46: I did not say that. That was Raymond, in #40.

Regards,
Bruce

59. #59 Kevin Donoghue
February 21, 2009

LancetStudy: But now Kevin has raised the important point that Lafta’s definition is actually a major issue. What a mess for L2!

Actually I just pointed out that Lafta managed the field survey. People who had read the paper closely already knew that.

BUT THEY EMPLOYED AN IRAQI!! IN IRAQ!! WHAT A MESS FOR L2!!

WTF? I mean really, is this what the defenders of Johnson et al. are reduced to?

60. #60 LancetStudy
February 21, 2009

Jody wrote: ‘grotesque…’. Yes, I agree. I think that everyone agrees that any death more than zero is grotesque. I am sure that the JPR team think the same. That is not the issue. The issue is to get to the bottom of whether science is being done in a reasonable way. To the extent that conflict surveys become a scientific method, then this method needs to be examined. If it is useful, great. If it needs to be corrected, then we should know — and we should try to estimate what these corrections might be. Only then will the method be treated as credible and full credit given to survey teams etc.

So yes, definitely grotesque. The question is: reasonable in terms of scientific standards or not? We are not doing any favours to anybody if it is not. 200,000 does the job in terms of being grotesque — no need to have an inflated figure, particularly if it then detracts from the main issue of how to prevent and stop wars.

61. #61 LancetStudy
February 21, 2009

KD wrote: “…is this what the defenders of Johnson et al. are reduced to?…”

As a whole, we readers of this thread are left with the remarkable issue that you helped fuel: Do *all* the L2 team know what main streets were possible candidates, and do they *all* agree on this? If they do not, then where does L2 stand?

I have an experiment that I did with someone, and the results are X. Well actually, I wasn’t there when they did the experiment, but the results are definitely still X. Well actually, I don’t know line-by-line what exactly they did, but the results are definitely still X. Hmmm???

I hope the investigative teams read these threads. This should help them with some of the issues.

62. #62 lancetStudy
February 21, 2009

As for JPR (which actually should be the least of your present worries if you are an L2 defender and have followed the above thread):

1. JPR article, and the possibility of MSB, stands strong (stronger, arguably, after these discussions)
2. the JPR maths is correct
3. the JPR estimates are indeed estimates, but not unreasonable ones given the uncertainty all round about L2

JPR does exactly what it says it does in the abstract. It does what it says on the can.

63. #63 LancetStudy
February 21, 2009

I’m sorry Kevin, but like it or not, you said in post 49:

“..the crucial question is, what does Riyadh Lafta understand by main street?”

You said it. I happen to agree with it, but you said it. You hit the nail on the head about why L2 should be investigated.

64. #64 LancetStudy
February 21, 2009

Jody wrote: “…Wouldn’t that involve doing something like, I don’t know, testing the [JPR] model?”

Sure, just tell the JPR team the main streets that were used. Failing that, what were the possible main street candidates? Failing that, what did Lafta understand to be a “main street” to use as a starting point? Failing that, what did US L2 team actually write in words in their design description to Lafta to translate the “main street” selection scheme?
Failing that, how did US L2 team conclude that Lafta’s implementation of the “main street” scheme gives R near to 1?

Of course, none of you know the answers to these questions. And yet you are surprised L2 is being investigated? Duh??

65. #65 Bruce Sharp
February 21, 2009

I guess Ron missed us.

66. #66 Lee
February 21, 2009

I will repeat (from 36), becae you guys are confirming it fo rme.:

“Roberts et al were designing a survey under very edgy ethical conditions. There was a credible very-non-zero risk of death to the surveyors, and to the people being surveyed for answering the questions. He was and is ethically constrained to to do everything possible to minimize the risk to the people being surveyed, and he designed the procedure with that in mind.

Information that was even borderline related to ways to identify surveyed neighborhoods or people was simply not transmitted to him, in order to maintain the confidentiality of people who were potentially at risk of their lives for answering the survey.

Now these guys come along and invent scenarios which allow them to demand that Roberts release information that he by design did not retain, for ethical reasons – and then use his ethically-constrained survey design and retention policy to lambaste his ethics, because he is not releasing the information that he for ethical reasons CAN NOT release and did not retain.

I suspect that this is intentional on the part of Robert’s critics. I know that it is despicable behavior.”

67. #67 sod
February 21, 2009

the map i posted above is from 2003.

i have serious doubts, that the Lancet team was cruising around towns, searching for “mainstreets”, when maps were easily available.

i also have doubts, that they would chose “mainstreets” only, that have “city” on one side only.

wouldn t you expect 4 iraqis to know at least one of the big towns they have to poll?

and the problem (as i said multiple times now, and still is being ignored by the “denialists”, the problem is much smaller in smaller towns..)

68. #68 LancetStudy
February 21, 2009

Lee wrote: “Information that was even borderline related to ways to identify surveyed neighborhoods or people was simply not transmitted to him [Roberts], in order to maintain the confidentiality of people who were potentially at risk of their lives for answering the survey.”

Yes, I absolutely agree that safety is paramount. No question.
**However**, that compromise brings us straight back to what I said earlier: **Given this admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists? They cannot. And so how can they put their names (in particular, as first author) on a paper when they do not know what the sample space actually was? They should not.** That is, I would think, one of the reasons (or the reason) behind the investigations.

If safety is such an issue, then be prepared to correct for any biases that might be introduced due to the safety constraint. Don’t just claim they are not there because you cannot estimate them.

69. #69 LancetStudy
February 21, 2009

Sod wrote: “…the problem is much smaller in smaller towns…”

Let’s assume the ‘problem’ Sod talks about is the uncertainty of main streets. In that case, it would be helpful if Sod could quantify his statement. What does ‘much smaller’ mean in numbers? Does he have a quantitative model for this, other than just words and presumption? In short, does he know what was done in terms of picking main streets? Saying that main streets are obvious when you see them, is hardly a scientific approach.

More generally, can someone offer an answer to the questions I have posed several times: Given the admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists? And so how can they put their names (in particular, as first author) on a paper when they do not know what the sample space actually was?

Silence will be taken as a ‘We don’t know’.

70. #70 LancetStudy
February 22, 2009

And as for the rest of Sod’s statements above: “i have serious doubts, that the Lancet team was cruising around towns, searching for “mainstreets”, when maps were easily available. i also have doubts, that they would chose “mainstreets” only, that have “city” on one side only. Wouldn t you expect 4 iraqis to know at least one of the big towns they have to poll?”

So, was the main street selection criterion through maps?? Or driving around?? Or asking locals for choice of main streets?? Or by choosing a fairly symmetric distribution of city on either side?? Or …. what?? You don’t know, nor do I, and nor does anyone reading this thread. And nor (it seems) do the US L2 team. So, again, I ask:
How can the US L2 team (i.e. Roberts and Burnham) be so sure that no main-street-bias exists, if they had no strict consensus with Lafta about what “main streets” were (as per Kevin Donoghue’s suggestion)? And so how can they put their names on a scientific research paper based on surveying, when they do not know the sample space — and they have no way of quantifying possible biases in the sample space?

By suggesting that Roberts and Burnham didn’t know exactly what Lafta did with regards the precise main-street selection, let alone the actual choices, you are shooting down L2. Kevin Donoghue started it, and now you all seem to be agreeing!

71. #71 lancetStudy
February 22, 2009

My final comment, is a direct challenge to Tim Lambert, who seemed to start this thread with the notion of shooting down the JPR ‘theory’ taken to mean their parameter estimates. But since n, and also the f’s and q’s, depend on the choice of Si and So, you are actually attacking the choice of Si and So made by JPR when estimating these parameter values. Specifically, you are attacking the JPR suggestion of what might constitute main streets, as confirmed by your focus on the Google map at the start of this thread. You think X are main streets, they think Y are main streets – and because of the differences, different parameter values emerge. No surprise.

Unfortunately for you, since the top of the thread, your fellow L2 supporters have started to offer a range of rather different methods for choosing main streets. From ‘it is obvious’, to asking Iraqi opinion, to using esoteric British road vs street definitions, to driving around looking for taco trucks! You yourself used Google maps, a method which is of course biased toward saying that a main street is a wide street when viewed from above (irrespective of levels of traffic etc.)

But what all this really comes down to, is “how did Lafta’s team choose main streets”? I doubt very much that they looked for taco trucks, so they must have followed some other scheme. But which?

I don’t know, you don’t know, nor does anyone else reading this thread — and the great revelation to me, prompted by Kevin Donoghue, is the fact that **nor do Burnham and Roberts know**. So there is indeed no point asking them. I get it now.

But now there is a very, very interesting scientific problem for you. What exactly was the sampling performed in the study that you support? How can you be so sure that no MSB exists in this sampling, without knowing anything about the MS’s (i.e. main streets)? And how can you support the quantitative conclusions of a survey-based paper (L2) containing several authors (i.e. Burnham, Roberts) who themselves do not know what the sample space was?

Of course you can say “No I don’t believe JPR”, but believing and scientifically rejecting are very different as you know as a scientist yourself).

72. #72 sod
February 22, 2009

“LancetStudy”, please tell us, what is keeping you from doing your own study in Iraq?
none of us is holding you back!

the big picture is this:
to show a problem, Spagat needs to show that the Lancet method causes a bias in households chosen (1), that these households have a significantly higher deathrate (2) and that this lead to false results of the study (3).

he has done none of these.

73. #73 sod
February 22, 2009

Let’s assume the ‘problem’ Sod talks about is the uncertainty of main streets. In that case, it would be helpful if Sod could quantify his statement. What does ‘much smaller’ mean in numbers? Does he have a quantitative model for this, other than just words and presumption? In short, does he know what was done in terms of picking main streets? Saying that main streets are obvious when you see them, is hardly a scientific approach.

small village will often only have one mainstreet. it is the one, on which you entered the village. and the 40 households rule allows for more “penetration” beyond the crossstreet, in a place that only consists of 100 households.

So, was the main street selection criterion through maps?? Or driving around?? Or asking locals for choice of main streets?? Or by choosing a fairly symmetric distribution of city on either side?? Or …. what?? You don’t know, nor do I, and nor does anyone reading this thread. And nor (it seems) do the US L2 team. So, again, I ask: How can the US L2 team (i.e. Roberts and Burnham) be so sure that no main-street-bias exists, if they had no strict consensus with Lafta about what “main streets” were (as per Kevin Donoghue’s suggestion)? And so how can they put their names on a scientific research paper based on surveying, when they do not know the sample space — and they have no way of quantifying possible biases in the sample space?

a mainstreet is something different in a village than in Baghdad. the strict definition that you and Spagat want, would have made work on the ground more difficult, not eassier. a “functional definition” (need enough mainstreet to achieve reasonable coverage) is fine for me.

all those criterions are fine. (even if we know now, that they didn t ask locals, this would have been a reasonable approach).

i don t know the details about most polls i read. (actually i know more about the lancet one than about many others together) with most polls, you will never know all details. (marketing or political polls keep some parts of their methods secret)

the Lancet results are in very good agreement with basically all other estimates of mortality in Iraq.

this is a huge problem, for all these bias claims.

Spagat has shown ZERO evidence, that supports a “mainstreets” being significantly more dangerous to people living nearby. he has at best done a sloppy (biased might even be a better term..) job, in showing a lack of “coverage”.

74. #74 Robert Shone
February 22, 2009

I had another look at Tim Lambert’s map (and the area immediately above it). I think Lambert has some explaining to do:

http://dissident93.files.wordpress.com/2009/02/msb-lambert.jpg

75. #75 LancetStudy
February 22, 2009

Sod wrote:”all those criterions [for choosing main streets] are fine”

Sure, **if** (and only if) resulting biases are then corrected (e.g. using JPR).

Sod wrote: “a mainstreet is something different in a village than in Baghdad”

Of course. So what is a “main street” in Baghdad? Tikrit? Basra? When we know that, we know Si using cross street selection. When we know Si, we know So. Then JPR can make improved estimates of n, f and q.

76. #76 LancetStudy
February 22, 2009

**Given the admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists? They cannot. And so how can they put their names (in particular, as first author) on a paper when they do not know what the sample space actually was? They should not. End of story.**

77. #77 Kevin Donoghue
February 22, 2009

Robert, since it’s past midnight in Australia, if you want a quick answer I’ll hazard a guess as to Tim’s meaning. Your own “junction A” suggestion seems to fit the bill. I take it that “above” refers to the map above Tim’s text, since he used the word twice in the same phrase and the first reference is clearly to his map.

78. #78 Robert Shone
February 22, 2009

Thanks, that’s what I thought (the road at the left going upwards through “junction A”). If that’s the case, I think Tim’s got an uphill struggle demonstrating that it’s an “obvious” main street, since it appears to be nothing of the sort.

I said I wasn’t going to post any more in the bowels of this thread, so I guess I’ll leave it until Tim’s sequel, MSB Debunk III: This Time it’s Personal.

79. #79 Robert Shone
February 22, 2009

I’ve amended the text slightly, as Kevin is probably right about the meaning of Tim’s wording, “one that it is off the map above”.

80. #80 LancetStudy
February 22, 2009

What Tim has to remember as he liberally defines streets as “main streets” is that he is automatically increasing the samplable space Si such that the survey team would (by definition) have to then be able to cover these distances and enter these neighbourhoods. In the end, if n=0, this means by definition that Si covers all of the governerate. So the survey team would have to be able to go **everywhere**.

Since they did not go everywhere, by their own admission for safety reasons and need for speed, we are back in the same position. What roads were left out? In other words, what is the space So in practice for Lafta, and hence what is Si for Lafta (since it is I-So in set formalism)? Only when we know this will better estimates of Ni, No, fi, fo, qi, qo be possible.

More generally, how different is Lafta’s implemented Si from Burnham and Gilbert’s imagined Si? They don’t know, so how can the US L2 team possibly claim R is near 1 for Lafta’s implementation?

81. #81 Kevin Donoghue
February 22, 2009

Robert, AFAICT from Google Earth it’s probably on a par with Oxford Street, London or Parnell Street, Dublin – going by the fact that it remains clearly visible as I zoom out from it. Those two streets are main streets in my dialect at least. Also if the accompanying photgraphs are correctly located (can’t be sure of that of course) then it’s a bustling street with a mosque and other prominent buildings.

I would suggest that those who have any doubts about this use Google Earth to do a comparison with main streets they know.

82. #82 Robert Shone
February 22, 2009

Kevin Donoghue writes:

it’s probably on a par with Oxford Street, London or Parnell Street, Dublin – going by the fact that it remains clearly visible as I zoom out from it.

Come off it. I know rural lanes that are as visible. It looks half the width of Tim’s other “main” street. The presence of a mosque doesn’t make it a “main” street – or does it?

83. #83 sod
February 22, 2009

Given the admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists? They cannot. And so how can they put their names (in particular, as first author) on a paper when they do not know what the sample space actually was? They should not. End of story.

you are using the term “MSB” as if it was a long standing,we ll known and defined phenomenon. it actually isn t. MSB is an invention by Spagat, and he has brought up ZERO evidence, to support his claim, that people living in those areas have a significantly higher violent deathrate.

84. #84 LancetStudy
February 22, 2009

Tim: I hope you can appreciate how the above to-and-fro exactly sums up the main-street weakness in L2. Even your own supporters cannot define what a “main street” is after 100+ (total) posts, so imagine what the conversations would have had to be between Lafta and Burnham/Gilbert, then Lafta and his team, in order to pin it down prior to survey-day? Impossible.

85. #85 LancetStudy
February 22, 2009

Sod writes: “you are using the term “MSB” as if it was a long standing,we ll known and defined phenomenon.”

Er, it has existed since October 2006 by my understanding, and has been published.
L2 has existed since October 2006 in published form.
Both are ‘long-standing, well known and defined phenomenon’.

86. #86 LancetStudy
February 22, 2009

Maybe Sod has raised another crucial new issue here, intentionally or unintentionally: Maybe previous/other studies of conflict mortality implicitly contained MSB as well! I hadn’t even looked into that. Oops, we should. Has this cross-street-to-main-street been used before? Please let me know. It would be very, very interesting to go back and look at such cases to see what was concluded.

Remember my point is: Yes this cross-street-to-main-street method could conceivably be used, but only if an estimate of MSB is made. It is such an obvious detail to wonder about, after all, even without the MSB name and prior to JPR article. If it is near 1, fine — but show how you obtained that estimate based on maps, taco trucks, or whatever… Just saying “R is near 1” isn’t convincing (hence hundreds of postings on these sites…)

87. #87 LancetStudy
February 22, 2009

Sod wrote: “MSB is an invention by Spagat”

Er again…. That is what original research is. New results not previously thought of or published. So yes, I guess Spagat et al. are guilty of ‘inventing’ the term MSB — but they are not guilty of inventing the effect. To the extent that R is not **exactly** equal to 1, the effect is generated exclusively by the L2 team’s failure to examine the Lafta implementation, and adjust their estimate of mortality for any street-selection bias.

Basic scientific thinking — even if R then turns out to be near 1, somebody has to account for such a source of potential bias during the final analysis prior to publication. Amazing that L2 apparently didn’t think of doing it prior to JPR.

88. #88 Kevin Donoghue
February 22, 2009

Robert Shone: It looks half the width of Tim’s other “main” street.

And Tim’s other main street looks about the same width as the one the MSB squad chose, which runs vertically at the right of their map. Actually I think the one Tim concentrated on looks a bit more prominent. As for the one you are contesting, which may or may not be the one Tim has in mind, I think it’s going through the old part of the town, with the Kirkuk citadel on one side and that mosque (a pretty big one judging by the dome) on the other. It’s not unusual for streets to be narrower in older parts of a city.

I refer you again to the examples I gave: Oxford Street looks a lot narrower than Marylebone Road, just to the north of it. So what? Surely Oxford Street is a main street? And Parnell Street is of course only a fraction of the width of O’Connell Street. But they are both visible relative to their surroundings – just about – when I zoom out to an altitude of 12,000 feet or so on Google Earth.

AFAICT for someone doing a survey it would be an obvious choice of main street, assuming they wanted to give as many households as possible a fair chance of inclusion. If you’re assuming anything other than that, then you must think they were either dishonest or sloppy; and if you think that, what do you need Johnson et al. for?

89. #89 Robert Shone
February 22, 2009

Thanks, Kevin, I get the gist. From 12,000 feet up, you think it has similarities with Oxford Street. That makes it an “obvious” choice as a “main” street.

90. #90 Kevin Donoghue
February 22, 2009

Your “politeness” overwhelms me, Robert. There was really no need to thank me. Just to observe your virtuosity with scare quotes makes all my efforts worthwhile.

91. #91 LancetStudy
February 22, 2009

Kevin D: The real issue has moved on from “what is a main street” to “how could Burnham/Roberts dismiss street-bias **without** knowing the streets that were samplable?”

**Let’s be crystal clear on this: Since R depends on Si, how could Burnham/Roberts state R=1 effectively without knowing Si?**

**R is defined through Si !!**

In terms of the more general bias question “Is R=1 or not?”, it matters little whether the scheme used involved main streets, very curvy streets, streets with the letter “s” in them, or streets with taco trucks — or whatever. What matters is: Is the scheme that was used by Lafta (whatever that was) open to street-bias? And most importantly, how can Burnham/Gilbert say it isn’t (i.e. R=1) without knowing a typical Lafta-class of streets (i.e. without knowing Lafta’s Si) or even possible Si’s that Lafta’s scheme would have employed? Actually, you helped raised this issue yourself.

Remember, unlike a disease which tends to spread over large distances without regard to street design, violence is largely fought by people on streets. So this issue is crucial. Violence may have a systematic predominance in particular street topologies, so there is an important potential source of bias depending on street selection process.

And Sod has now raised the amazing possibility that this L2 technique has been implemented in other conflict studies. (That is very, very interesting and I would like to pursue this).

Kevin: One of the main values of the JPR paper is that is provides a concrete framework for this whole discussion. Parameters, estimates etc.

(Tim: Please take this as a personal and informal side comment, from a colleague. With all due respect, maybe you should revisit the title of your thread. Stating that something is ‘badly flawed’ sounds like something a lawyer would have a field day with in terms of libel. I am no lawyer, but how do you know lawyers don’t read this? Since that whole investigation issue arose, maybe we should all be careful about direct accusations. Questions are fine, and valid, and important. Opinions as well, but more direct things are perhaps more dubious….)

92. #92 sod
February 22, 2009

Even your own supporters cannot define what a “main street” is after 100+ (total) posts, so imagine what the conversations would have had to be between Lafta and Burnham/Gilbert, then Lafta and his team, in order to pin it down prior to survey-day? Impossible.

my functional definition from #73 is fine. this is just one of many false claims you make!

Maybe Sod has raised another crucial new issue here, intentionally or unintentionally: Maybe previous/other studies of conflict mortality implicitly contained MSB as well! I hadn’t even looked into that. Oops, we should. Has this cross-street-to-main-street been used before? Please let me know. It would be very, very interesting to go back and look at such cases to see what was concluded.

i have two answers to this:

1. this is the answer i have given before: the lancet method was chosen, because of an EMERGENCY. with GPS becoming smaller and cheaper and google maps, it most likely wont be repeated.
now ask yourself: why did Spagat chose an example for his THEORETICAL paper, that isn t a typical case and wont be repeated? one that (according to them) has multiple other problems, one among them missing vital data?!?

2. on the other hand, you are upon something BIG. while there aren t many casualty studies using this approach, you will find thousands of example in marketing!
before telephone polling became state of the art, people did real polling, on real streets. mainstreets, mostly! according to the Spagat paper, the majority of people polled on a mainstreeet (those are the ones who would be killed by a bomb) are living just around the corner, in a street crossing that mainstreet! for decades (?) marketing decisions were based on a small special subset of people, those living with a mainstreet bias!!!

93. #93 sod
February 22, 2009

sorry for the font above. the list seems to be using some auto-function.

Er again…. That is what original research is. New results not previously thought of or published. So yes, I guess Spagat et al. are guilty of ‘inventing’ the term MSB — but they are not guilty of inventing the effect. To the extent that R is not exactly equal to 1, the effect is generated exclusively by the L2 team’s failure to examine the Lafta implementation, and adjust their estimate of mortality for any street-selection bias.

slow again, for you: MSB has two parts in the Spagat paper:

1. MSB meaning that Lancet concentrated on polling mainstreet areas. the argument that Spagat makes on this point, is at best weak, as Tim and others have demonstrated.

2. MSB meaning that more people LIVING in that area get killed by violence. Spagat offers ZERO evidence to make this point. none. nada. zilch.

your claim that the L2 authors should act on such a weak problem, is weak. at best.

Remember, unlike a disease which tends to spread over large distances without regard to street design, violence is largely fought by people on streets. So this issue is crucial. Violence may have a systematic predominance in particular street topologies, so there is an important potential source of bias depending on street selection process.

so you claim that diseases don t spread better, in places where many people meet?

according to Spagat, diseases should concentrate on people living along mainstreets…..

94. #94 sod
February 22, 2009

Tim: Please take this as a personal and informal side comment, from a colleague. With all due respect, maybe you should revisit the title of your thread. Stating that something is ‘badly flawed’ sounds like something a lawyer would have a field day with in terms of libel.

Tim is stating facts. he is attacking the paper, not making claims about the authors and fraud.

other have made much harder accusations. against the lancet authors. you must have missed this, by chance….

95. #95 Tim Lambert
February 22, 2009

Lancetstudy says:

>Tim: Please take this as a personal and informal side comment, from a colleague. With all due respect, maybe you should revisit the title of your thread. Stating that something is ‘badly flawed’ sounds like something a lawyer would have a field day with in terms of libel. I am no lawyer, but how do you know lawyers don’t read this?

for those who haven’t guessed yet,

LancetStudy = ron = ozzy = Nick = Lancet Debunker = Tell

And they are all sock puppets for one of the authors of the MSB paper.

96. #96 Tim Lambert
February 22, 2009

Robert Shone, the other main street I added was the one that runs through your intersection A, intersecting with your secondary roads 1 and 2.

97. #97 Robert Shone
February 22, 2009

Tim Lambert writes:

Robert Shone, the other main street I added was the one that runs through your intersection A, intersecting with your secondary roads 1 and 2.

Thanks for confirming it. In that case I think your map loses all credibility, since the best you can say for one of your cross roads (road 1 in my map) is that it joins another cross road (2) at a junction (A), and that another road which you have arbitrarily designated as a “main” street runs through the same junction.

That’s very weak, and I’m not surprised that you didn’t illustrate this part of your selection scheme in your (arguably misleading) map.

98. #98 Jody Aberdein
February 22, 2009

Maps.

So although this certainly doesn’t help with q or f0 or fi, looking at maps perhaps helps with n.

I’ve been playing around with QGIS and OSM, with some files from GEOFABRIK. It’s a bit tough for a non-compsci, non epidemiologist but actually not too bad once you get into it. So far I think I can tell you that for Enfield, using the sampling process as in Burnham et al, with main street being strictly motorway, trunk or primary road, i.e about 6 roads only, N seems to be for this map around 2.5 judging by road length samplable.

Anyhow you can pull data directly off OSM, and hence for anywhere you like in the world. You can classify it and extract intersections, coordinates, lengths and such like in QGISS, and you can export files to good old Open Office or some such to do the maths.

99. #99 Jody Aberdein
February 22, 2009

Diseases.

Yes this is interesting. Cluster surveys have of course much earlier been used to study not only vaccination coverage, but for example diarrhoea prevalence. As you might imagine inhomogeneity is important in such cases. As you also might imaging there is a reasonable literature on what to do – to account for ‘design effect’, the proper term for which we have the euphemism ‘main street bias’. So you’ll note that Roberts et al 2004 specifically did their analysis with methodology taking into account ‘design effect’. Perhaps tellingly the seminal paper from Johnson et al doesn’t mention ‘design effect’, or the scientific discussion thereof even once.

100. #100 Tim Lambert
February 22, 2009

Robert, I take it that you are no longer disputing that the big street running along the bottom of my map is a main street. So even if you don’t count my second additional main street as a main street, the only change to my map is that the yellow street that clips the upper right corner goes away, resulting in a small blue area around that street and the rest of the map reachable and n at most 0.1.

And I know that you will now make a big song and dance about how this proves that the Lancet methodology is completely unknowable, but whether n = 0 or 0.1 makes no significant difference to the results of the Lancet study.

And I note that no-one is disputing my point that the blue areas should not include the 39 neighbours of the start house. This makes a huge difference and the MSB authors somehow forget to include this in their map.