# Journal of Peace Research publishes badly flawed paper, part 2

This post is some more notes on a reply to the badly flawed “Main Street Bias” paper.

The authors claim that it is plausible that the Lancet paper’s sampling scheme could have missed 91% of the houses in Iraq. (That is, their parameter n, the number of households in the unsampled area divided by the the number in the sampled area could plausibly be 10 or more.) The only support they offer for this is a reference to this analysis of Iraqi maps.

To the right is a detail from their map. The red lines are main streets and the yellow are secondary streets. They assert that the blue areas are not samplable using the Lancet scheme, and yes, the blue area covers 90%+ of households. But there are two things wrong with their map.

Look at this larger scale version of their map:

First have a closer look at their yellow road that they say should not be counted as a main street and compare it with the one they count as a main street. (Scroll around on Google Maps if you want to see more of each street.) It’s just as wide and has a similar amount of traffic and a similar number of large buildings. Clearly it should have been classified as a main street and it is not even slightly plausible that someone who was trying to get every house in the sample frame would leave it out.

Second, the random house on the secondary street is only the start point for the cluster, which also includes 39 houses neighbouring the start point. This means that you can sample houses on tertiary streets that are a few houses from the secondary street. I added two obvious main streets to the ones they chose (the one across the bottom of the map above and one that it is off the map above), and redrew the blue areas to taken this factor into account.

What’s that? You don’t see any blue areas in my map? That’s because there aren’t any. Make just these two corrections to their map and the unsampled area is 0. In their model, that means n=0 and there is no main street bias.

1. #1 Robert Shone
February 22, 2009

I’ve written a new blog piece expanding on the areas which Lambert misleadingly left out of his map:

http://dissident93.wordpress.com/2009/02/22/msb-lambert-update/

2. #2 Eli Rabett
February 22, 2009

If you blow up the Google map, just about every street labeled as secondary by the Sapgatti is as busy or busier than one or more of the streets they label as main. Whatever. Look at the maps

3. #3 David Kane
February 22, 2009

Tim claims:

And I note that no-one is disputing my point that the blue areas should not include the 39 neighbours of the start house.

Untrue! I am disputing it. Recall what L2 itself claimed:

On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed.

I (and, I think, the MSB folks) interpreted this to mean that households had to be on cross streets to be in the sample. That is, at least, one reading of what “adjacent” means. Do you have proof that your definition of “adjacent” is more correct that Johnson et al’s?

No. I have shown conclusively that anyone who claims to know the details of the sampling procedure for L2 is lying.

Now, your guess that the string of houses starting from house X could go down all sorts of side streets might be correct. Similarly, Johnson et al’s guess that “adjacent” means on the same street as the starting house might be correct. No one knows.

By the way, although this is somewhat forward of me, I would appreciate it if Tim would start a new thread devoted to just the issue of what we know and don’t know about the sampling plan used by L2. It would be helpful to gather some of the collective wisdom of L2 defenders on this point.

4. #4 Robert Shone
February 22, 2009

Tim Lambert writes:

I take it that you are no longer disputing that the big street running along the bottom of my map is a main street.

I do dispute that it “is” a main street. But only because that’s a meaningless statement. We have no universal main-street-o-meters. We have to first agree on what defines a main street, then we can make meaningful statements about whether streets fit that definition. I think the Lancet authors defined main streets as “major commercial streets or avenues”, which isn’t much help, as the MSB team rightly pointed out.

Perhaps the best definition for the Lancet study is: main streets constitute that set of roads which when selected as “start” roads will guarantee, via the Lancet sampling scheme, n=10. Of course, that doesn’t help in their selection, unless one knows in advance which roads will guarantee that result.

5. #5 Robert Shone
February 22, 2009

n=0, I mean, 😉

6. #6 Lee
February 22, 2009

“LancetStudy = ron = ozzy = Nick = Lancet Debunker = Tell

And they are all sock puppets for one of the authors of the MSB paper.”

So at least one of the authors of the MSB paper is both fundamentally dishonest and technically incompetent.

Who would ever have guessed?

7. #7 LancetStudy
February 22, 2009

Tim Lambert wrote: “..for those who haven’t guessed yet,
LancetStudy = ron = ozzy = Nick = Lancet Debunker = Tell
And they are all sock puppets for one of the authors of the MSB paper.”

Er, wrong again Tim. You have misled yourself through a superficial analysis of apparent assigned IP addresses — which, come to think of it, sounds a bit like how you have misled yourself through superficial looks at the MSB paper, and the methodology issues around L2 discussed by some of us on this thread. Hmm, maybe you should improve your detective work by focusing on what L2 actually did (or didn’t do), and spend less time worrying about who might be who on this thread, or deflecting your general frustrations onto JPR’s honest attempt to interpret L2’s uncertain methodology?

(PS My interest in this thread was certainly started by a conversation with one of the JPR authors, no secret that. Want to guess which author it was? Or want to guess which street is a main street in city X? Or want to guess what Lafta and Burnham/Gilbert exchanged in terms of specific instructions about starting points? …Or maybe just admit your anti-MSB campaign ran out of steam…)

8. #8 Lee
February 22, 2009

Just to point it out again – the MSB authors are implicitly accusing the Lancet authors of incompetence or fraud. Burnham has said that the sampling scheme was designed to give every house an equal chance of being picked – the scheme that gives the coefficients that the MSB authors use misses that mark by a factor of more than 10. The only way they can make street assignments to get such a scheme is to assume that Burnham was lying, or that the design was an incompetent failure by more than an order of magnitude. MSB authors are accusing Lancet authors of incompetence or fraud, while attempting to not actually use the words.

And now we have LancetSTudy here, trying to explain that it only looks like “LancetStudy = ron = ozzy = Nick = Lancet Debunker = Tell”, because Lambert “misled yourself through a superficial analysis of apparent assigned IP addresses.” Which just happens, it seems, to be the “apparent” IP address of one of the Lancet authors.

To which I can only say: guffaw!

9. #9 sod
February 22, 2009

Perhaps the best definition for the Lancet study is: main streets constitute that set of roads which when selected as “start” roads will guarantee, via the Lancet sampling scheme, n=10. Of course, that doesn’t help in their selection, unless one knows in advance which roads will guarantee that result.

no need to know everything. but it would keep them from making the lousy choices that Spagat assumes they took.

I (and, I think, the MSB folks) interpreted this to mean that households had to be on cross streets to be in the sample. That is, at least, one reading of what “adjacent” means. Do you have proof that your definition of “adjacent” is more correct that Johnson et al’s?

no. but their lists of assumptions (adjanced means in the crossroad. mainstreet means city highway. the Lancet team chose only few roads and mostly those at the edge of town) basically is the SINGLE REASON, why they got the result that they wanted (n=10).

so you admit that their analysis is based on a list of completely biased and wild assumptions?!?

PS My interest in this thread was certainly started by a conversation with one of the JPR authors, no secret that.

10. #10 sod
February 22, 2009

By the way, although this is somewhat forward of me, I would appreciate it if Tim would start a new thread devoted to just the issue of what we know and don’t know about the sampling plan used by L2. It would be helpful to gather some of the collective wisdom of L2 defenders on this point.

i think it is telling, how Kane and the others are trying to shift this discussion into “we don t know anything about the lancet methodology” direction.

let me sum this topic up:

Tim and others, have shown without doubt, that by even a tinny change to the Spagat picture, you get a completely different result!

what we know or don t know about the sampling procedure, is completely irrelevant to this point. Spagat came to his conclusions, by very specific assumptions, that contradict what the Lancet authors say.

11. #11 Tim Lambert
February 22, 2009

So “LancetStudy” are you really trying to argue that the identical IP addresses, writing style and familiarity with the MSB paper was just some whacky coincidence? I only checked the IP addresses because it seemed a bit odd that six new commenters who all wrote the same would all show up one after the other.

12. #12 Lee
February 23, 2009

@108
edit the penultimate sentence to read – “.. of one of the MSB authors”

13. #13 Michael
February 23, 2009

It’s amazing what a cacophany of complaint the Lancet Study has thrown up.

From the simple “it’s wrong” of George W. to the more sophisticated claims of “MSB”, the common thread seems to be that of wanting it not to be true.

14. #14 Robert Shone
February 23, 2009

Tim Lambert writes:

So even if you don’t count my second additional main street as a main street, the only change to my map is that the yellow street that clips the upper right corner goes away, resulting in a small blue area around that street and the rest of the map reachable and n at most 0.1.

Since Tim’s usual response would be along the lines of “Shone is wrong, it is a main street”, I take the above to be his way of conceding that his second additional main street was not an “obvious” main street.

Unfortunately Tim’s n=0.1 claim is just as misleading as his map. If he’d included the area just above his map, it’s pretty clear the value for n would be much higher than 0.1. Perhaps if he crops the top inch off his map, he can return to his n=0 assertion?

His map is fundamentally misleading, as I demonstrate here.

15. #15 sod
February 23, 2009

His map is fundamentally misleading, as I demonstrate here.

i don t like to break it to you, but your post demonstrated nothing at all.

you make a big fuss about junctions. but (as always) i think this is just symptomatic of you, not having this well thought out. in the absence of roadsigns and names, a road ends when it ends, or when it clearly flows into another one, or when it gets smaller for a significant stretch.

the Spagat map is fatally flawed. they chose city highways as mainstreets. in the map that they use to illustrate their point, they chose streets that are on the edge of town. their choice of streets is the real bias!

16. #16 Robert Shone
February 23, 2009

Lambert has said that the road he redesignated as a “main” street (but which wasn’t shown in his own map) was the one which runs upwards through junction A (on my map). Lambert originally described this as an “obvious” main street, but it’s nothing of the sort (it’s half the width of Lambert’s other designated “main” street, for example).

The best that Lambert can claim for one of his secondary (or “cross”) roads (road 1 in my map) is that it joins another cross road (2) at junction A, and that the road which he has arbitrarily designated as a “main” street runs through the same junction.

In other words, even if a survey team agreed with Lambert’s arbitrary designation of the junction A road as a “main” street, it’s debatable whether (using the published Lancet sampling methodology) they would select road 1 as a cross street. Looking at Junction A, it seems equally (or perhaps more) likely that they’d class road 2 as the secondary street, with road 1 as a tertiary road leading off road 2. But nobody actually knows, since nobody knows how the Lancet sampling scheme worked in reality.

What this ambiguity over classes of road shows, at this level of detail, is that Lambert is misleading his readers when he claims that the MSB map (which he redrew) is “wrong”. The most he can say is that he has a different subjective designation of roads, which has its own problems in terms of plausibility.

Lambert’s map left out the least plausible part of his selection scheme. For further details, see: http://dissident93.wordpress.com/2009/02/22/msb-lambert-update/

17. #17 sod
February 23, 2009

the road at the bottom is a mainstreet. (it is VERY similar to the other mainstreets that Spagat chose) just including that one road, changes the outcome of the “bias” significantly!

the reason for this is, that this road (in contrast to the ones that Spagat prefers to choose) has many crossroads and runs through the middle of a build up area.

your claim is, that you know the Lancet sampling good enough, to be sure that Spagat got it right down to every single road!

and you are still forgetting that Lancet idea of including enough roads to get a big sampled area…

18. #18 Robert Shone
February 23, 2009

sod writes:

Thanks – I take your point about repetition. And you’re certainly the expert on that (having repeated yourself about a thousand times in bold emphasis on the gender issue in the last MSB thread).

Still, an important point is sometimes worth repeating a few times: Lambert’s map is fundamentally misleading (why? See #116).

19. #19 sod
February 23, 2009

nice Shone, you managed to write another posts, without addressing that road at the bottom (making a very big difference).

and talking about gender, this reminds me of your silence on the mechanism, that kills local working age males on the street, where they are outnumbered by non-local males and in their houses (where they are outnumbered by females/kids/elderly). Spagat gives a whole new meaning to the term “precision bombing”.

20. #20 LancetStudy
February 23, 2009

Tim wrote: “So “LancetStudy” are you really trying to argue that the identical IP addresses, writing style and familiarity with the MSB paper was just some whacky coincidence? I only checked the IP addresses because it seemed a bit odd that six new commenters who all wrote the same would all show up one after the other.”

Just as R is more than 1, there are more than 1 persons who think your arguments are wrong and that you are driven by non-scientific goals with the L2 defence.

21. #21 Robert Shone
February 23, 2009

sod writes:

…gives a whole new meaning to the term “precision bombing”.

Talking of which, do you know what Les Roberts meant by the following:

Our data suggests that the (March 2003) shock-and-awe campaign was very careful, that a lot of the targets were genuine military targets. So, I think it is correct that in 2006, probably in almost any month, there were more civilians dying than during shock-and-awe. http://tinyurl.com/4yo5uw

Is he really saying that shock-and-awe was “careful”, with “genuine” targets?

It sure looks like it.

22. #22 Tim Lambert
February 23, 2009

So LancetStudy, you are not denying that you are Ron/ozzy etc.

23. #23 elspi
February 23, 2009

This has been another addition of SOCK PUPPET THEATER.

Tune in next we as hear Mary Rosh say ” As Herr Doctor Professor Lot complete his lecture, all the women in the lecture hall had an I’ll-have-what-she’s-having moment.”

24. #24 LancetStudy
February 23, 2009

Tim Lambert wrote: “So LancetStudy, you are not denying that you are Ron/ozzy etc.”

Yes I am denying it. Sharing common IP addresses does not mean sharing genes!

This is an interesting insight into you/your thread. More interested in working out who is who, that what is what in L2??

25. #25 Bruce Sharp
February 23, 2009

Tim, you’ve failed to consider that some IP addresses are more common than others. This is obviously because of MSB: Main Subnet Bias.

26. #26 Robert Shone
February 23, 2009

Tim Lambert seems to have form when it comes to really obsessive policing of identity, and there does seem to be a police mentality behind it – the business of “catching people in the act”, which is what Tim’s blog is mostly about.

He also has form when it comes to revealing the identities of posters without their consent.

27. #27 sod
February 23, 2009

Shone, two more posts, not a word on the road at the bottom of the map, changing the Spagat result significantly..

Is he really saying that shock-and-awe was “careful”, with “genuine” targets?

yes. as all so often, i agree with Roberts. the spike of violence after the Samarra bombing in 2006 was more deadly for iraqi civilians, that the original US campaign.

This is an interesting insight into you/your thread. More interested in working out who is who, that what is what in L2??

Tim wrote a pretty good post about the weakness of the “stronger” part of the Spagat “analysis”. you failed to address the most of it. pretty weak, for someone with access to the authors.

Tim Lambert seems to have form when it comes to really obsessive policing of identity, and there does seem to be a police mentality behind it – the business of “catching people in the act”, which is what Tim’s blog is mostly about.

you too, haven t taken a look at the post at the top of this topic, have you?

i have seen “police mentality” and outings on denialist blogs. this is nothing of that sort.

i think it is rather interesting to know, that all those posters share IP and that LS has contacts to the authors.

28. #28 Robert Shone
February 23, 2009

sod writes:

I’ve already addressed it in #3, #104, etc, but if you want the obvious stating:

1. It is what it is – a wide, straight road, busy with traffic, cutting directly through a residential area, but with buildings on the left section of it that appear to be larger than residences.

2. Obviously, you could easily classify it as a “main street”, depending on your criteria.

3. There might be reasons for not classifying it as a main street, again depending on your criteria (it might not be “commercial” enough – that being one of the Lancet team’s criteria apparently: see my comment #3. It’s difficult to tell for certain from Google maps).

4. Assuming we classify it as a “main street”, what does that say? That for that one street our corresponding “blue” area would shrink. If you cut out a little section of the map, as TL did, you could even show an area with no blue in it.

5. Does that mean n=0 for the whole of Iraq based on the Lancet sampling scheme? Clearly not. Tim misleads a little here (and when he adds his second “main” street, which he doesn’t show, he misleads a lot with regard to his n=0 claim).

6. Tim’s map is fundamentally misleading (see #116).

29. #29 Kevin Donoghue
February 23, 2009

It was hardly Tim’s aim to show that “n=0 for the whole of Iraq based on the Lancet sampling scheme.” He does not claim that in his post. What he does show is that the map presented by Johnson et al. is wildly misleading. The point he is making is clear from his title: the JPR paper is badly flawed.

30. #30 Robert Shone
February 23, 2009

What Tim actually says is:

Make just these two corrections to their map and the unsampled area is 0. In their model, that means n=0 and there is no main street bias.

That’s totally misleading. The unsampled area is not 0 in “their map” – it’s 0 in Tim’s tiny portion of their map (and it would be even tinier if we dismissed Tim’s second correction).

31. #31 Aly
February 23, 2009

Robert S et al.

Surely the real question is not whether n=0 or not, but whether a reasonable selection of main streets will lead to a bias that is way outside the confidence levels (i.e. the estimated accumulation of inaccuracy) that the Lancet team calculated?

In this case all this talk of “misleading” because n may or may not be actually zero is a complete non-starter, and pretty much irrelevant.

The point that I see here is that with any reasonable definition of what main streets are (and by the way, someone with any kind of local knowledge would not have any trouble at all identifying a main street to start off from, which renders much of the linguist wrangling pretty much irrelevant too) it’s extremely unlikely to come up with a bias anywhere near the size that the Johnson, Spagat, Gourley team come up with.

You and I have discussed this before, and I was one of the early folks who drew up a few tables to analyse the sensitivity of their model to changes in the parameters (in fact I even suggested to you that the authors include such an analysis in their paper – I’m glad they did), and what I found then was that the bias is pretty sensitive to the value of the parameters that are selected.

N doesn’t have to be zero, then. All Tim has to show (and has shown to my mind) is that it’s pretty small for any reasonable understanding of what is a main street. The sensitivity analysis that I have already done shows that the bias drops off rather rapidly to 1 (i.e. only a small likely bias if any at all). Also notice that as we move out of the large cities to smaller ones, the definition of main streets etc tilts the balance away from MSB even further.

And all of the above is giving the MSB paper a huge benefit of the doubt. In fact I am pretty unconvinced (as I was the last time we spoke about this) that their dynamics accurately capture anything close to the reality of either the pattern of violence or the daily movements of Iraqis, thereby rendering it an interesting but purely academic example.

cheers

32. #32 Robert Shone
February 23, 2009

Aly writes:

it’s extremely unlikely to come up with a bias anywhere near the size that the Johnson, Spagat, Gourley team come up with

Unless you know more about how the Lancet sampling procedures worked in reality, most of your statements, like the one I quote above, are nothing more than unsupported assertion or wishful thinking. We’ve been through this at great length in the above thread.

You’re right that n doesn’t have to be zero. However, Tim misleadingly suggests that it is zero for “their map”.

33. #33 Aly
February 23, 2009

Robert

No, my comments are based on the sensitivity analysis of the parameter space (something it seems I thought of before the authors themselves) and Tim’s analysis of the map above.

It’s pretty easy.

The question is how does one have to twist and turn to come up with a sensational bias, and the answer is becoming increasingly obvious that you have to twist and turn quite a lot.

Again, you say:

“However, Tim misleadingly suggests that it is zero for “their map”.”

Again I say this is irrelevant and just a way for you to try and obscure the real question. Even you take Tim’s second main street out, the bias still drops to something near 10-20% or well within the confidence interval in the Lancet paper, thus proving nothing at all.

Cheers

34. #34 LancetStudy
February 23, 2009

Aly wrote: “The real question is … whether a reasonable selection of main streets will lead to a bias that is way outside the confidence levels (i.e. the estimated accumulation of inaccuracy) that the Lancet team calculated”

No Aly, the **real** question is: What on earth did the Iraq survey team actually do? What did Lafta, or Lafta’s team, actually do on each survey day? More correctly…

**What did Lafta’s team members do on each survey day to guarantee that there is no street bias? Apparently Burnham and Gilbert do not know (they have admitted they do not have the details) so how can they possibly make any valid statements dismissing street-biases?**

Not only do none of us know, but we are being told that **Gilbert doesn’t know, and nor does Burnham**. And that is what all the fuss **should** be about in my humble opinion.

Now, you might (and have already) hazard a guess at what Lafta’s team did — and so might Tim et al. And so might the rest of us, including Gilbert and Burnham.

But here is the problem:

**Did Gilbert and Burnham put their name to a survey paper, as principal authors, where they did not know the exact survey methodology used?**

Whatever the rest of us on this thread are guilty of (e.g overstatements, moments of emotion etc.) none of the rest of us did that. So, therein lies your problem (as an L2 defender) and their problem as L2 authors. This is undoubtedly why they are being investigated, again in my humble opinion. The rest of this MSB topic is simply various academics attempts to make sense of little (and seemingly contradictory) information. Don’t like it? Fine. Opinions are healthy, of course when backed up by a quantitative argument.

But how come you don’t bother to ask yourself the questions in bold above?? I’ll repeat them:

Question 1: **What did Lafta’s team members do on each survey day to guarantee that there is no street bias? Apparently Burnham and Gilbert do not know (they have admitted they do not have the details) so how can they possibly make any valid statements dismissing street-biases?**

Question 2: **Did Gilbert and Burnham put their name to a survey paper, as principal authors, where they did not know the exact survey methodology used?**

35. #35 LancetStudy
February 23, 2009

OK Tim, you are a computer scientist: Give us all a working algorithm that can be implemented in a day, for the survey team, which guarantees R=1 plus/minus 20 percent, *but* also taking into account all ‘ifs’ and ‘buts’ related to adding in nearby streets etc. etc. (i.e. must be consistent with everything Burnham/Gilbert claims is there).

Oh, and write it in a way such that when translated from English it does not create problems, inconsistencies, ambiguities in interpretation.
Oh, and make it fit on one page such that it can be memorized.
Oh, and make it very simple so that it can be implemented without potential error.
… and I nearly forgot, make it work for every town, city etc. that is sampled throughout the country.

An interesting homework for you. We’re all waiting….

(Of course, even in the infinitessimally remote chance that you come up with one and somehow prove in principle that it works, you have no idea if it is *the* one that was used….. Probably isn’t ;-))

[BTW, there is no suggestion here that Lafta and his team did not do the best job they could. I am sure they did, and that they did something truly heroic. The issue is entirely different: If Burnham/Gilbert do not know the scheme used *exactly*, how can they make any claims about street bias being unimportant?]

36. #36 Tim Lambert
February 23, 2009

LancetStudy I followed the scheme as described in paper on the map that the authors presented when arguing that n=10 was plausible and got n=0. Yes there are ambiguities in deciding what happens at complex intersections but they don’t make much difference since there are usually multiple ways to get to a particular house. Even Robert Shone can’t seem to get past n=0.2 on that map.

No-one is disputing that the L1 scheme using GPS gives a better sample, but this makes little difference. In order to conceal this the MSB people present a misleading map that they don’t include in their paper but merely reference.

37. #37 LancetStudy
February 23, 2009

Tim, now you really are trying to be misleading. You say “I followed the scheme as described in paper on the map that the authors presented when arguing that n=10 was plausible ”

Scheme? The MSB authors are *not* presenting a *scheme*, … that is what L2 should have done. They are the samplers, after all. But they didn’t — so, I am afraid, MSB authors are within their rights to present what they see as plausible.

The JPR-related maps are not meant to be taken literally to the street since **nobody** knows what the actual streets are. They are just plausibility arguments — and reasonable ones at that in my opinion.

I wonder what Lafta would say, if faced with the Google map? Probably not a lot, since maybe he didn’t have a map. We just don’t know….

So sorry, my homework assignment for you still stands. Instead of saying what didn’t happen, tell us a plausible argument of what did happen. (PS Have you checked the danger-element on these maps, such that scheme needed to be adapted on-site as suggested by Burnham/Gilbert? No, unfair question. Impossible to know)

38. #38 Tim Lambert
February 23, 2009

Again, if you want to argue that n=10 is plausible you need to present a map with n=10. You haven’t.

39. #39 Robert Shone
February 23, 2009

Tim Lambert writes:

Even Robert Shone can’t seem to get past n=0.2 on that map.

Misleading again, Tim – tut tut. By “that map”, you mean the tiny portion of it which suits your particular needs.

Try working out the value of n for the whole map, keeping your first additional main street, but forgetting the second one (for reasons I gave in #116).

It’ll be less than 10, obviously, but much higher than 0.1 or 0.2.

40. #40 LancetStudy
February 23, 2009

Tim writes: “Again, if you want to argue that n=10 is plausible you need to present a map with n=10. You haven’t.”

I haven’t, since I haven’t tried. But the JPR team did.

Since you define main street in particular way, you get your n. They define it in their way, estimate n=10 (which follows pretty much from the figure at the top of this thread by the way). But the big, big question is:

HOW DID LAFTA DEFINE IT TIM?

Get it? How… did… Lafta… define… it?

41. #41 LancetStudy
February 23, 2009

By the way, Tim, you have not checked that your ‘main street scheme’ is in principle feasible in terms of coverage for a survey team in the allotted time. To make your scheme plausible, you need to show that the extra Si that is introduced (as compared to the JPR choice) across all the survey areas, will still enable the team to make it round a random sampling of these Si’s within the allotted time. Selecting what would be ‘nice’ for an Si, and then implementing it as a survey team in the allotted time, are two very different things. I think this is something JPR had in mind when thinking about the practicalities of the whole ‘main street’ scheme of L2 — and is clearly(?) what US L2 and Lafta must surely have thought through (or not??). Saying that large areas are in Si, and then not actually sampling them, is equivalent of course to a Si which is small and biased.

Anyway, I notice you haven’t answered my two questions (maybe because you are doing that homework I set you? Well, please drop it one minute, and just answer this:…):

Question 1: What did Lafta’s team members do on each survey day to guarantee that there is no street bias? Apparently Burnham and Gilbert do not know (they have admitted they do not have the details) so how can they possibly make any valid statements dismissing street-biases?

Question 2: Did Gilbert and Burnham put their name to a survey paper, as principal authors, where they did not know the exact survey methodology used?

You can leave Question 1 to later, just answer ‘Yes’ or ‘No’ to Question 2. Please. Now….
Lee, Jody, Sod, Kevin D., anyone???

42. #42 LancetStudy
February 23, 2009

Another quick question for Tim, …..sort of a recap of the same old obvious, obvious question:

Take the map at the top that you pulled from the MSB site. Estimate n given *their* choice of main street. Though not cherry picked for any particular value, n=10 is certainly plausible.

You, as you have said, would have defined main streets differently, and got a smaller n.

What did Lafta do? (a), (b) or some unknown (to us) alternative (c)?

Answer: Not (a) in your opinion. So was it exactly (b)? And your proof that he did is…?. More likely (c), right Tim?

[PS. Thanks Tim, this thread is definitely fun… I was told it would be, and it is….]

43. #43 LancetStudy
February 23, 2009

Time wrote: “LancetStudy I followed the scheme as described in paper on the map that the authors presented when arguing that n=10 was plausible and got n=0. ”

(Just saw this…)

Drivel, Tim. Pure drivel.
Got to hone those experimental skills. Computer science won’t do that for you mate…

44. #44 Lee
February 23, 2009

One more fucking time, LancetStudy, and all your various sockpuppets.

Lafta and the rest of the authors defined main streets and secondary streets such that every house in Iraq had an approximately equal chance of being surveyed.
They said so. That has been quoted to you. I’ve repeated it several times.

Only if one ignores this statement (or acknowledge it and assume Lancet2 to be either massively incompetent or fraudulent), and only if one ALSO goes out of one’s way to exclude obvious main streets from main street category, and ALSO does so in a way that is clearly designed to exclude massive quantities of homes from the survey – only if one does all that, can one get a value for N that has any appreciable impact on the Lancet2 analysis.

It appears that the authors together designed a strategy. Lafta implemented the strategy. Ethical confidentiality makes it unethical to release all elements of the designed strategy. Ethical confidentiality makes it unethical to release any elements of the implementation.

Such ethical constraints are unfortunate reality when one is doing survey work that can bring a risk to the people being surveyed. They also open the authors up to completely unsupported and ludicrous charges such as the ones you dismal f***** are flogging to death here – because you all know that they cant defend themselves, because they are ethically constrained to not release the information you ratf***** are demanding.

It is despicable stuff you are doing – and of a kind with the sock puppetry y’all are engaging.

45. #45 Lee
February 23, 2009

and to address yet another of LancetStudy (et al)’s scurrilous insinuations-but-not-actual-accusations.

I can get off the freeway and drive into a town of 5,000 people, and in 5 minutes identify and understand the layout of every main street in the town – with main street defined as any street with a reasonable probability of some kind of commercial activity on it. I know I can do it, because I do it routinely when trying to find those taco trucks I mentioned above – which are often parked on obscure but clearly commercially-identified ‘main streets.’

I can do the same thing in 15-20 minutes in a town of 25-35k people. I know this because Ive done the experiment.

You on the other hand, LancetStudy (et al) are talking out your ass on this, and using your ass-wind to insinuate fraud or incompetence – and again, it is despicable behavior on your part.

46. #46 David Kane
February 23, 2009

As much fun as this thread is, we now know for a fact that Burnham/Lambert have been lying for more than two years about whether or not individual data was collected. (It was, they said it wasn’t.) Given that, how can Tim (or any other Lancet supporter) rely on their description of the sampling plan?

Just curious.

By the way, Lee huffs about “insinuate fraud.” We know that there was fraud. Hopkins just told us. The only remaining disagreement is over the extent of the fraud.

47. #47 LancetStudy
February 23, 2009

The expletives are out, so I am off this thread until someone provides a proper answer to my two questions (see above in this thread).

Until then, I just cannot resist replying to Lee’s comment:

(1) “Lafta and the rest of the authors defined main streets and secondary streets such that every house in Iraq had an approximately equal chance of being surveyed.”

Oh dear, back to square one: This is a *desired* outcome, not a statement of the actual selection probabilities following their choices of the samplable space.

Personally, I would bet a large amount of money that Lafta and his team did indeed do a great job subject to the constraints. And, if properly corrected for systematic bias, the numbers might be very, very useful. I repeat: A little analysis by the US L2 team along the lines of R estimate (or design whatever…) *could have* made his estimates really valuable. But they didn’t. Or if they did, they are not telling how. I wish they would, since I for one do not see how it could be done accurately without knowing the main street selection that he used.

(2) Lee, I actually have something favourable to say about your taco truck scheme. At least it is a scheme. In other words, if we all knew the schedule of the taco trucks, and the day of the survey, we could then simulate what houses would have been sampled in a given realisation of the sampling in a given (portion of a) city. Then we could repeat the simulated sampling, and we could build up a profile of probability of being picked for each house. Not necessarily related to main streets (whatever they are) but at least well-defined, and repeatable.
To the extent that taco tricks move around randomly, we might then (on a good day) even remove bias tied to particular streets. And your scheme would translate into other languages pretty well as well. So well done.

However, it is irrelevant. It is the *actual* L2 scheme that we all need. You know, the one the survey team actually used. Actually Used Every Day In Iraq…..

48. #48 LancetStudy
February 23, 2009

I just read David Kane’s message about the censure and then Tim’s other page. Thank you for letting us know, David and Tim.
So now we know that at least that data existed post-survey. I do feel sorry for Lafta, he must feel let down. I don’t know anything about the Burnham/Gilbert split, so I have no comment on any of that.

But what this probably means is that in principle some independent, assigned person (overviewed, of course) could reconstruct where the sampling actually occurred. With some funding etc. that would probably be useful. Conflict mortality estimates through epidemiology could be a very powerful scientific method, if properly applied with correct estimates of street-bias etc. It would be a shame not to push it forward properly. In fact, we probably all agree on this…. That way we would all really learn what MSB is or isn’t about in practice, and a lot of things we don’t even imagine possibly. Maybe MSB gets cancelled with some other effect? Or enhanced? Or essentially zero in certain places and large in others? etc.?
Who would fund it though, who would do it, and do enough people want it to happen. MSB or not MSB…that is the question.

49. #49 LancetStudy
February 23, 2009

My apologies: I have made the typo Burnham/Gilbert in several places instead of Burnham/Roberts….
Burnham/Gilbert —> Burnham/Roberts under all my entries

50. #50 Lee
February 23, 2009

LancetStudy(and various sock puppets) said:
“This is a desired outcome”

Bullshit. “There is no polite way to say it.”
They didn’t say it was their desired outcome. They said it was the criterion upon which they designed their scheme. Those ar ehugely different statements, and the fact that you keep misrepresenting it is telling. Especially while deflecting nearly every substantive point in my post.

Also LancetStudy et al:
“could reconstruct where the sampling actually occurred. With some funding etc. that would probably be useful.” and “It would be a shame not to push it forward properly. In fact, we probably all agree on this.”

So, Roberts was involved in a serious ethical mistake, and LancetStudy here proposes to amplify and extend that ethical mistake into an intentional greater violation of ethical constraints.

How -ethical- of him.

51. #51 dsquared
February 24, 2009

Given the admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists?

Surely the answer to this is the simple one that Tim has given:

“Because main street bias only exists in a small number of likely unrepresentative corner cases of sampling schemes, like the one that Spagat, Johnson & Gourley picked – for most definitions of “a main street” it is small or nonexistent”.

In the end, if n=0, this means by definition that Si covers all of the governerate. So the survey team would have to be able to go everywhere.

Since they did not go everywhere, by their own admission for safety reasons and need for speed, we are back in the same position

This is an absolute 100% cast-iron howler, which frankly calls “Lancet Study”‘s credibility into question on every single other point. Having every street included in the potential sample space (n=0) obviously does not mean that the survey team would need to “go everywhere”. The survey team would sample exactly as many clusters, whatever the size of the sample space. If you don’t understand this, you really don’t know what you’re talking about. If “Lancet Study” is actually one of the Spagat et al authors, then I profoundly hope this was a silly mistake born out of lateness and anger, because otherwise it’s really embarrassing.

The consistent inability/refusal of the Spagat et al party to distinguish between “people killed on main streets” and “people kiled whose houses are on main streets” when making their arguments about the “topology of violence” is also embarrassing if it’s an inability and irritating if it’s a refusal.

52. #52 sod
February 24, 2009

As much fun as this thread is, we now know for a fact that Burnham/Lambert have been lying for more than two years about whether or not individual data was collected.

ah, David the defender.

But what this probably means is that in principle some independent, assigned person (overviewed, of course) could reconstruct where the sampling actually occurred.

again, for a person with access to the authors of the Spagat paper, your proposals are pretty illiterate.

Only if one ignores this statement (or acknowledge it and assume Lancet2 to be either massively incompetent or fraudulent), and only if one ALSO goes out of one’s way to exclude obvious main streets from main street category, and ALSO does so in a way that is clearly designed to exclude massive quantities of homes from the survey – only if one does all that, can one get a value for N that has any appreciable impact on the Lancet2 analysis.

very good sum up, Lee.

i expect the Spagat supporters to prefer a discussion about the Burnham subject. it will allow them to continue to ignore, how adding a single street completely changes the outcome of the Spagat paper for an area. and as i have said repeatedly and as Lee supported with his “taco example”, the problem is much smaller in smaller towns anyway.

53. #53 Aly
February 24, 2009

LancetStudy

this is pretty much my last post on this, as we have gone round in circles about this for a while. I have certainly discussed this in the past ad nauseum and have no appetite to restart the whole discussion.

You say:

What did Lafta’s team members do on each survey day to guarantee that there is no street bias? Apparently
Burnham and Gilbert do not know (they have admitted they do not have the details) so how can they possibly
make any valid statements dismissing street-biases?

With respect I disagree. You are including a number of hidden assumptions in this question and I disagree with almost all of them. A couple of obvious assumptions are:

1. There would be street bias (I presume you mean MSB) be default. I disagree. There might or might not be, and this remains to be shown. As interesting as the paper by Johnson, Spagat et al. is, it is not proof that the thing called MSB exists in Iraq, i.e. there is no evidence that given the actual patterns of violence in Iraq, one has a greater chance of being killed by violence if one lives off a street that is close to a main street.

2. That Lafta et al. did not do what Burnham said they did, perhaps with some variation here and there. Again, I disagree. I think there is no evidence to suggest that Lafta and co. did more or less follow the instructions from Burnham, and unless there is some strong evidence to the contrary I see no reason to question that.

A question (which is essentially a rephrasing of the one I asked Robert Shone, and to which he did not really bother to respond) that one might then ask is:

If Lafta and his team deviated now and then from the methodology, and assuming that some streets were mis-characterised (whatever that might actually mean) as main streets, or not main streets, how likely is it, assuming that MSB actually exists that a large bias was introduced?

I submit that based on the sensitivity analysis it is quite unlikely, even in the best case for the MSB team, that is assuming that their model actually captures something of the reality of the situation in Iraq. By the way, and for complete disclosure, I don’t accept that the MSB formula is even close to capturing the reality of the violence in Iraq, but I am assuming it does for the sake of argument.

Cheers

54. #54 Robert Shone
February 24, 2009

Aly, to address your point no. 1: Burnham/Roberts have specifically stated that they made efforts to “to reduce the selection bias that more busy streets would have.” (see #13)

In other words, the “hidden assumption” you refer to (that there is a bias that needs to be reduced) is already accepted by Burnham/Roberts.

55. #55 LancetStudy
February 24, 2009

“… The survey team would sample exactly as many clusters, whatever the size of the sample space…”

Yes, yes, yes.. of course. Goodness! Not born out of lateness or anger, pure laziness in description. OK, if you want to be pedantic: Yes I agree, but after many realizations they would have found themselves at some stage penetrating deep away from busy streets. Like drawing numbers from a distribution that in principle goes to infinity, they would in practice draw houses tucked away deep in a neighborhood…. if Si is very large.

So when I say “they would have to go everywhere’, then it means that after many realizations of their scheme, eventually they would have to penetrate deep to such a house. Hence ‘everywhere’ as opposed to just mainly next to main streets. Get it?
Based on their scheme of selection, this simply would never happen. Please lets not get into a pedantic debate about epsilon –> 0 in continuous probability distributions as opposed to likelihoods in finite realizations. I don’t mind, but it is boring and obvious and I certainly ain’t going to type out a whole lecture course for you.

So if Si is large, eventually after many realizations they would probe ‘everywhere’ (i.e. every type of house location).

So in practical terms of running a survey, do this throughout Iraq chosen clusters and you are out of time (and into areas they seemed to want to avoid for reasons X).

Robert Shone re-iterates a key point: “the “hidden assumption” you refer to (that there is a bias that needs to be reduced) is already accepted by Burnham/Roberts”. Excellent point.

So L2 defenders, please quantify what exactly is the ‘magic ingredient’ that US L2 seem to have added to the algorithm in an ad hoc way?

(PS Tim doing the homework? And what about the responses to my two questions?? SIlence is golden guys……)

56. #56 LancetStudy
February 24, 2009

“..Roberts was involved in a serious ethical mistake, and LancetStudy here proposes to amplify and extend that ethical mistake into an intentional greater violation of ethical constraints….”

Nonsense.

I am sure that conflict epidemiology could be a quantitative scientific method, if (and only if) street-bias is quantified a priori for the particular street-selection protocol. Simple as that.

57. #57 Jody Aberdein
February 24, 2009

…homework…

How is the QGIS analysis goig on your side?

58. #58 LancetStudy
February 24, 2009

Jody write: “..How is the QGIS analysis goig on your side?
…”

59. #59 Aly
February 24, 2009

Robert and LS

Let me make my points a little clearer.

1. You are both assuming that the mechanism by which the streets were chosen (as described by Burnham et al.) will lead to a street bias by default that needs to be further considered and reduced.

I disagree. So far I have seen no evidence this is the case and I think Tim’s analysis of the maps above shows this pretty conclusively.

2. How large (in the case that I am wrong with point 1 above) is the street bias should it prove to exist and is characterised by the formula in the MSB paper likely to be?

By looking at the sensitivity of the model’s parameter space, it seems that it will likely be small and hence not change the confidence intervals of the original study.

So far you have contributed nothing new to what we already could see from the L2 paper.

3. How likely is it that the MSB model accurately captures the dynamics of the actual violence in Iraq and is actually more than a meaningless toy example?

Until I see some attempt by the authors of the MSB paper to show, somehow or another that their model accurately captures some of what we are seeing on the ground, through calibration or analysis of patterns of violence or pretty much anything other than what amounts to accusatory hand waving, whilst cherry-picking parameter values that lead to sensational bias estimates, I consider their model to be an interesting but useless toy. Akin to the legendary “spherical cow” in my undergraduate physics lessons.

I hope this is clearer.

Cheers

60. #60 LancetStudy
February 24, 2009

Data recorded by Lafta is X. It then gets passed through a signal processing box Y by US L2 team, producing result Z (i.e. 650,000 plus/minus…). Hence Lancet article.

Process Y accounts for MSB, deduces R is near 1, and discards any possible effects of MSB.

What is Y?

61. #61 LancetStudy
February 24, 2009

Spherical cow models can be useful for highlighting issues. They will never explain where to hang a bell on a cow, the colour of a cow’s eyes etc. But they can help give answers to other questions (e.g. rate of heat loss etc.). And at the very least they highlight issues, provide a basis for discussion, and pinpoint some parameters (not their values necessarily, but the embodiment of the parameter’s meaning). And I guess some people just don’t like those kind of models. Apollo 11 landed on the moon with spherical cow arguments. 747’s fly with similar arguments. Etc. etc.

Actually, personally I would have classified JPR as one step beyond mean-field theory (i.e. first level beyond an assumption of uniformity). That’s all. No more, and no less. Certainly limited, but not ‘flawed’. Just one level beyond the uniformity assumption which (unless Y is a magical algorithm) is what L2 implicitly assumed or hoped.

62. #62 Bruce Sharp
February 24, 2009

Robert in #154: “the ‘hidden assumption’ you refer to (that there is a bias that needs to be reduced) is already accepted by Burnham/Roberts.”

I alluded to this much earlier when I pointed out that one of the goals of the study design was to eliminate geographic bias. Jody also pointed it out, noting that surveys must be designed in such a way as to account for “design effect.”

It is your assumption, Robert, that surveying on main streets would increase the death toll. As sod and others have pointed out repeatedly, that may or may not be the case. Burnham et al made no such assumptions about how it would bias the results. You are implying that their results are probably inaccurate, because they attempted to design a scheme that would make them accurate. Do you see the problem with that argument?

63. #63 LancetStudy
February 24, 2009

Bruce wrote: ” surveys must be designed in such a way as to account for “design effect.””

Yes. Design is statement about what is planned to happen: W

Then there is what actually happened: X

This is fine *if* there is then a clever algorithm Y on the US L2 side, which can discount ways in which X might differ from W (e.g. possible street bias in X).
So Y must be able to assess it, and quantify any difference approximately, before being able to discard it to produce inference Z.

So what is Y?

64. #64 Robert Shone
February 24, 2009

Bruce Sharp writes:

You are implying that their results are probably inaccurate, because they attempted to design a scheme that would make them accurate.

No. I’m saying they claimed they used (unpublished) procedures which reduced MSB to zero, and should disclose the details – regardless of my (or any one else’s) speculation over the value of q.

65. #65 Bruce Sharp
February 24, 2009

Robert, if the selection of main streets was sensible, then the amount of space that cannot be surveyed is neglible, and q — which you cannot quantify anyway — doesn’t even matter.

As far as I can tell, all the fuss over “MSB” can be reduced to: “The L2 team made poor main street choices, perhaps because they were clueless, or perhaps because they are dishonest. Here is a long paper with some equations where you can insert arbitrary parameters to change the L2 estimates to a number that you’ll like better.”

Everything else is window dressing.

66. #66 dsquared
February 24, 2009

Based on their scheme of selection, this simply would never happen

No, as Tim has explained. Based on Spagat et al’s proposed scheme of selection, this would never happen. Based on other selection schemes very similar to Spagat et al’s (like the one Tim proposes), it would happen. In other words, in all except a small number of seemingly unprepresentative cases, it would happen. This is the point of the map examples above. They show that the JHU team would have had to have been very unlucky to have happened upon a scheme that would have excluded large proportions of houses from the sample space.

So if Si is large, eventually after many realizations they would probe ‘everywhere’ (i.e. every type of house location).

So in practical terms of running a survey, do this throughout Iraq chosen clusters and you are out of time (and into areas they seemed to want to avoid for reasons X).

This is not very comprehensible. The first paragraph is simply the statement (admission?) that if a sensible scheme is devised, all houses in Iraq are in the sample space. “Many realisations” is irrelevant here – there was only one survey carried out.

The second paragraph I don’t understand at all. “[D]o this throughout Iraq chosen clusters and you are out of time” just describes the fact that for obvious resource reasons, the JHU team carried out one survey, rather than thousands. “[A]nd into areas they seemed to want to avoid for reasons X” just seems to assert that the JHU team would have intentionally designed a sample scheme so as to avoid houses far from main streets. Which a) they didn’t, and b) the idea that houses a long way from main streets were more dangerous to sample than houses next to main streets, is obviously inconsistent with the central proposition of the Spagat et al paper, which is that houses a long way from main streets are less dangerous than houses next to main streets.

67. #67 Lee
February 24, 2009

Bruce at 165:

You left out a critical part in your summary. I’ll add it here in bold:

Based on no evidence whatsoever, we are certain that… “The L2 team made poor main street choices, perhaps because they were clueless, or perhaps because they are dishonest. Here is a long paper with some equations where you can insert arbitrary parameters to change the L2 estimates to a number that you’ll like better.”

68. #68 Aly
February 24, 2009

LancetStudy

Spherical cow models can be useful for highlighting issues. [snip] at the very least they highlight issues, provide a basis for discussion, and pinpoint some parameters [snip]

Agreed. They can be useful, but not when they are simply used as Spagat et al. did to cherry pick a particular set of parameters and come up with a sensational bias. In that case they are only useful to piggyback off an actual good and well known study to get extra publishing points for their respective university departments. They are useful to try and talk about what a reasonable set of parameters might mean, and how likely these are. This is not what has been done so far, more’s the pity.

I guess some people just don’t like those kind of models. Apollo 11 landed on the moon with spherical cow arguments. 747’s fly with similar arguments. Etc. etc.

This is rubbish and you know it. To compare the MSB model with the same models that fly aircraft and got Apollo 11 to the moon is fantasy.

Aircraft manufacturers and designers run thousands of simulations and tests and sensitivity analyses (plural) and callibrations and and and and…

This is how they can be sure that their models represent something more than nonsense combined with hot air. This is what separates an actual model from a meaningless toy (MSB).

If the authors of this paper were serious about this (and I hope you are not one) then they need to do some kind of work to show that their model callibrates to reality somehow. There are a hundred different ways they could do this, and they have done precisely nothing except draw random marks on random maps.

Poor effort.

69. #69 LancetStudy
February 24, 2009

Aly wrote: “…they have done precisely nothing except draw random marks on random maps….”

Oh dear, Aly. Like your friends, I see that your logic is full of the waste that a spherical cow emits……

70. #70 LancetStudy
February 24, 2009

…OK Aly: I want to be kind to you: We now know that you don’t really have much of an idea what a model is. Maybe you have never done any research, never been to graduate school, not even some kind of research project?? Anyway, for your education/information, MSB is typical of a first-approximation model in a problem.

But the good news is that you can redeem yourself, here and now in 10 minutes. Just answer a simple question, which of course you know the answer to (otherwise, how could you be so sure of yourself?). Here it is:

Data recorded by Lafta is X. It then gets passed through a signal processing box Y by US L2 team, producing result Z (i.e. 650,000 plus/minus…). But somehow process Y accounts for MSB, allows the deduction that there is no street bias in X, and hence allows L2 to discard any possible effects of street bias. So what on earth is this marvellous Y data procedure?

71. #71 Lee
February 24, 2009

Lancet Study ( and your various sock puppets):

You keep asking what “marvellous Y data procedure” … “allows the deduction that there is no street bias.”

That is a deeply disengenious and dishonest question. The fact is that THERE IS NO EVIDENCE THAT THERE IS ANY STREET BIAS!!!!!

You guys are making this shit up.

N does not equal 10. Or anything close to it. As I’ve aid over and over now – the lancet authors stated as a design criteria (not a desired goal, but a criteria upon which they based their sampling design) that every house in Iraq should, to use your language, be in the sampling frame.

Only if one ignores this statement (or acknowledge it and assume Lancet2 to be either massively incompetent or fraudulent), and only if one ALSO goes out of one’s way to exclude obvious main streets from main street category, and ALSO does so in a way that is clearly designed to exclude massive quantities of homes from the survey – only if one does all that, can one get a value for N that has any appreciable impact on the Lancet2 analysis.

Stop being despicable.

72. #72 LancetStudy
February 25, 2009

To Tim and friends. I have a new question:

**Data recorded by Lafta is X. It then gets passed through a signal processing box Y by US L2 team, producing result Z (i.e. 650,000 plus/minus…). But somehow, this mysterious process Y allows the deduction that there is no possible street bias in X, by quantifying it as negligible using some unknown quantitative model, and hence allows L2 to ignore even mentioning the possibility of street bias in the Lancet article. So what on earth is this miraculous Y data procedure?**

Oops, it is the same question as always. You never answer it, and yet it addresses what makes L2 valid or not scientifically. So maybe you should quit snooping around IP addresses and address this apparently simple question. Or will the Lancet erratum to do so?

73. #73 LancetStudy
February 25, 2009

Lee claims there is no evidence for street bias. There is certainly no evidence that there is elephant bias, or taco bias etc… But that is because elephants and tacos were not involved either in the design, survey implementation, or the violence itself, as far as I know.

However, streets were involved in all 3. Jody tells us that the design would have incorporated any STREET bias effect (in some way that she does not explain, or know..), the survey involved people walking on a STREET to houses on one STREET or another, and the violence occurred on a STREET. And the common word is…? Yes, STREET.

So the doubt about street bias is reasonable, and the issue should be addressed by the scientists which dismiss it. Just saying ‘no, it doesn’t exist’ is not a scientific argument.

74. #74 Jody Aberdein
February 25, 2009

…possibility of street bias…

So presumably as I’m not an epidemiologist I am incorrect when I read:

‘The SE for mortality rates were calculated with robust variance estimation that took into account the correlation between rates of death within the same cluster over time. The log-linear regression model assumed that the variation in mortality rates across clusters is proportional to the average mortality rate; to assess the effect of this assumption we also obtained non-parametric CI’s by use of bootstrapping. As an additional sensitivity analysis we assessed the effect of differences across clusters by extending models to allow the baseline mortality rate to vary by cluster’.

As being some statement of the efforts spent trying to discern systematic error?

So presumably the difference between rural, small town and city clusters generated by ‘main street bias’ is not detectable in this way, and further ‘main street bias’ leads to a level of intra-cluster correlation that is also undetectable by the above process?

75. #75 LancetStudy
February 25, 2009

Jody writes: “so presumably as I’m not an epidemiologist I am incorrect when I read:…”

1. Not incorrect when you read it. But incorrect to link that phrase to streets. It talks about sampling on the scale of clusters, not the streets within a cluster.

2. “.. street bias’ leads to a level of intra-cluster correlation that is also undetectable by the above process…”

Yes, correct.

76. #76 LancetStudy
February 25, 2009

**What quantitative analysis did Burnham/Roberts do, either post-survey or pre-survey, to correct for possible street bias during the survey?**

77. #77 LancetStudy
February 25, 2009

Again….

What quantitative analysis did Burnham/Roberts do, either post-survey or pre-survey, to correct for possible street bias during the survey?

street, **street**, _street_, STREET….

78. #78 dsquared
February 25, 2009

Surely the answer to Lancet Sock’s question is:

“The inherent unlikelihood of any such bias existing in the first place”.

79. #79 LancetStudy
February 25, 2009

dsquared wrote: “…inherent unlikelihood …”

What???

80. #80 Lee
February 25, 2009

One again, LancetStudy (and your army of mousy sock puppets) – bullcrap!

“Only if one ignores this statement (or acknowledge it and assume Lancet2 to be either massively incompetent or fraudulent), and only if one ALSO goes out of one’s way to exclude obvious main streets from main street category, and ALSO does so in a way that is clearly designed to exclude massive quantities of homes from the survey – only if one does all that, can one get a value for N that has any appreciable impact on the Lancet2 analysis.”

See that? Ive posted it several times now in response to your spammed-all-over straw man question.

If N has a value much smaller than 10, then street bias does not alter the Lancet2 conclusions in any appreciable way.
This is true also of several other of their parameters, which have been adressed in other threads.

Lambert’s post that started this thread is an answer to your oft-spammed question. Lancet 2’s authors statement that a sampling design parameter was that every house in Iraq be in the sample frame, ADDRESSES YOUR F****** QUESTION!!!!!!!!!!!!

81. #81 dsquared
February 25, 2009

What???

Are you slow or something, LancetSock? As has been shown above, main street bias only exists in a small number of seemingly cherry-picked and for the most part obviously incorrect schemes for identifying “main streets”, and even then only in large towns. As Tim showed above, even in the best-case example chosen by Spagat et al. to illustrate their point, it only existed thanks to a really quite perverse decision to fail to identify a particular road as a main street. From this I conclude that the vast majority of road-identification schemes which a competent researcher on the ground would choose, would not lead to material street bias. Hence, the fact that it is very unlikely that the sampling space was distorted by main street bias, is good grounds for assuming that the sample is not distorted by main street bias.

82. #82 LancetStudy
February 25, 2009

You guys get personal very quickly don’t you. So be it…

Since Lee likes cross-postings, here is mine:
Lee writes: “If N has a value much smaller than 10, then street bias does not alter the Lancet2 conclusions in any appreciable way…”

Putting Tim’s incorrect analysis to the side for one moment, the question is: Why didn’t Burnham/Roberts do a street-bias analysis? Where did they do it? Do they reach the same conclusions? If so, how did they manage to do that prior to JPR, if they didn’t have the JPR formula? Did they derive the JPR formula and plug in numbers? If not, what formula? What are the assumptions? What numbers did they plug in? Did they use a map? Which map? Which streets did they pick as main streets? The streets that Lafta generally picked?

Questions, questions…. ‘irrelevant’ questions? No.

Lee, my friend, invite dquared to grab a taco from one of your main-street-defining-taco-trucks, and read the following out loud, together, very, very slowly while munching…

The survey involved people walking on a STREET to houses on one STREET or another, and the violence occurred on a STREET. So how, where and when did Burnham/Roberts manage to quantify and discard any street-bias effect?

83. #83 LancetStudy
February 25, 2009

… and that concludes my side of the proceedings.

**Outcomes**

1. possibility of street bias is reasonable, and given the lack of details from L2 and apparent communication issues within L2, sounding more likely to me every day

2. JPR model reasonable. A good first-approximation model.

3. JPR estimates are plausible, reasonable even. Above all, their invite to readers (in the JPR abstract) to suggest their own values is honourable and honest

4. L2 does not contain a quantitative consideration of street-bias in the paper

5. The opponents of JPR, actually use the JPR model framework to discuss street bias through its parameters. A classic proof of the usefulness of a model!

6. Tim’s title is incorrect, … and may be seen as slanderous

84. #84 dsquared
February 25, 2009

You guys get personal very quickly don’t you

Sock puppets aren’t people. If you want to get treated like a person, be a person.

Why didn’t Burnham/Roberts do a street-bias analysis?

Because there’s no such thing as a “street-bias analysis”. There is no such thing, because, absent cherry-picked cases like the one in the badly flawed JPR paper, the “contiguous houses down secondary street” method samples the space effectively.

In answer to the six new silly assertions which have sprung up as a result of my answering one silly question:

1. No; it has been established that only rare and cherry-picked street choices would result in measurable truncation of the sample space, which is in itself a long leap from “Bias”, as nobody has established that any such censoring was informative.

2. No, clearly a bad approximation, as it presents only a single set of parameters which are not robust. As a model of “bias”, even worse because its discussion of whether the hypothetical censoring was informative is nugatory.

3. No, the suggested values are unreasonable (as demonstrated above). Since nearly all peturbations of the street identification code give parameter values close to zero, it is neither honourable nor useful to ignore this.

4. Nor does it give consideration of a million and one other almost certainly fictitious phenomena. Not the Lancet’s job to do Spagat et al’s work for them in inventing the hitherto unknown “Main street bias” phenomenon, which remains wholly speculative.

5. Clearly silly – a model whose only use is to prove that the issues it raises are not important, is not a useful model.

6. No it isn’t, and (how it pains me to mention this), if it is true that “LancetStudy” is a pseudonym for one of the authors, they should perhaps look closer to home; Michael Spagat has considerable past history of making slanderous statements about researchers with whom he disagrees, and attracted an open letter of complaint from Human Rights Watch Colombia (IIRC) for doing so.

85. #85 elspi
February 25, 2009

Sock puppet,
Try for just one second to think like a scientist. Suppose JUST for the sake of argument that people who lived on main streets did have a higher death rate. What Tim has shown is that it is inconceivable that all (or even a significant majority) of the clusters would be on the main streets. There is simply no way to rationally choose main streets without getting many of the clusters off the main streets.

These clusters would have very different rates of death than the clusters on the main streets and this would show up in the analysis and as #174 point out
“GAME OVER MAN”. Since this did not happen, there is no MSB. The end.

86. #86 LancetStudy
February 25, 2009

elspi wrote: “..The end..”

Au contraire, it is just the beginning…

87. #87 Lee
February 25, 2009

oh god

LancetStudy :”So how, where and when did Burnham/Roberts manage to quantify and discard any street-bias effect?”
They didn’t. They didn’t have to – BECAUSE THEY FUCKING DESIGNED IT OUT UP FRONT!!!!!!!!!

Get it yet?

The taco truck example was an answer to your dishonest insinuation that the sampling teams didn’t have enough time to identify main streets. Misrepresent much?

They also didn’t quantify and discard possible time of day effects, possible scary interviewer effects, possible ‘nice interviewer I want to please’ effects, possible biased question effects, and on to a multitude of possible effects. They design to minimize all of these – including skewed sampling effects – as in EVERY FUCKING SURVEY EVER TAKEN!!!!

Stop being despicable, LancetSTudy.

88. #88 Jody Aberdein
February 26, 2009

Neat..

Invent a sort of incomplete data question where you cant even know how incomplete the data are, and demand the set is complete before believing the result.

Presumably even if we had the full set of household addresses lets say from a true random sample there would still be some clustering, and you could pull some post hoc ‘ah but look they cluster near this or that, there’s bias’

Anyhow I’m quite interested in how ‘main street bias’ works without creating increased intra over inter cluster correlation given that the clusters were in different places in different sized towns etc.

89. #89 sod
February 26, 2009

They also didn’t quantify and discard possible time of day effects, possible scary interviewer effects, possible ‘nice interviewer I want to please’ effects, possible biased question effects, and on to a multitude of possible effects. They design to minimize all of these – including skewed sampling effects – as in EVERY FUCKING SURVEY EVER TAKEN!!!!

they also did ignore all positive effects on the polling process on mainstreets (Taco truck next to cluster: TTntC-effect, reducing massively the time needed for lunch breaks)

neither did they quantify effects that give a too low deathrate (wiped out or fled households) down to the third digit behind the comma!

90. #90 dsquared
February 26, 2009

Interesting, of course, that the potentially quite serious problem of “Didn’t Visit Anbar Because It Was Too Dangerous Bias” in the UN IFHS survey has attracted so much less attention.

91. #91 bernard J.
February 26, 2009

LancetStudy.

This may appear to be a nit-pick, but to me it speaks volumes…

At #173 you refer to Jody as ‘she’. Given the number of times in this thread that you have referenced Jody I am surprised that you remain ignorant of said person’s identity.

Google Jody’s name, add ‘Dr’ and ‘London’ to the string, and find out why your oversight is rather glaring.

If I was extensively debating with someone who hadn’t done even a cursory amount of research on basic background such as this, I would be very sceptical of this person’s capacity to construct a cogent argument.

Of course, that’s just my bias.

92. #92 sod
February 26, 2009

nteresting, of course, that the potentially quite serious problem of “Didn’t Visit Anbar Because It Was Too Dangerous Bias” in the UN IFHS survey has attracted so much less attention.

i think i have an explanation for this: as they did not go there, they did not have to travel on roads, so there can not be any street bias.

on a more serious note, the IFHS study replaced the numbers for those dangerous regions with numbers calculated from the IBC project. and the IBC take their numbers from newspaper reports.

the serious bias in this approach is, that the number of reports is certainly linked to the number of journalists in Iraq.

and this number gets reduced [during times of high violence](http://www.nytimes.com/2004/10/02/international/middleeast/02media.html)

and even more with the [recent withdrawal of journalists](http://www.brookings.edu/opinions/2008/1015_iraq_media_nessen.aspx)

so let us check some numbers:

he New York Times, Los Angeles Times, and Washington Post—which all continue to staff small Baghdad bureaus—published a total of 858 front-page stories from Iraq in 2003, but only 138 in the first nine months of this year.

ooops, this gives us a JWB (journalist withdrawal bias) of 6.2!!!

93. #93 Kevin Donoghue
February 26, 2009

dsquared: a model whose only use is to prove that the issues it raises are not important, is not a useful model.

True, if you are looking at the matter from a scientific perspective. But from a troll’s point of view, if you can annoy people and get them arguing and lavishing attention on you – even if the attention mostly takes the form of scorn – that’s success.

In the previous thread on this subject a professor of demography wrote a lengthy comment on Burnham et al. and where it may have gone wrong; he didn’t assume that it did of course. There was practically no response (not that I suppose the author particularly wanted responses). A sock puppet, whose sock-puppetry was obvious from the similar quality of writing and reasoning even before Tim confirmed the shared IP address, got umpteen responses.

If a troll was a scientist, the MSB paper is just the sort of paper a troll would want to write. Whether or not this particular troll is one of the authors, it’s easy to see why s/he admires their work.

94. #94 sod
February 26, 2009

i think the use of the Lancet iraq study as the main example of their model is extremely dubious.

if you had a good theoretical model, would you use an example from a paper, that you consider to be completely flawed in many aspects? one which you consider to consists of faked, incomplete and unavailable data? a paper that, according to you, does not accurately describe its methodology? one that forces you to guess all parameters, with no way to check them?

it just doesn t make any sense, unless you consider other motives. (mainly publicity)

95. #95 Bruce Sharp
February 26, 2009

A good test of the model would be to apply it to something like traffic fatalities, or pedestrians injured by cars. We’d expect more accidents on busy streets. If we applied the L2 sampling scheme, would our results match the actual accident rate, as reflected in official statistics? And if they didn’t match, would applying the formulas from the MSB paper give us a correction factor?

My guess is that we’d find that the L2 sampling scheme would result in pretty accurate numbers to begin with, and the utility of an MSB algorithm to adjust those numbers would be negligible.

96. #96 AlyKas
February 26, 2009

Well LancetStudy

Touchy aren’t you? I guess you are one of the MSB authors after all… what a shame.

I’ll leave you to your spherical cow to do what comes naturally. Oh yes, and to speculate about whether or not I have a graduate degree…

cheers, and goodbye