This post is some more notes on a reply to the badly flawed "Main Street Bias" paper.
The authors claim that it is plausible that the Lancet paper's sampling scheme could have missed 91% of the houses in Iraq. (That is, their parameter n, the number of households in the unsampled area divided by the the number in the sampled area could plausibly be 10 or more.) The only support they offer for this is a reference to this analysis of Iraqi maps.
To the right is a detail from their map. The red lines are main streets and the yellow are secondary streets. They assert that the blue areas are not samplable using the Lancet scheme, and yes, the blue area covers 90%+ of households. But there are two things wrong with their map.
Look at this larger scale version of their map:
First have a closer look at their yellow road that they say should not be counted as a main street and compare it with the one they count as a main street. (Scroll around on Google Maps if you want to see more of each street.) It's just as wide and has a similar amount of traffic and a similar number of large buildings. Clearly it should have been classified as a main street and it is not even slightly plausible that someone who was trying to get every house in the sample frame would leave it out.
Second, the random house on the secondary street is only the start point for the cluster, which also includes 39 houses neighbouring the start point. This means that you can sample houses on tertiary streets that are a few houses from the secondary street. I added two obvious main streets to the ones they chose (the one across the bottom of the map above and one that it is off the map above), and redrew the blue areas to taken this factor into account.
What's that? You don't see any blue areas in my map? That's because there aren't any. Make just these two corrections to their map and the unsampled area is 0. In their model, that means n=0 and there is no main street bias.
- Log in to post comments
nice work Tim.
they also prefer to chose Baghdad when talking about "mainstreet bias". choose a random village and use the lancet approach, and you will nearly always find nearly everything covered...
Another thing is that you don't tend to find markets and such on wide main streets cause you can't get goods in and out and customers cannot cross easily.
Tim Lambert writes:
No, they don't assert this. Nobody knows what's samplable using the Lancet scheme, because nobody knows how the Lancet scheme worked in reality (since the details haven't been released).
The MSB authors make it clear that Burnham et al's definition of main street "is too vague to answer these questions definitively". They continue: "For starters, we cannot be sure of what is a major commercial street or avenue. But it is possible to explore some scenarios and we do this below. We encourage everyone to do their own explorations based using (the miraculous) Google Earth."
If that's not clear enough for Tim, they also write the following [my emphasis]:
One can see more clearly from the full map (rather than from the particular corner of it that Tim has chosen to focus on) why the road marked yellow might be considered a cross street - it cuts straight through the residential area, in contrast to the red-marked roads.
I've no idea whether n=10 in the real study. It seems infinitely more plausible than n=0.
why the road marked yellow might be considered a cross street - it cuts straight through the residential area, in contrast to the red-marked roads.
so they assume that only roads not leading through residential areas are "mainstreets", and then conclude some bias of residents not being sampled?
A more liberal rendering of major commercial streets or avenues would enable deeper penetration.
for some weird reason, they didn t provide too much data, on such a "deeper penetration".
and instead went for n=10 in their number works. just by chance?
Incidentally, if you take Tim Lambert's "yellow road" scenario for producing n=0, and apply it to the whole of Iraq, you get interesting (but not remotely credible) implications for how the Lancet sampling scheme "must" have worked.
i am seriously puzzled by the Tikrit map on the Spagat page.
How did they chose those red "mainstreets"
and why is there so much pink shading around blue roads?
There is relevant information from a BBC intervies
especially d
I couldn't make any sense of their Tikrit map. These are the only two maps they offer in support of their n=10 theory.
1) Thanks to Tim for taking the time to do this analysis and start a new thread. This is the way open-minded scientists search for the truth.
2) Tim: Why don't you ask Les Roberts whether your "two corrections" are plausible? After all, surely Roberts knows what the survey teams considered to be "main streets" . . . right?
3) I believe that I have listened to more publicly available presentation by the Lancet authors than almost anyone. I have never heard them give a description that is consistent with a methodology that would lead to a map like the one that Tim describes above. They weren't using Google maps after all! That doesn't mean that Tim's description is false. We just have no idea what they actually did. (Pointers to the contrary are welcome.)
4) I hope this mean that Tim has given up on his "proof by contradiction" that the formula is wrong from here. If so, it would be nice of Tim to confirm that he no longer maintains that there is a mistake in the maths.
DK:
2) I don't need to ask Roberts. He's already answered this.
3) I disagree. I just followed the methodology they described in their paper. Of course they didn't use Google maps, but you can use Google maps to determine what someone at street level would do.
4) No. Their formula is wrong as I demonstrated. I never said that there was a mistake in their maths. Their formula is wrong because their model is wrong. Stay tuned for part 3.
Here is an overview of what we know and what we don't know about the sampling for L2. Summary:
Tim is correct that his description of L2 sampling could be correct. Alas, he has no real evidence for it since there is no clear, final statement by the authors themselves. How to make progress? Tim should ask Les Roberts to clarify, once and for all, the exact methodology followed by the Iraqi interviewers. For example, did they have maps?
Tim:
We cross-posted. You write: "I just followed the methodology they described in their paper."
But we already know that the methodology described in the paper is wrong. Burnham/Roberts have told us that! Do you disagree?
Can you provide a citation to your claim that Roberts "already answered this?" I am not doubting that you have such a citation, but collecting the various contradictory things that Roberts has said about the sampling plan is a hobby of mine.
I look forward to part 3.
DK, see [here](http://scienceblogs.com/deltoid/2006/11/burnham_and_roberts_reply_to_s…) for Roberts' comments. They added this to the description in the paper.
>"As far as selection of the start houses, in areas where there were residential streets that did not cross the main avenues in the area selected, these were included in the random street selection process, in an effort to reduce the selection bias that more busy streets would have."
Clearly this makes no difference to my result above. Adding streets to the set of main streets will only improve the coverage.
And note that I didn't go out a search for a map where the scheme described in the paper worked, this was the one they offered to show that n=10 was plausible. It isn't.
Well thus far with my map of london centred on where I live, a bit of string, a ruler and a random number generator I'm getting pretty wide dispersal with only officially designated A roads as main streets even. Will hopefully get this process automated so as to do some proper analysis.
Tim:
In the link you provide (thanks), Burnham/Roberts claim that "Sampling for our study was designed to give all households an equal chance of being included." This is a contradiction of the sampling scheme you describe above. For example, consider all the houses that are near the intersections of two (or more) cross streets. Those houses are much more likely to be included in the sample since picking either cross street still gives them a chance. Other houses are near (meaning within a possible 40 house sample) only one cross street. Your picture illustrates that perfectly.
So, either Burnham/Roberts were wrong in their letter to Science or you are wrong to claim that your picture illustrates the sampling plan they undertook. Which is it?
Tim Lambert should read the following comment from Les Roberts (in a 2007 radio interview, via David Kane's blog) as it makes Tim's latest "deep, thorough penetration" scenario questionable:
And then there's this comment from Burnham (MIT, 2007):
Huh?
Jody is doing what Spagat and Co should have done.
Eli: Instead of telling Spagat and Co what they should have done, why don't you explain to us what Burnham and Co actually did? You have followed this debate closely. Do you think that this is a true statement by Burnham/Lambert?
If so, then isn't Tim's example wrong?
For example, consider all the houses that are near the intersections of two (or more) cross streets. Those houses are much more likely to be included in the sample since picking either cross street still gives them a chance. Other houses are near (meaning within a possible 40 house sample) only one cross street. Your picture illustrates that perfectly.
David, this is getting stupid (again). you have exactly the same problems if you randomize GPS coordinates (single house will get higher chance of getting picked) or telephone interviews (people with multiple numbers or people only using mobiles or people who travel a lot..).
all of this is "as best as circumstances allow".
i doubt that anyone including Burnham takes
"Sampling for our study was designed to give all households an equal chance of being included. "
to mean EXACTLY the same chance. you do understand, that populations move a lot in Iraq?
David Kane:
>If so, then isn't Tim's example wrong?
No.
I take it that no-one disputes that if you follow the sampling scheme described in the Lancet paper and apply it to the map that the MSB authors chose, you get almost complete coverage, n at most 0.1 rather than n=10 as they claim. Don't you think that this is a serious flaw in their paper? Is that why you are trying to change the subject?
Tim Lambert writes:
Tim is confusing the "Lancet sampling scheme" with the "Lambert sampling scheme".
We actually know more about the latter than the former, which is a pretty sad reflection on the Lancet authors.
Or, to spell it out for the slow-learners, Lambert's analysis is disputed by those who realise that since Lambert doesn't know what L2 took to be a main street, his own guess wrt "main" street vs cross street is irrelevant.
His choice (suggesting that nearly every street in Iraq was in the samplable space) is at odds with arguments made by the L2 researchers about limitations due to safety and shortage of time, etc.
The Lancet authors claim "all households" had equal chance of inclusion. The burden of evidence is clearly upon them.
Shone and Kane are claiming perfect ignorance, which in their cases is an acceptable starting point. However, down here on earth it is clear that you should be able to look at any city, impose a condition for what a main street is and look at what the results are as that condition is varied. That is what Jody is doing. It is also clear that Spagat, et al (Hi there et, or is it al) imposed an unreasonable condition a la McIntrye and McKitrick, e.g. they picked a set of "main streets" that yielded the effect they wanted.
Now this may not be an easy problem. For example, when Eli looked at the maps, he saw that one reasonable way to pick main streets was to see where the mosques are (domed structures), the markets (lots of sheds and people) and a few other tells. In the main these were NOT the streets picked out by Spagat, Johnson, et al.
So, let us ask Spagat, Johnson and co. what condition THEY used for picking main streets and see if it is reasonable.
Further, a point which appears to be missed, is that most of the deaths by violence were from gunshot wounds, something not as likely to be concentrated on main streets as bombings might be (depending on your definition of a main street).
Eli Rabett writes:
Have you asked the Lancet authors which "conditions" they "imposed" to establish whether a street is a "main" street? I thought not. Why don't you try asking them, and then share their response with us.
Suppose, for the sake of argument, that Gilbert Burnham is such a shameless fraudster that he merits a place on Tim Radfordâs list. Would that excuse what Johnson et al. did with that map? In my view it would not. Is anybody prepared to defend them? So far in this thread nobody has even tried. All weâve seen from Robert Shone and David Kane is attempts to change the subject.
Shone, Kane:
Why on earth are you two not responding on target to THIS criticism of THIS paper, which is what THIS thread is about?
Spagat et al claim that n=10 is a reasonable estimate. Use of a value near that - at least much greater than 1 - is necessary to their conclusion. They support their estimate of n=10 by the maps to which Tim links. One of those maps is incomprehensible.
This map, the one Tim analyzes, shows that an estimate of n=10 can only be arrived at by making facially absurd exclusions of major streets from the category of 'main streets,' which in turn excludes obviously secondary streets from the category of "secondary streets."
This is a criticism of Spagat's argument, which is based on an absurd classification of streets. Rather than divert, would the two of you please deal directly with that point, and its consequences to Spagat's argument?
Or you can continue being irrelevant, if you prefer.
Robert Shone doesn't get it. By examining the result of a sequence of choices for main streets you can get a very good idea of how resilient the result is to the specific method of choice without asking anyone what their choice was.
Frankly the question that I have about main street choice is based on experience. Many places have small commercial zones on long streets/roads. This is especially true about small towns and from the maps also appears to be true about Iraqi cities. If this is so, and the length of the street is much longer than the commercial zone, Spagat, Johnson and Cos argument about main streets vanishes.
Lee writes:
Try reading my comments #3, #5, #16, etc - they address Tim's points directly (in fact they address little else). There's nothing in Tim's new post that's not already covered in my blog entry:
http://dissident93.wordpress.com/2009/02/15/pope-of-debunkers/
Slightly related, gives you an idea at least perhaps of the challenge in surveying this kind of thing. Unfortunately behind a paywall:
http://tinyurl.com/adufua
More Lancet Stuff
Robert, I'll admit that I fall into your category of "slow-learners." I have only a high school education, I have no particular math skills, no background in statistics, and no special knowledge of Iraq. I am, however, trying to keep an open mind, and I'm trying to grasp the whether or not the main street bias argument makes sense.
You've repeatedly said that Tim doesn't know what the Lancet team actually did. How is Tim's guess about what the L2 authors did any less valid than the Johnson et al's guess? I think Tim makes a convincing case that the methods Johnson et al are using to determine the extent of the coverage are not plausible.
Do you have a comment on Sod's observation that the criticisms here have focused almost exclusively on Baghdad? And how would main street bias affect the results in smaller towns? Was this what you were referring to in comment #5? I'm sorry if this is tedious for you, but I can't understand what you are saying, and I think I'm probably not alone. What do you think Tim's "yellow road" scenario implies when applied to the whole of Iraq, and what is it about these implications that is implausible?
Earlier, in the previous JPR thread, I asked you to clarify what information the L2 authors could have provided that would allow Johnson et al to come up with a reasonable value for q. (You conceded that the parameter value Johnson et al chose might not be correct.) You didn't reply, but Ozzy/Ron/LancetDebunker/Tell suggested that BBC maps of violent incidents could be used to determine the relative frequency inside and outside of the survey space. Kevin, Sod and I pointed out that this doesn't help at all: Sod noted that the maps show where the killing happened, and not where the people involved lived, and Kevin and I pointed out that virtually everyone agrees that the media is only reporting a fraction of the deaths, and that the pattern and scope of unreported deaths is exactly what we need to determine.
You say in #22 that the burden of evidence is on the Lancet authors to show that all households had an equal chance of inclusion. Why do you believe that they didn't have a (practically) equal chance of inclusion? (As Sod noted in responce to David's comment, yes, there are some minor differences of probability... but that would be true of ANY sampling scheme.) Do you agree that if the Lancet team used "the Lambert sampling scheme," they would have had a good chance of including nearly any household? As far as I can tell, your only reason for arguing that the burden of proof rests with the Lancet authors is that you have a prior assumption that they were either dishonest or incompetent. Maybe they are, but it seems to me that the burden of proof rests with you, particularly when Tim's example shows that, depending on whether or not one sensibly defines a "main street," the scheme is not necessarily subject to "main street bias."
You've admitted that you don't know what "q" should be, and you don't know what "n" is. As Kevin noted in the last thread: if we can plug whatever parameters we want into the model, there doesn't seem to be a whole lot of point to this exercise.
Regards,
Bruce
Bruce did sum up those points much better than i ever could. thanks!
Bruce writes:
Tim's "guess" suffers from two problems.
First, he refuses to admit that it is a guess. He acts like he knows (and everyone else should know) exactly what the Lancet sampling plan was. But he doesn't know! (See here for extensive documentation of that claim.)
Second, he refuses to help us figure out what happened. Tim could easily e-mail Les Roberts and seek clarification. (Roberts has replied to Tim's e-mails in the past.)
So, the first step in making progress is for Tim to admit that he is guessing and then to work on turning his guesses into facts. Why won't he?
But, to answer your question, I think that Tim's guesses about what the Lancet team did are not unreasonable. I think that Johnson et al's guesses are not unreasonable. Which ones are better? Tough to say! And, to be honest, not that interesting (to me).
Really? What evidence does Tim cite for this? Start with a specific example. How many main streets are there in a city like the example that Tim creates? One? Five? 50? 500? The number of main streets is, obviously, a crucial input in deciding what the coverage is and, therefore, what n is. But Tim, you, I and everyone else has no idea how many main streets there are in this cluster or in any cluster because the authors refuse to tell us (or anyone else). From my link above:
Again, it could be that Tim's guess about n is more accurate than Johnson et al's guess. Or maybe your guess is better still! But no one in the debate has any good evidence for why there guess is good because the Lancet authors refuse to provide any useful information for estimating n (or any other parameter).
Shone, having read this and the previous thread, I find I must bite my tongue, err on the side of politeness from what I think deserves to be said, and simply say "bullpucky."
In 3, you dont respond AT ALL to Lamberts demonstration that the Spagat street selection procedure is absurd. You simply quote the paper's authors to say that other selection procedures might give a different answer. Well, duh! That isn't a response, it is a deflection.
Your 5 is content free, as far as I can see.
In 16, you again fail to respond to any point Lambert is making, instead quoting the L2 authors on selection procedure. How do those quotes address Lambert's point that THIS PAPER's authors use an absurd criteria for distinguishing main streets form secondary streets, and that if one uses more reasonable criteria, their argument fails?
David Kane: Seppo Laaksonen, a professor of survey methodology in Helsinki, requested and was denied any information on main streets, even the average number of main streets per cluster.
The way I read it, Laakonsen wasnât âdeniedâ information :
A man canât be said to deny things to people if he doesnât have them to give. Do you have any reason to believe Burnham has the information Laakonsen requires?
Kane, Roberts has said that their selection procedure was designed to attempt to give every house a chance of being selected.
Unless you are going in with the assumption that they are lying and committing fraud - and you clearly are, Kane, and it is IMO despicable behavior - then that adequately answers the question about the numbers of main streets. Their procedure is designed to attempt to give every house an equal chance of selection, and that means that the map Lambert is criticizing in this thread very obviously and clearly does not match the Roberts sampling scheme.
Donoghue at 34:
I suspect you've just hit on the motivation for much of this.
Roberts et al were designing a survey under very edgy ethical conditions. There was a credible very-non-zero risk of death to the surveyors, and to the people being surveyed for answering the questions. He was and is ethically constrained to to do everything possible to minimize the risk to the people being surveyed, and he designed the procedure with that in mind.
Information that was even borderline related to ways to identify surveyed neighborhoods or people was simply not transmitted to him, in order to maintain the confidentiality of people who were potentially at risk of their lives for answering the survey.
Now these guys come along and invent scenarios which allow them to demand that Roberts release information that he by design did not retain, for ethical reasons - and then use his ethically-constrained survey design and retention policy to lambaste his ethics, because he is not releasing the information that he for ethical reasons CAN NOT release and did not retain.
I suspect that this is intentional on the part of Robert's critics. I know that it is despicable behavior.
Bruce Sharp writes:
I may be wrong, but your questions suggest to me that you haven't read the MSB paper or its associated explanatory web pages. It's pretty clear, for example, on the above point. The sampling scheme as published in the Lancet journal wouldn't give all households an equal chance of being included (not even a "roughly" equal chance). Even Burnham/Roberts seemed to acknowledge this - eg they stated that they omitted from the published account the procedures which would enable them to "reduce the selection bias that more busy streets would have" (see comment #13 for link).
That comment from Burnham/Roberts makes no sense if they believed that the published sampling procedures by themselves "reduced" the bias from "busy streets".
They've never released the details of these necessary "additional" procedures. They simply made assertions that they somehow reduced this bias to zero (or near zero). Hence the burden of evidence.
Forgive me if I don't answer all your other questions. I've made the point repeatedly that people are free to suggest their own parameter values based on their own assumptions, and that until we hear from the Lancet authors these are speculative as far as the Lancet study is concerned. Based on what I've read about the study (eg see my comment #16) I do think that the MSB example parameter values are plausible - certainly far more plausible than Tim's n=0 suggestion. That's just my opinion based on the limited information available. It may turn out to be wrong. The MSB authors suggested parameter values which they found plausible based on the limited information available. You may disagree with them, but they've committed no sin by stating what they find plausible. And if you don't like their suggested values, they've provided an exploration of the parameter space.
The bottom line is that it's down to the Lancet authors to support their own assertions wrt how they "reduced" the bias to zero (or near zero).
The tail end of these threads tend to get repetitive (and abusive and pointless). I'm not going to repeat myself further - this is my final post in this thread.
A summary of my take on Lambert's feeble attempt to debunk MSB is available at my blog:
http://dissident93.wordpress.com/2009/02/15/pope-of-debunkers/
Not to answer for Shone, who's done a good job on his own, but Bruce Sharp writes:
How is Tim's guess about what the L2 authors did any less valid than the Johnson et al's guess?
It's not really. They are just different guesses. Johnson et al's presentation of this is more valid than Tim's though because they say they do not know which streets or type of streets the L2 authors (field team actually, not the authors) called main streets in practice. Tim claims to be "correcting" their guess by replacing it with his guess. So he is making invalid claims about his guess while Johnson is not.
I think Tim makes a convincing case that the methods Johnson et al are using to determine the extent of the coverage are not plausible.
I don't. The issue here is basically that the wider the definition of main street in practice, the more coverage and the less bias. The narrower the definition, the less coverage and the more bias. Tim guesses a wider definition was used than the guess in Johnson's example. Is this more plausible? I don't think so. To the extent we want to debate these guesses I think Tim's guesses are less plausible in practice than Johnson's for a number of reasons.
There doesn't seem to be a formal definition of what a main street was for L2. So this definition would have to be a post hoc one, simply the outcome of whatever the field team did. At least three factors work against the wide definition hypothesis: the extreme time constraint on the field teams, the fact that they had to discover the streets in question in each neighborhood, and the security issue.
The time burden here is large as they already have to do 40 interviews in a day. In the case of this example, the teams would have had to drive to Kirkuk (from where?), then select the area to work with (how big is the area?), then travel around that area trying to find every "main street" in the area to put in their list. If a very wide definition of main street is used, their job of discovering and enumerating the main streets becomes harder, more time consuming and more dangerous. If a narrow definition is used the job is easier and takes less time, and they can get right on to cramming those 40 interviews into whatever is left of the day. Then there is security. The wider the definition, the more time they have to spend driving or walking around these strange neighborhoods looking for all the main streets while exposed to whatever dangers there might be.
There appears to have been little in the way of any specific criteria or conditions about what had to be called a main street. This was up to them to decide. Did they choose to make their job as difficult and time consuming and dangerous as possible, or did they choose to make it less? Which is more plausible?
Let's bend over backwards for the Lancet study as Tim is doing and assume the field teams chose to make their job as hard as possible and used a wide definition of main street. They would then have to travel around the selected area (again how big is this area?) looking for all the main streets to list them. In the best case scenario, they find them all, but that is only the best case. In every other case they don't discover all these widely defined main streets to begin with. Or, they get a couple of the most obvious ones down on the list and then stop traveling around looking for more main streets that they don't even really know are out there to find. The same problems would apply to enumerating the cross streets.
More plausible, I think, is that in practice the definition would be narrow. They'd grab the most obvious main streets and get to work.
Do you have a comment on Sod's observation that the criticisms here have focused almost exclusively on Baghdad?
Yes. The observation is wrong. The map in question is Kirkuk, for example. There are three maps shown on the MSB page linked above and none are for Baghdad.
Why do you believe that they didn't have a (practically) equal chance of inclusion?
Because the sampling methodology does not establish an equal chance, practically or otherwise. In the best possible scenario it would give a very highly unequal chance of inclusion for each house, but some chance for all. One of Tim's claims is that his guess about the yellow lines would get every house into the sample frame for the piece of the map he chose because of possible spillover onto tertiary streets when progressing through the 40 houses. But even if this were to get every house some chance of selection it would be nothing like an equal chance of selection.
This issue of progressing through the 40 houses also raises another can of worms. It is again a problem of vague sampling protocol and lots of discretion left to the team in the field to do whatever they want. Tim uses the phrase "neighbouring the start house" to describe how the teams progress from the start house to the rest. The Lancet authors have used the words "adjacent" or "nearest". All are vague and allow discretion (and therefore bias) to enter in. For example, the team does it's start house, there's a house on either side that are both about as "near" to the first, they are both "neighboring" or "adjacent". The team looks to one side and see a row of bombed or battered houses. They look to the other side and don't see anything like this. Which way do they go? They could go either way and still be following the (vague) protocol. This allows the team to both follow their instructions and (consciously or unconsciously) seek out high mortality houses.
The way to progress from house to house is wide open. Do they always stay on the same side of the street and travel in a line from one house to the next? What do they do when they hit a street corner? Do they instead go in a kind of concentric circle out from the start house, where maybe the house across the street in front and behind and the two on either side are the next four "nearest" houses? They could conceivably do any of these or something else and still be within the vague method. This problem worsens if the locals are involved (such as neighborhood children used in the survey) who might try to direct the teams to houses they know have deaths to report. Since there's discretion in how teams progress from the start house, how or why would they refuse a local who's telling them to go to those houses over there? They can't tell them it's against the rules to go to them because it's not. Nor would they want to do something that might insult the locals or make some of them start to question the "benign intent" of the survey. They can just choose to progress in the direction of those houses from the start house. In each cluster they'd have 38 chances to employ this discretion (and introduce bias).
As Kevin noted in the last thread: if we can plug whatever parameters we want into the model, there doesn't seem to be a whole lot of point to this exercise.
If you mean trying to determine the extent of this bias in L2, you're probably right that there is not much point in the exercise. The necessary information isn't there and isn't likely to ever be there. So almost any of the parameters could be the right ones. This of course means directly that there is not much point in deriving an estimate of nationwide deaths from L2 data because it hinges on choosing (knowingly or unknowingly) whatever of these parameters we want. The L2 authors (unknowingly) chose to plug in the parameters that gives them zero bias. These parameters make the number come out to 600,000. Another perhaps more plausible guess made (knowingly) by Johnson is the parameters that make the number come out to about 200,000. Since there's no solid case for either parameter set, or others besides, there's not much point in the exercise of making an estimate based on L2 data if the aim is to discover the level of deaths in Iraq. We may as well cut out the middle man and just guess the number, rather than guess the parameters that produce the number.
If you mean trying to explain or explore the potential for main street bias (what Johnson is doing) in a survey such as L2, then being able to plug a lot of different parameters into a model and showing the wide range of these and their outcomes has a useful point that, unlike L2, advances our knowledge about the topic it is addressing.
Raymond says:
"Johnson et al's presentation of this is more valid than Tim's though because they say they do not know which streets or type of streets the L2 authors (field team actually, not the authors) called main streets in practice. Tim claims to be "correcting" their guess by replacing it with his guess. So he is making invalid claims about his guess while Johnson is not."
One more time - BULLPUCKY!!!
Roberts has said that their main street - secondary street selection procedure was designed to give all houses an equal chance of inclusion. Lambert has shown that it is very easy to make a selection of main streets - secondary streets such that all houses have an approximately equal chance of inclusion. Johnson's scheme is at odds with Roberts description - Lambert's is not.
Johnson's scheme is preferable only if you assume that Roberts is intentionally lying and L2 is fraudulent. You are adding to the despicable behavior here, Raymond.
"Roberts has said that their main street - secondary street selection procedure was designed to give all houses an equal chance of inclusion."
All houses having an equal chance of inclusion is a desired outcome. A design intended to achieve this may succeed or not. The design the L2 authors have described can not claim to achieve this. If there is some hidden elements of their design that do achieve this they should let other people know what they are. That they don't makes such statements seem like so much wishful thinking.
"Lambert has shown that it is very easy to make a selection of main streets - secondary streets such that all houses have an approximately equal chance of inclusion."
No he hasn't. He's shown you can make a guess about the definition of main streets in one corner of a particular neighborhood map that might give all those households a highly unequal chance of getting into the sample.
"Johnson's scheme is preferable only if you assume that Roberts is intentionally lying and L2 is fraudulent."
I don't do "faith" Lee. Nor should science. If you want me to believe a sample design achieved equal probability for all households then, like the maths teacher says, show your work. The design they've described doesn't do it.
Dave Kane has a good posting up on his blog right now with a lot of comments from Roberts about the sampling which contradict each other and can't all be true at the same time. You should check it out, and leave your faith for Sunday.
Raymond and David, thanks for taking the time to reply. The first point which you both raised -- that Tim doesn't describe his guess as a guess -- seems like a non-issue to me. I'm not interested in how he describes his argument. I'm interested in his argument. Suppose you said to me, "Hey, I've got the funniest joke ever. This guy walks into a bar..."
I'll either laugh or not laugh based on the joke. The fact that you told me it's the funniest joke ever isn't going to make any difference.
When I look at Tim's example, I'm using a dirt-simple method to evaluate it: I look at the map, and I ask myself, "What would I describe as a 'main street'?" I think I'd pick the same things Tim picked. I think most people would.
Incidentally, although I read the draft of the Johnson paper, I had not (until now) looked at the web pages Tim linked to, on the introductory pages. You are right: The authors do discuss other locations, and not just Baghdad. The comments there regarding smaller towns, however, seem plainly at odds with common sense: "It is not clear to us how the major commercial street sampling would work when there is no major commercial street available." I don't think any reasonable person would use the same criteria for what constitutes a "main street" in both a town of 1,000 people and a city of 1,000,000.
Raymond, in your explanation of why you don't think the teams would have found all the main streets, it seems to me that your three reasons would lead to a bias in the opposite direction from what you suggest. If the teams were worried about time and safety constraints, and needed to find main streets, a broader definition of "main street" would be easier and take less time. ("Do you think this is a main street?" "No, let's just wander around aimlessly for a while until we find a different one that looks 'main-ier.'")
In your reply to Lee, you said that the survey design can't claim to have an equal chance of including all houses. I don't agree: depending on the implementation of the survey, that may or may not have been the case. Again, the outcome depends on whether or not the chosen "main streets" were at all reasonable. I'm not sure why you say the chance of inclusion is highly unequal, but I'm going to guess that this, too, will come back to an unsettled question of implementation.
Although the discussion in this thread has focused primarily on the inclusion (or exclusion) of particular areas, it seems to me that even if this question were settled, we'd still be left with the impossible-to-calculate ratio of violence inside and outside the survey space. As Kevin has previously noted, the L2 scheme was borne of exigency, and it doesn't seem likely that it will be used again. I don't see how Johnson et al is useful, if divorced from L2. Its only value is in convincing people that L2 has no value... and I'm inclined to believe that most of the people who think L2 has no value had reached that conclusion long before they ever heard of main street bias.
Regards,
Bruce
Yes. The observation is wrong. The map in question is Kirkuk, for example. There are three maps shown on the MSB page linked above and none are for Baghdad.
their [main example](http://www.rhul.ac.uk/Economics/Research/conflict-analysis/iraq-mortali…), the one from which they draw their horrible "visual summary" of "mainstreet bias" is Baghdad.
it is obvious. what Spagat is doing, is choosing only "city highways" with few exits/crossroads. this is, how he keeps the covered area small.
Because the sampling methodology does not establish an equal chance, practically or otherwise. In the best possible scenario it would give a very highly unequal chance of inclusion for each house, but some chance for all.
again: so does a GPS approach. bigger houses or "lonelier" houses have a higher chance than others.
There appears to have been little in the way of any specific criteria or conditions about what had to be called a main street. This was up to them to decide.
funny. i tend to recognise the "mainstreets", when i enter a town or village. the one you enter the town on, is one. the big ones leaving that road (mostly leading to other other towns..) are.
in very big cities, it gets slightly more complicated, but even there its no rocket science.
For example, the team does it's start house, there's a house on either side that are both about as "near" to the first, they are both "neighboring" or "adjacent". The team looks to one side and see a row of bombed or battered houses. They look to the other side and don't see anything like this. Which way do they go?
"bombed and battered houses" will often be abandoned, leading to a NEGATIVE bias, reducing (polled) mortality!
how much experience do you have with polling? imagine you do a telephone poll. one voice on the line is a nice young female one, the other a barely understandable elderly man. who will you try harder to keep finishing the poll?
The L2 authors (unknowingly) chose to plug in the parameters that gives them zero bias. These parameters make the number come out to 600,000. Another perhaps more plausible guess made (knowingly) by Johnson is the parameters that make the number come out to about 200,000. Since there's no solid case for either parameter set, or others besides, there's not much point in the exercise of making an estimate based on L2 data if the aim is to discover the level of deaths in Iraq.
let me see, whether i got this right: the methodology of the Lancet authors, made with the aim to give an equal chance of cover for houses is "as plausible" as the Spagat guess at what happened, which is based on a choice of streets that contradict the target of the study?
this is insane!
In common parlance, a road is a public right of way wide enough for vehicles. A street is a stretch of road with lots of houses, shops and public buildings on it. Industrial estates have roads, not streets. On the map, the red line is a road, the yellow a street.
It beggars belief that any not half-witted survey team with a quota to meet would start from unpopulated stretches of road.
James Wimberley must be from UK (or UK-educated) given the way he distinguishes between streets and roads, which highlights a major misunderstanding on the part of all non-US people on this thread:
**The L2 scheme was designed by a US team. In the US, the "main street" of a town is the major highway. It does not mean a small street in the UK sense.**
So the main street for a US person is a "city highway" as Sod says. People do not tend to live on the main street. L2 started their survey scheme from "main street" in the sense that they then choose cross streets. So very few properties, if any, get included from the main street.
Likewise, in the JPR paper, the main street can happily be included in So, i.e. outside the survey space. The survey space includes cross streets to main streets, not necessarily the main streets themselves.
So the criticisms of JPR based on misconceptions (i.e. non-US interpretations) of "main street" are not valid. The L2 team included residences on cross streets, not main streets. Likewise JPR is completely consistent with main streets being outside the samplable space.
what part of Kirkuk is that on this [map](http://www.lib.utexas.edu/maps/middle_east_and_asia/karkuk_2003.jpg)?
Bruce:
"The design the L2 authors have described can not claim to achieve this."
Dude, Lambert just showed a map that matches what the L2 authors describe, and that does exactly that.
Lee wrote: "..what the L2 authors describe..."
Until we all know what the "main streets" were for L2, any claims are debatable.
The problem is that the L2 authors did not describe this is in sufficiently clear detail to avoid these repetitive debates.
Personally I don't consider either Tim or JPR "right" or "wrong" in their interpretations of the implementation of the scheme. But I do wish I know something more from L2 to narrow down the discussion.
Side note: In terms of applicability of JPR more generally, I personally find it a useful and original contribution since many areas of medical imaging (my field) come across a very similar situation, where the objects which pick up tags (and hence can be seen) may be a biased sample of the population which are actually reacting, and the precise area of the sample being investigated under the microscope may also be biased.
sod,
I think the east-west road that Tim is taking as his main street is roughly where you see the words NAHIYAT PIRYADI on your 2003 map. I fact I wouldnât be surprised if it was just that map which prompted Sean Gourleyâs choice of colours. But the roads in red on that map seem more like major traffic arteries â the sort of routes you would take if you were passing through Kirkuk, or driving from one side of town to the other.
Incidentally, your 2003 map highlights two things: (1) Gourley made lousy choices for his âmain streetsâ, picking one which is actually at the eastern edge of the town; and (2) Kirkuk is a very big place. My guess is that, in such a large city, the JHU team would have sampled their âmain streetsâ from a map (or maps) of the entire area, narrowing down their target cluster to within a few square miles before they even drove to Kirkuk. Thatâs what Iâd do anyway.
James Wimberley is, I am reliably informed, an Englishman raised in the Channel Islands. But I wouldn't be too quick to assume that "all non-US people on this thread" have got the wrong end of the stick. The Lancet is a UK publication and the authors of the JPR paper are mostly from that side of the pond. Actually, if it comes down to dialects, the crucial question is, what does Riyadh Lafta understand by main street? Given Iraq's history I think it's very likely that his English derives from British sources.
None of which takes anything away from Tim's point: the efforts of Johnson et al. to derive n = 10 from that map should have prompted the editor of the JPR to ask what they were playing at.
Kevin Donoghue, you have it wrong. The authors of the L2 paper are mostly from the US, so it is their definition of "main street" which is under debate, since that is what would have formed the basis for the L2 survey plan. Looking at the JPR maps, it seems to me that JPR proceeded based on a US interpretation of "main street".
The possibility that you raise that Lafta may have interpreted "main street" differently, because of British-biased education (your suggestion, not mine) then makes the whole L2 survey even more questionable. You are therefore suggesting that Lafta is the only one who actually knows what main streets would then have been included!! This would indeed be a serious scientific oversight of the US L2 team, since they would be co-authoring a paper (as lead author etc) based on a study where they do not know exactly what constituted a main street in practice.
You just shot down L2.
JPR team offered a theory of a generic situation of sampling in a population where all objects are not equal in terms of sampling. Very interesting. Whether their theory is general enough, or too specific or just right, depends on what you want to do with it. They then offered some values for a specific case, L2, and invited readers to use their own values. Seems like good science to me....
By the way, if you take the JPR interpretation of "main street" from their maps (which they freely admit are just a guide) then I count about 10 lengths of road which is non-samplable, for every given length of road that is samplable. Assuming a constant housing density per length of road, then n=10. So JPR are not unreasonable.
(Note that counting numbers of roads is not the issue, it is lengths of road if we assume constant housing density on residential roads. If we don't assume that, then we are off onto another debate, and another can of worms for this thread).
'good science....'
Wouldn't that involve doing something like, I don't know, testing the model?
LancetStudy,
Well if you say I have it wrong, thatâs that I suppose. Here is the passage which led me astray â and it still does despite your best efforts to show me the light. I hope youâre not going to ask me where I found it.
To me, that means that Riyadh Lafta was in the driverâs seat. Iâm not prepared to entertain the notion that my Hiberno-English-biased education (as you would presumably term it) is causing me to misread that paragraph. Iâm really quite sure that what it says to me, is precisely what it says to Richard Horton.
I note that you are shocked, shocked at the suggestion that a bunch of Americans might have got into something where they did not know exactly what they were doing. That makes me wonder where youâve been for the last six years or thereabouts. Personally I give a bit more credence to the JHU team precisely because they had the good sense to give the crucial task to a group of Iraqis whom they deemed to have the required skills. If that invalidates the whole study in your eyes, so be it. Perhaps the guys who are losing their jobs on Wall Street will team up with former employees of the Justice Department to produce the sort of study you feel you can trust.
The attempts to defend this paper have turned grotesque.
Are these guys really arguing that because in the US 'main streets' means 'the main street' (which is absurd), that street which are clearly among the main streets on the map are to be teated as secondary streets?
Becasue I'm from teh US, and that definition of main street is absurd beyond words. I drive up and down the California central valley frequently. When I do, I often stop at small freeway towns (5,000 - 20,000 people) to find a 'taco truck' to get lunch. To find one, I simply get off the freeway and DRIVE AROUND ON FREAKING MAIN STREETS!!! MULTIPLE. PLURAL.
They are easy to recognize, clearly distinct from secondary streets, there are always more than one even in small towns, usually more like 3-5, and that map at th etop fp this thread clearly excludes a main street from its main street category.
And I am from the US.
Lee: How many people live on your taco-rich "main street"? Very few. They are main streets/high streets in UK terminology. Few people live there, they are not in the sample space of L2 or JPR. They are just a starting point for the cross street method which L2 used. But knowing what they are is crucial since they are the starting point.
Jody: On this topic, many US citizens would consider Euston Road a "main street". There are no markets on Euston Road, nor are cars allowed to park. Few people also live there in relative terms compared to rest of London. Instead, the action in terms of markets, cars parked, people walking and hanging around is on streets off of Euston Road. And people who live on streets off of these, sufficiently close, will walk there and tend to spend more time there. People who live way off will not.
Your analysis using London is from the perspective of someone who defines "main street" according to a European definition.
Kevin D: I am afraid you hit the nail on the head, albeit unintentionally, when you raised the possibility that there are "main street" definitions which Lafta might have used which differ from the US L2 team. So how can the US L2 team be so sure that no MSB exists? They cannot. How can they put their names first etc. on a paper where they do not know what the sample space actually was? They should not.
This is clearly why the AAPOR and JH college themselves, are worried enough to start investigations. Your argument validates this action.
Pre-empting somebody's comment that the subject of this thread is JPR's n=10 value, and hence that users should address the n=10 issue, I have above. Here it is again:
If you take the JPR interpretation of "main street" from their maps (which they freely admit are just a guide) then I count about 10 lengths of road which is non-samplable, for every given length of road that is samplable. Assuming a constant housing density per length of road, then n=10. So JPR are not unreasonable. (Note that counting numbers of roads is not the issue, it is lengths of road if we assume constant housing density on residential roads. If we don't assume that, then we are off onto another debate, and another can of worms for this thread).
But the most important outcome of this thread, apart from this point, is the bombshell issue that Kevin Donoghue has raised about Lafta and Burnham/Roberts not being on the same page with regards "main street" definition. Should we have a thread on this?
(P.S. What Lee, Tim, Kevin, Jody guess as constituting a "main street" in the L2 survey sense, is irrelevant. What L2 define as a main street was, I had assumed, the main issue. But now Kevin has raised the important point that Lafta's definition is actually a major issue. What a mess for L2!)
'grotesque...'
Burnham et al central estimate of excess mortality 650 000 (390 000 - 940 000), allowing Johnson et al their guessed at R = 3, central estimate becomes 217 000, not sure what it does to the confidence interval.
Just to put this grotesqueness into context.
Lee, re your comment #46: I did not say that. That was Raymond, in #40.
Regards,
Bruce
LancetStudy: But now Kevin has raised the important point that Lafta's definition is actually a major issue. What a mess for L2!
Actually I just pointed out that Lafta managed the field survey. People who had read the paper closely already knew that.
BUT THEY EMPLOYED AN IRAQI!! IN IRAQ!! WHAT A MESS FOR L2!!
WTF? I mean really, is this what the defenders of Johnson et al. are reduced to?
Jody wrote: 'grotesque...'. Yes, I agree. I think that everyone agrees that any death more than zero is grotesque. I am sure that the JPR team think the same. That is not the issue. The issue is to get to the bottom of whether science is being done in a reasonable way. To the extent that conflict surveys become a scientific method, then this method needs to be examined. If it is useful, great. If it needs to be corrected, then we should know -- and we should try to estimate what these corrections might be. Only then will the method be treated as credible and full credit given to survey teams etc.
So yes, definitely grotesque. The question is: reasonable in terms of scientific standards or not? We are not doing any favours to anybody if it is not. 200,000 does the job in terms of being grotesque -- no need to have an inflated figure, particularly if it then detracts from the main issue of how to prevent and stop wars.
KD wrote: "...is this what the defenders of Johnson et al. are reduced to?..."
As a whole, we readers of this thread are left with the remarkable issue that you helped fuel: Do *all* the L2 team know what main streets were possible candidates, and do they *all* agree on this? If they do not, then where does L2 stand?
I have an experiment that I did with someone, and the results are X. Well actually, I wasn't there when they did the experiment, but the results are definitely still X. Well actually, I don't know line-by-line what exactly they did, but the results are definitely still X. Hmmm???
I hope the investigative teams read these threads. This should help them with some of the issues.
As for JPR (which actually should be the least of your present worries if you are an L2 defender and have followed the above thread):
1. JPR article, and the possibility of MSB, stands strong (stronger, arguably, after these discussions)
2. the JPR maths is correct
3. the JPR estimates are indeed estimates, but not unreasonable ones given the uncertainty all round about L2
JPR does exactly what it says it does in the abstract. It does what it says on the can.
I'm sorry Kevin, but like it or not, you said in post 49:
"..the crucial question is, what does Riyadh Lafta understand by main street?"
You said it. I happen to agree with it, but you said it. You hit the nail on the head about why L2 should be investigated.
Jody wrote: "...Wouldn't that involve doing something like, I don't know, testing the [JPR] model?"
Sure, just tell the JPR team the main streets that were used. Failing that, what were the possible main street candidates? Failing that, what did Lafta understand to be a "main street" to use as a starting point? Failing that, what did US L2 team actually write in words in their design description to Lafta to translate the "main street" selection scheme?
Failing that, how did US L2 team conclude that Lafta's implementation of the "main street" scheme gives R near to 1?
Of course, none of you know the answers to these questions. And yet you are surprised L2 is being investigated? Duh??
I guess Ron missed us.
I will repeat (from 36), becae you guys are confirming it fo rme.:
"Roberts et al were designing a survey under very edgy ethical conditions. There was a credible very-non-zero risk of death to the surveyors, and to the people being surveyed for answering the questions. He was and is ethically constrained to to do everything possible to minimize the risk to the people being surveyed, and he designed the procedure with that in mind.
Information that was even borderline related to ways to identify surveyed neighborhoods or people was simply not transmitted to him, in order to maintain the confidentiality of people who were potentially at risk of their lives for answering the survey.
Now these guys come along and invent scenarios which allow them to demand that Roberts release information that he by design did not retain, for ethical reasons - and then use his ethically-constrained survey design and retention policy to lambaste his ethics, because he is not releasing the information that he for ethical reasons CAN NOT release and did not retain.
I suspect that this is intentional on the part of Robert's critics. I know that it is despicable behavior."
the map i posted above is from 2003.
i have serious doubts, that the Lancet team was cruising around towns, searching for "mainstreets", when maps were easily available.
i also have doubts, that they would chose "mainstreets" only, that have "city" on one side only.
wouldn t you expect 4 iraqis to know at least one of the big towns they have to poll?
and the problem (as i said multiple times now, and still is being ignored by the "denialists", the problem is much smaller in smaller towns..)
Lee wrote: "Information that was even borderline related to ways to identify surveyed neighborhoods or people was simply not transmitted to him [Roberts], in order to maintain the confidentiality of people who were potentially at risk of their lives for answering the survey."
Yes, I absolutely agree that safety is paramount. No question.
**However**, that compromise brings us straight back to what I said earlier: **Given this admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists? They cannot. And so how can they put their names (in particular, as first author) on a paper when they do not know what the sample space actually was? They should not.** That is, I would think, one of the reasons (or the reason) behind the investigations.
If safety is such an issue, then be prepared to correct for any biases that might be introduced due to the safety constraint. Don't just claim they are not there because you cannot estimate them.
Sod wrote: "...the problem is much smaller in smaller towns..."
Let's assume the 'problem' Sod talks about is the uncertainty of main streets. In that case, it would be helpful if Sod could quantify his statement. What does 'much smaller' mean in numbers? Does he have a quantitative model for this, other than just words and presumption? In short, does he know what was done in terms of picking main streets? Saying that main streets are obvious when you see them, is hardly a scientific approach.
More generally, can someone offer an answer to the questions I have posed several times: Given the admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists? And so how can they put their names (in particular, as first author) on a paper when they do not know what the sample space actually was?
Silence will be taken as a 'We don't know'.
And as for the rest of Sod's statements above: "i have serious doubts, that the Lancet team was cruising around towns, searching for "mainstreets", when maps were easily available. i also have doubts, that they would chose "mainstreets" only, that have "city" on one side only. Wouldn t you expect 4 iraqis to know at least one of the big towns they have to poll?"
So, was the main street selection criterion through maps?? Or driving around?? Or asking locals for choice of main streets?? Or by choosing a fairly symmetric distribution of city on either side?? Or .... what?? You don't know, nor do I, and nor does anyone reading this thread. And nor (it seems) do the US L2 team. So, again, I ask:
How can the US L2 team (i.e. Roberts and Burnham) be so sure that no main-street-bias exists, if they had no strict consensus with Lafta about what "main streets" were (as per Kevin Donoghue's suggestion)? And so how can they put their names on a scientific research paper based on surveying, when they do not know the sample space -- and they have no way of quantifying possible biases in the sample space?
By suggesting that Roberts and Burnham didn't know exactly what Lafta did with regards the precise main-street selection, let alone the actual choices, you are shooting down L2. Kevin Donoghue started it, and now you all seem to be agreeing!
My final comment, is a direct challenge to Tim Lambert, who seemed to start this thread with the notion of shooting down the JPR 'theory' taken to mean their parameter estimates. But since n, and also the f's and q's, depend on the choice of Si and So, you are actually attacking the choice of Si and So made by JPR when estimating these parameter values. Specifically, you are attacking the JPR suggestion of what might constitute main streets, as confirmed by your focus on the Google map at the start of this thread. You think X are main streets, they think Y are main streets - and because of the differences, different parameter values emerge. No surprise.
Unfortunately for you, since the top of the thread, your fellow L2 supporters have started to offer a range of rather different methods for choosing main streets. From 'it is obvious', to asking Iraqi opinion, to using esoteric British road vs street definitions, to driving around looking for taco trucks! You yourself used Google maps, a method which is of course biased toward saying that a main street is a wide street when viewed from above (irrespective of levels of traffic etc.)
But what all this really comes down to, is "how did Lafta's team choose main streets"? I doubt very much that they looked for taco trucks, so they must have followed some other scheme. But which?
I don't know, you don't know, nor does anyone else reading this thread -- and the great revelation to me, prompted by Kevin Donoghue, is the fact that **nor do Burnham and Roberts know**. So there is indeed no point asking them. I get it now.
But now there is a very, very interesting scientific problem for you. What exactly was the sampling performed in the study that you support? How can you be so sure that no MSB exists in this sampling, without knowing anything about the MS's (i.e. main streets)? And how can you support the quantitative conclusions of a survey-based paper (L2) containing several authors (i.e. Burnham, Roberts) who themselves do not know what the sample space was?
Of course you can say "No I don't believe JPR", but believing and scientifically rejecting are very different as you know as a scientist yourself).
"LancetStudy", please tell us, what is keeping you from doing your own study in Iraq?
none of us is holding you back!
the big picture is this:
to show a problem, Spagat needs to show that the Lancet method causes a bias in households chosen (1), that these households have a significantly higher deathrate (2) and that this lead to false results of the study (3).
he has done none of these.
Let's assume the 'problem' Sod talks about is the uncertainty of main streets. In that case, it would be helpful if Sod could quantify his statement. What does 'much smaller' mean in numbers? Does he have a quantitative model for this, other than just words and presumption? In short, does he know what was done in terms of picking main streets? Saying that main streets are obvious when you see them, is hardly a scientific approach.
small village will often only have one mainstreet. it is the one, on which you entered the village. and the 40 households rule allows for more "penetration" beyond the crossstreet, in a place that only consists of 100 households.
So, was the main street selection criterion through maps?? Or driving around?? Or asking locals for choice of main streets?? Or by choosing a fairly symmetric distribution of city on either side?? Or .... what?? You don't know, nor do I, and nor does anyone reading this thread. And nor (it seems) do the US L2 team. So, again, I ask: How can the US L2 team (i.e. Roberts and Burnham) be so sure that no main-street-bias exists, if they had no strict consensus with Lafta about what "main streets" were (as per Kevin Donoghue's suggestion)? And so how can they put their names on a scientific research paper based on surveying, when they do not know the sample space -- and they have no way of quantifying possible biases in the sample space?
a mainstreet is something different in a village than in Baghdad. the strict definition that you and Spagat want, would have made work on the ground more difficult, not eassier. a "functional definition" (need enough mainstreet to achieve reasonable coverage) is fine for me.
all those criterions are fine. (even if we know now, that they didn t ask locals, this would have been a reasonable approach).
i don t know the details about most polls i read. (actually i know more about the lancet one than about many others together) with most polls, you will never know all details. (marketing or political polls keep some parts of their methods secret)
the Lancet results are in very good agreement with basically all other estimates of mortality in Iraq.
this is a huge problem, for all these bias claims.
Spagat has shown ZERO evidence, that supports a "mainstreets" being significantly more dangerous to people living nearby. he has at best done a sloppy (biased might even be a better term..) job, in showing a lack of "coverage".
I had another look at Tim Lambert's map (and the area immediately above it). I think Lambert has some explaining to do:
http://dissident93.files.wordpress.com/2009/02/msb-lambert.jpg
Sod wrote:"all those criterions [for choosing main streets] are fine"
Sure, **if** (and only if) resulting biases are then corrected (e.g. using JPR).
Sod wrote: "a mainstreet is something different in a village than in Baghdad"
Of course. So what is a "main street" in Baghdad? Tikrit? Basra? When we know that, we know Si using cross street selection. When we know Si, we know So. Then JPR can make improved estimates of n, f and q.
**Given the admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists? They cannot. And so how can they put their names (in particular, as first author) on a paper when they do not know what the sample space actually was? They should not. End of story.**
Robert, since it's past midnight in Australia, if you want a quick answer I'll hazard a guess as to Tim's meaning. Your own "junction A" suggestion seems to fit the bill. I take it that "above" refers to the map above Tim's text, since he used the word twice in the same phrase and the first reference is clearly to his map.
Thanks, that's what I thought (the road at the left going upwards through "junction A"). If that's the case, I think Tim's got an uphill struggle demonstrating that it's an "obvious" main street, since it appears to be nothing of the sort.
I said I wasn't going to post any more in the bowels of this thread, so I guess I'll leave it until Tim's sequel, MSB Debunk III: This Time it's Personal.
I've amended the text slightly, as Kevin is probably right about the meaning of Tim's wording, "one that it is off the map above".
New version:
http://dissident93.files.wordpress.com/2009/02/msb-lambert-2.jpg
What Tim has to remember as he liberally defines streets as "main streets" is that he is automatically increasing the samplable space Si such that the survey team would (by definition) have to then be able to cover these distances and enter these neighbourhoods. In the end, if n=0, this means by definition that Si covers all of the governerate. So the survey team would have to be able to go **everywhere**.
Since they did not go everywhere, by their own admission for safety reasons and need for speed, we are back in the same position. What roads were left out? In other words, what is the space So in practice for Lafta, and hence what is Si for Lafta (since it is I-So in set formalism)? Only when we know this will better estimates of Ni, No, fi, fo, qi, qo be possible.
More generally, how different is Lafta's implemented Si from Burnham and Gilbert's imagined Si? They don't know, so how can the US L2 team possibly claim R is near 1 for Lafta's implementation?
Robert, AFAICT from Google Earth it's probably on a par with Oxford Street, London or Parnell Street, Dublin - going by the fact that it remains clearly visible as I zoom out from it. Those two streets are main streets in my dialect at least. Also if the accompanying photgraphs are correctly located (can't be sure of that of course) then it's a bustling street with a mosque and other prominent buildings.
I would suggest that those who have any doubts about this use Google Earth to do a comparison with main streets they know.
Kevin Donoghue writes:
Come off it. I know rural lanes that are as visible. It looks half the width of Tim's other "main" street. The presence of a mosque doesn't make it a "main" street - or does it?
Given the admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists? They cannot. And so how can they put their names (in particular, as first author) on a paper when they do not know what the sample space actually was? They should not. End of story.
you are using the term "MSB" as if it was a long standing,we ll known and defined phenomenon. it actually isn t. MSB is an invention by Spagat, and he has brought up ZERO evidence, to support his claim, that people living in those areas have a significantly higher violent deathrate.
Tim: I hope you can appreciate how the above to-and-fro exactly sums up the main-street weakness in L2. Even your own supporters cannot define what a "main street" is after 100+ (total) posts, so imagine what the conversations would have had to be between Lafta and Burnham/Gilbert, then Lafta and his team, in order to pin it down prior to survey-day? Impossible.
Sod writes: "you are using the term "MSB" as if it was a long standing,we ll known and defined phenomenon."
Er, it has existed since October 2006 by my understanding, and has been published.
L2 has existed since October 2006 in published form.
Both are 'long-standing, well known and defined phenomenon'.
Your point is...?
Maybe Sod has raised another crucial new issue here, intentionally or unintentionally: Maybe previous/other studies of conflict mortality implicitly contained MSB as well! I hadn't even looked into that. Oops, we should. Has this cross-street-to-main-street been used before? Please let me know. It would be very, very interesting to go back and look at such cases to see what was concluded.
Remember my point is: Yes this cross-street-to-main-street method could conceivably be used, but only if an estimate of MSB is made. It is such an obvious detail to wonder about, after all, even without the MSB name and prior to JPR article. If it is near 1, fine -- but show how you obtained that estimate based on maps, taco trucks, or whatever... Just saying "R is near 1" isn't convincing (hence hundreds of postings on these sites...)
Sod wrote: "MSB is an invention by Spagat"
Er again.... That is what original research is. New results not previously thought of or published. So yes, I guess Spagat et al. are guilty of 'inventing' the term MSB -- but they are not guilty of inventing the effect. To the extent that R is not **exactly** equal to 1, the effect is generated exclusively by the L2 team's failure to examine the Lafta implementation, and adjust their estimate of mortality for any street-selection bias.
Basic scientific thinking -- even if R then turns out to be near 1, somebody has to account for such a source of potential bias during the final analysis prior to publication. Amazing that L2 apparently didn't think of doing it prior to JPR.
Robert Shone: It looks half the width of Tim's other "main" street.
And Tim's other main street looks about the same width as the one the MSB squad chose, which runs vertically at the right of their map. Actually I think the one Tim concentrated on looks a bit more prominent. As for the one you are contesting, which may or may not be the one Tim has in mind, I think it's going through the old part of the town, with the Kirkuk citadel on one side and that mosque (a pretty big one judging by the dome) on the other. It's not unusual for streets to be narrower in older parts of a city.
I refer you again to the examples I gave: Oxford Street looks a lot narrower than Marylebone Road, just to the north of it. So what? Surely Oxford Street is a main street? And Parnell Street is of course only a fraction of the width of O'Connell Street. But they are both visible relative to their surroundings - just about - when I zoom out to an altitude of 12,000 feet or so on Google Earth.
AFAICT for someone doing a survey it would be an obvious choice of main street, assuming they wanted to give as many households as possible a fair chance of inclusion. If you're assuming anything other than that, then you must think they were either dishonest or sloppy; and if you think that, what do you need Johnson et al. for?
Thanks, Kevin, I get the gist. From 12,000 feet up, you think it has similarities with Oxford Street. That makes it an "obvious" choice as a "main" street.
Your "politeness" overwhelms me, Robert. There was really no need to thank me. Just to observe your virtuosity with scare quotes makes all my efforts worthwhile.
Kevin D: The real issue has moved on from "what is a main street" to "how could Burnham/Roberts dismiss street-bias **without** knowing the streets that were samplable?"
**Let's be crystal clear on this: Since R depends on Si, how could Burnham/Roberts state R=1 effectively without knowing Si?**
**R is defined through Si !!**
In terms of the more general bias question "Is R=1 or not?", it matters little whether the scheme used involved main streets, very curvy streets, streets with the letter "s" in them, or streets with taco trucks -- or whatever. What matters is: Is the scheme that was used by Lafta (whatever that was) open to street-bias? And most importantly, how can Burnham/Gilbert say it isn't (i.e. R=1) without knowing a typical Lafta-class of streets (i.e. without knowing Lafta's Si) or even possible Si's that Lafta's scheme would have employed? Actually, you helped raised this issue yourself.
Remember, unlike a disease which tends to spread over large distances without regard to street design, violence is largely fought by people on streets. So this issue is crucial. Violence may have a systematic predominance in particular street topologies, so there is an important potential source of bias depending on street selection process.
And Sod has now raised the amazing possibility that this L2 technique has been implemented in other conflict studies. (That is very, very interesting and I would like to pursue this).
Kevin: One of the main values of the JPR paper is that is provides a concrete framework for this whole discussion. Parameters, estimates etc.
(Tim: Please take this as a personal and informal side comment, from a colleague. With all due respect, maybe you should revisit the title of your thread. Stating that something is 'badly flawed' sounds like something a lawyer would have a field day with in terms of libel. I am no lawyer, but how do you know lawyers don't read this? Since that whole investigation issue arose, maybe we should all be careful about direct accusations. Questions are fine, and valid, and important. Opinions as well, but more direct things are perhaps more dubious....)
Even your own supporters cannot define what a "main street" is after 100+ (total) posts, so imagine what the conversations would have had to be between Lafta and Burnham/Gilbert, then Lafta and his team, in order to pin it down prior to survey-day? Impossible.
my functional definition from #73 is fine. this is just one of many false claims you make!
Maybe Sod has raised another crucial new issue here, intentionally or unintentionally: Maybe previous/other studies of conflict mortality implicitly contained MSB as well! I hadn't even looked into that. Oops, we should. Has this cross-street-to-main-street been used before? Please let me know. It would be very, very interesting to go back and look at such cases to see what was concluded.
i have two answers to this:
1. this is the answer i have given before: the lancet method was chosen, because of an EMERGENCY. with GPS becoming smaller and cheaper and google maps, it most likely wont be repeated.
now ask yourself: why did Spagat chose an example for his THEORETICAL paper, that isn t a typical case and wont be repeated? one that (according to them) has multiple other problems, one among them missing vital data?!?
(my answer: PUBLICITY!!!)
2. on the other hand, you are upon something BIG. while there aren t many casualty studies using this approach, you will find thousands of example in marketing!
before telephone polling became state of the art, people did real polling, on real streets. mainstreets, mostly! according to the Spagat paper, the majority of people polled on a mainstreeet (those are the ones who would be killed by a bomb) are living just around the corner, in a street crossing that mainstreet! for decades (?) marketing decisions were based on a small special subset of people, those living with a mainstreet bias!!!
sorry for the font above. the list seems to be using some auto-function.
Er again.... That is what original research is. New results not previously thought of or published. So yes, I guess Spagat et al. are guilty of 'inventing' the term MSB -- but they are not guilty of inventing the effect. To the extent that R is not exactly equal to 1, the effect is generated exclusively by the L2 team's failure to examine the Lafta implementation, and adjust their estimate of mortality for any street-selection bias.
slow again, for you: MSB has two parts in the Spagat paper:
1. MSB meaning that Lancet concentrated on polling mainstreet areas. the argument that Spagat makes on this point, is at best weak, as Tim and others have demonstrated.
2. MSB meaning that more people LIVING in that area get killed by violence. Spagat offers ZERO evidence to make this point. none. nada. zilch.
your claim that the L2 authors should act on such a weak problem, is weak. at best.
Remember, unlike a disease which tends to spread over large distances without regard to street design, violence is largely fought by people on streets. So this issue is crucial. Violence may have a systematic predominance in particular street topologies, so there is an important potential source of bias depending on street selection process.
so you claim that diseases don t spread better, in places where many people meet?
according to Spagat, diseases should concentrate on people living along mainstreets.....
Tim: Please take this as a personal and informal side comment, from a colleague. With all due respect, maybe you should revisit the title of your thread. Stating that something is 'badly flawed' sounds like something a lawyer would have a field day with in terms of libel.
Tim is stating facts. he is attacking the paper, not making claims about the authors and fraud.
other have made much harder accusations. against the lancet authors. you must have missed this, by chance....
Lancetstudy says:
>Tim: Please take this as a personal and informal side comment, from a colleague. With all due respect, maybe you should revisit the title of your thread. Stating that something is 'badly flawed' sounds like something a lawyer would have a field day with in terms of libel. I am no lawyer, but how do you know lawyers don't read this?
for those who haven't guessed yet,
LancetStudy = ron = ozzy = Nick = Lancet Debunker = Tell
And they are all sock puppets for one of the authors of the MSB paper.
Robert Shone, the other main street I added was the one that runs through your intersection A, intersecting with your secondary roads 1 and 2.
Tim Lambert writes:
Thanks for confirming it. In that case I think your map loses all credibility, since the best you can say for one of your cross roads (road 1 in my map) is that it joins another cross road (2) at a junction (A), and that another road which you have arbitrarily designated as a "main" street runs through the same junction.
That's very weak, and I'm not surprised that you didn't illustrate this part of your selection scheme in your (arguably misleading) map.
Maps.
So although this certainly doesn't help with q or f0 or fi, looking at maps perhaps helps with n.
I've been playing around with QGIS and OSM, with some files from GEOFABRIK. It's a bit tough for a non-compsci, non epidemiologist but actually not too bad once you get into it. So far I think I can tell you that for Enfield, using the sampling process as in Burnham et al, with main street being strictly motorway, trunk or primary road, i.e about 6 roads only, N seems to be for this map around 2.5 judging by road length samplable.
Anyhow you can pull data directly off OSM, and hence for anywhere you like in the world. You can classify it and extract intersections, coordinates, lengths and such like in QGISS, and you can export files to good old Open Office or some such to do the maths.
Diseases.
Yes this is interesting. Cluster surveys have of course much earlier been used to study not only vaccination coverage, but for example diarrhoea prevalence. As you might imagine inhomogeneity is important in such cases. As you also might imaging there is a reasonable literature on what to do - to account for 'design effect', the proper term for which we have the euphemism 'main street bias'. So you'll note that Roberts et al 2004 specifically did their analysis with methodology taking into account 'design effect'. Perhaps tellingly the seminal paper from Johnson et al doesn't mention 'design effect', or the scientific discussion thereof even once.
Robert, I take it that you are no longer disputing that the big street running along the bottom of my map is a main street. So even if you don't count my second additional main street as a main street, the only change to my map is that the yellow street that clips the upper right corner goes away, resulting in a small blue area around that street and the rest of the map reachable and n at most 0.1.
And I know that you will now make a big song and dance about how this proves that the Lancet methodology is completely unknowable, but whether n = 0 or 0.1 makes no significant difference to the results of the Lancet study.
And I note that no-one is disputing my point that the blue areas should not include the 39 neighbours of the start house. This makes a huge difference and the MSB authors somehow forget to include this in their map.
I've written a new blog piece expanding on the areas which Lambert misleadingly left out of his map:
http://dissident93.wordpress.com/2009/02/22/msb-lambert-update/
If you blow up the Google map, just about every street labeled as secondary by the Sapgatti is as busy or busier than one or more of the streets they label as main. Whatever. Look at the maps
Tim claims:
Untrue! I am disputing it. Recall what L2 itself claimed:
I (and, I think, the MSB folks) interpreted this to mean that households had to be on cross streets to be in the sample. That is, at least, one reading of what "adjacent" means. Do you have proof that your definition of "adjacent" is more correct that Johnson et al's?
No. I have shown conclusively that anyone who claims to know the details of the sampling procedure for L2 is lying.
Now, your guess that the string of houses starting from house X could go down all sorts of side streets might be correct. Similarly, Johnson et al's guess that "adjacent" means on the same street as the starting house might be correct. No one knows.
By the way, although this is somewhat forward of me, I would appreciate it if Tim would start a new thread devoted to just the issue of what we know and don't know about the sampling plan used by L2. It would be helpful to gather some of the collective wisdom of L2 defenders on this point.
Tim Lambert writes:
I do dispute that it "is" a main street. But only because that's a meaningless statement. We have no universal main-street-o-meters. We have to first agree on what defines a main street, then we can make meaningful statements about whether streets fit that definition. I think the Lancet authors defined main streets as "major commercial streets or avenues", which isn't much help, as the MSB team rightly pointed out.
Perhaps the best definition for the Lancet study is: main streets constitute that set of roads which when selected as "start" roads will guarantee, via the Lancet sampling scheme, n=10. Of course, that doesn't help in their selection, unless one knows in advance which roads will guarantee that result.
n=0, I mean, ;)
"LancetStudy = ron = ozzy = Nick = Lancet Debunker = Tell
And they are all sock puppets for one of the authors of the MSB paper."
---
So at least one of the authors of the MSB paper is both fundamentally dishonest and technically incompetent.
Who would ever have guessed?
Tim Lambert wrote: "..for those who haven't guessed yet,
LancetStudy = ron = ozzy = Nick = Lancet Debunker = Tell
And they are all sock puppets for one of the authors of the MSB paper."
Er, wrong again Tim. You have misled yourself through a superficial analysis of apparent assigned IP addresses -- which, come to think of it, sounds a bit like how you have misled yourself through superficial looks at the MSB paper, and the methodology issues around L2 discussed by some of us on this thread. Hmm, maybe you should improve your detective work by focusing on what L2 actually did (or didn't do), and spend less time worrying about who might be who on this thread, or deflecting your general frustrations onto JPR's honest attempt to interpret L2's uncertain methodology?
(PS My interest in this thread was certainly started by a conversation with one of the JPR authors, no secret that. Want to guess which author it was? Or want to guess which street is a main street in city X? Or want to guess what Lafta and Burnham/Gilbert exchanged in terms of specific instructions about starting points? ...Or maybe just admit your anti-MSB campaign ran out of steam...)
Just to point it out again - the MSB authors are implicitly accusing the Lancet authors of incompetence or fraud. Burnham has said that the sampling scheme was designed to give every house an equal chance of being picked - the scheme that gives the coefficients that the MSB authors use misses that mark by a factor of more than 10. The only way they can make street assignments to get such a scheme is to assume that Burnham was lying, or that the design was an incompetent failure by more than an order of magnitude. MSB authors are accusing Lancet authors of incompetence or fraud, while attempting to not actually use the words.
And now we have LancetSTudy here, trying to explain that it only looks like "LancetStudy = ron = ozzy = Nick = Lancet Debunker = Tell", because Lambert "misled yourself through a superficial analysis of apparent assigned IP addresses." Which just happens, it seems, to be the "apparent" IP address of one of the Lancet authors.
To which I can only say: guffaw!
Perhaps the best definition for the Lancet study is: main streets constitute that set of roads which when selected as "start" roads will guarantee, via the Lancet sampling scheme, n=10. Of course, that doesn't help in their selection, unless one knows in advance which roads will guarantee that result.
no need to know everything. but it would keep them from making the lousy choices that Spagat assumes they took.
I (and, I think, the MSB folks) interpreted this to mean that households had to be on cross streets to be in the sample. That is, at least, one reading of what "adjacent" means. Do you have proof that your definition of "adjacent" is more correct that Johnson et al's?
no. but their lists of assumptions (adjanced means in the crossroad. mainstreet means city highway. the Lancet team chose only few roads and mostly those at the edge of town) basically is the SINGLE REASON, why they got the result that they wanted (n=10).
so you admit that their analysis is based on a list of completely biased and wild assumptions?!?
PS My interest in this thread was certainly started by a conversation with one of the JPR authors, no secret that.
for a person with access to the authors, your level of knowledge, the simplicity of your arguments and your behaviour can only be described as pathetic.
By the way, although this is somewhat forward of me, I would appreciate it if Tim would start a new thread devoted to just the issue of what we know and don't know about the sampling plan used by L2. It would be helpful to gather some of the collective wisdom of L2 defenders on this point.
i think it is telling, how Kane and the others are trying to shift this discussion into "we don t know anything about the lancet methodology" direction.
let me sum this topic up:
Tim and others, have shown without doubt, that by even a tinny change to the Spagat picture, you get a completely different result!
what we know or don t know about the sampling procedure, is completely irrelevant to this point. Spagat came to his conclusions, by very specific assumptions, that contradict what the Lancet authors say.
So "LancetStudy" are you really trying to argue that the identical IP addresses, writing style and familiarity with the MSB paper was just some whacky coincidence? I only checked the IP addresses because it seemed a bit odd that six new commenters who all wrote the same would all show up one after the other.
@108
edit the penultimate sentence to read - ".. of one of the MSB authors"
It's amazing what a cacophany of complaint the Lancet Study has thrown up.
From the simple "it's wrong" of George W. to the more sophisticated claims of "MSB", the common thread seems to be that of wanting it not to be true.
Tim Lambert writes:
Since Tim's usual response would be along the lines of "Shone is wrong, it is a main street", I take the above to be his way of conceding that his second additional main street was not an "obvious" main street.
Unfortunately Tim's n=0.1 claim is just as misleading as his map. If he'd included the area just above his map, it's pretty clear the value for n would be much higher than 0.1. Perhaps if he crops the top inch off his map, he can return to his n=0 assertion?
His map is fundamentally misleading, as I demonstrate here.
His map is fundamentally misleading, as I demonstrate here.
i don t like to break it to you, but your post demonstrated nothing at all.
the road at the bottom of Tim s map is obviously a mainstreet. i take your silence on that road as your admittance that Tim is right!
you make a big fuss about junctions. but (as always) i think this is just symptomatic of you, not having this well thought out. in the absence of roadsigns and names, a road ends when it ends, or when it clearly flows into another one, or when it gets smaller for a significant stretch.
the Spagat map is fatally flawed. they chose city highways as mainstreets. in the map that they use to illustrate their point, they chose streets that are on the edge of town. their choice of streets is the real bias!
Lambert has said that the road he redesignated as a âmainâ street (but which wasnât shown in his own map) was the one which runs upwards through junction A (on my map). Lambert originally described this as an âobviousâ main street, but itâs nothing of the sort (itâs half the width of Lambertâs other designated âmainâ street, for example).
The best that Lambert can claim for one of his secondary (or âcrossâ) roads (road 1 in my map) is that it joins another cross road (2) at junction A, and that the road which he has arbitrarily designated as a âmainâ street runs through the same junction.
In other words, even if a survey team agreed with Lambertâs arbitrary designation of the junction A road as a âmainâ street, itâs debatable whether (using the published Lancet sampling methodology) they would select road 1 as a cross street. Looking at Junction A, it seems equally (or perhaps more) likely that theyâd class road 2 as the secondary street, with road 1 as a tertiary road leading off road 2. But nobody actually knows, since nobody knows how the Lancet sampling scheme worked in reality.
What this ambiguity over classes of road shows, at this level of detail, is that Lambert is misleading his readers when he claims that the MSB map (which he redrew) is âwrongâ. The most he can say is that he has a different subjective designation of roads, which has its own problems in terms of plausibility.
Lambertâs map left out the least plausible part of his selection scheme. For further details, see: http://dissident93.wordpress.com/2009/02/22/msb-lambert-update/
Shone, just repeating your claims and continous links to your own article that actually doesnt address the main point, wont help you.
the road at the bottom is a mainstreet. (it is VERY similar to the other mainstreets that Spagat chose) just including that one road, changes the outcome of the "bias" significantly!
the reason for this is, that this road (in contrast to the ones that Spagat prefers to choose) has many crossroads and runs through the middle of a build up area.
your claim is, that you know the Lancet sampling good enough, to be sure that Spagat got it right down to every single road!
and you are still forgetting that Lancet idea of including enough roads to get a big sampled area...
sod writes:
Thanks - I take your point about repetition. And you're certainly the expert on that (having repeated yourself about a thousand times in bold emphasis on the gender issue in the last MSB thread).
Still, an important point is sometimes worth repeating a few times: Lambert's map is fundamentally misleading (why? See #116).
nice Shone, you managed to write another posts, without addressing that road at the bottom (making a very big difference).
and talking about gender, this reminds me of your silence on the mechanism, that kills local working age males on the street, where they are outnumbered by non-local males and in their houses (where they are outnumbered by females/kids/elderly). Spagat gives a whole new meaning to the term "precision bombing".
Tim wrote: "So "LancetStudy" are you really trying to argue that the identical IP addresses, writing style and familiarity with the MSB paper was just some whacky coincidence? I only checked the IP addresses because it seemed a bit odd that six new commenters who all wrote the same would all show up one after the other."
Just as R is more than 1, there are more than 1 persons who think your arguments are wrong and that you are driven by non-scientific goals with the L2 defence.
sod writes:
Talking of which, do you know what Les Roberts meant by the following:
Is he really saying that shock-and-awe was "careful", with "genuine" targets?
It sure looks like it.
So LancetStudy, you are not denying that you are Ron/ozzy etc.
This has been another addition of SOCK PUPPET THEATER.
Tune in next we as hear Mary Rosh say " As Herr Doctor Professor Lot complete his lecture, all the women in the lecture hall had an I'll-have-what-she's-having moment."
Tim Lambert wrote: "So LancetStudy, you are not denying that you are Ron/ozzy etc."
Yes I am denying it. Sharing common IP addresses does not mean sharing genes!
This is an interesting insight into you/your thread. More interested in working out who is who, that what is what in L2??
Tim, you've failed to consider that some IP addresses are more common than others. This is obviously because of MSB: Main Subnet Bias.
Tim Lambert seems to have form when it comes to really obsessive policing of identity, and there does seem to be a police mentality behind it - the business of "catching people in the act", which is what Tim's blog is mostly about.
He also has form when it comes to revealing the identities of posters without their consent.
Shone, two more posts, not a word on the road at the bottom of the map, changing the Spagat result significantly..
Is he really saying that shock-and-awe was "careful", with "genuine" targets?
yes. as all so often, i agree with Roberts. the spike of violence after the Samarra bombing in 2006 was more deadly for iraqi civilians, that the original US campaign.
This is an interesting insight into you/your thread. More interested in working out who is who, that what is what in L2??
Tim wrote a pretty good post about the weakness of the "stronger" part of the Spagat "analysis". you failed to address the most of it. pretty weak, for someone with access to the authors.
Tim Lambert seems to have form when it comes to really obsessive policing of identity, and there does seem to be a police mentality behind it - the business of "catching people in the act", which is what Tim's blog is mostly about.
you too, haven t taken a look at the post at the top of this topic, have you?
i have seen "police mentality" and outings on denialist blogs. this is nothing of that sort.
i think it is rather interesting to know, that all those posters share IP and that LS has contacts to the authors.
sod writes:
I've already addressed it in #3, #104, etc, but if you want the obvious stating:
1. It is what it is - a wide, straight road, busy with traffic, cutting directly through a residential area, but with buildings on the left section of it that appear to be larger than residences.
2. Obviously, you could easily classify it as a "main street", depending on your criteria.
3. There might be reasons for not classifying it as a main street, again depending on your criteria (it might not be "commercial" enough - that being one of the Lancet team's criteria apparently: see my comment #3. It's difficult to tell for certain from Google maps).
4. Assuming we classify it as a "main street", what does that say? That for that one street our corresponding "blue" area would shrink. If you cut out a little section of the map, as TL did, you could even show an area with no blue in it.
5. Does that mean n=0 for the whole of Iraq based on the Lancet sampling scheme? Clearly not. Tim misleads a little here (and when he adds his second "main" street, which he doesn't show, he misleads a lot with regard to his n=0 claim).
6. Tim's map is fundamentally misleading (see #116).
It was hardly Tim's aim to show that "n=0 for the whole of Iraq based on the Lancet sampling scheme." He does not claim that in his post. What he does show is that the map presented by Johnson et al. is wildly misleading. The point he is making is clear from his title: the JPR paper is badly flawed.
What Tim actually says is:
That's totally misleading. The unsampled area is not 0 in "their map" - it's 0 in Tim's tiny portion of their map (and it would be even tinier if we dismissed Tim's second correction).
Robert S et al.
Surely the real question is not whether n=0 or not, but whether a reasonable selection of main streets will lead to a bias that is way outside the confidence levels (i.e. the estimated accumulation of inaccuracy) that the Lancet team calculated?
In this case all this talk of "misleading" because n may or may not be actually zero is a complete non-starter, and pretty much irrelevant.
The point that I see here is that with any reasonable definition of what main streets are (and by the way, someone with any kind of local knowledge would not have any trouble at all identifying a main street to start off from, which renders much of the linguist wrangling pretty much irrelevant too) it's extremely unlikely to come up with a bias anywhere near the size that the Johnson, Spagat, Gourley team come up with.
You and I have discussed this before, and I was one of the early folks who drew up a few tables to analyse the sensitivity of their model to changes in the parameters (in fact I even suggested to you that the authors include such an analysis in their paper - I'm glad they did), and what I found then was that the bias is pretty sensitive to the value of the parameters that are selected.
N doesn't have to be zero, then. All Tim has to show (and has shown to my mind) is that it's pretty small for any reasonable understanding of what is a main street. The sensitivity analysis that I have already done shows that the bias drops off rather rapidly to 1 (i.e. only a small likely bias if any at all). Also notice that as we move out of the large cities to smaller ones, the definition of main streets etc tilts the balance away from MSB even further.
And all of the above is giving the MSB paper a huge benefit of the doubt. In fact I am pretty unconvinced (as I was the last time we spoke about this) that their dynamics accurately capture anything close to the reality of either the pattern of violence or the daily movements of Iraqis, thereby rendering it an interesting but purely academic example.
cheers
Aly writes:
Unless you know more about how the Lancet sampling procedures worked in reality, most of your statements, like the one I quote above, are nothing more than unsupported assertion or wishful thinking. We've been through this at great length in the above thread.
You're right that n doesn't have to be zero. However, Tim misleadingly suggests that it is zero for "their map".
Robert
No, my comments are based on the sensitivity analysis of the parameter space (something it seems I thought of before the authors themselves) and Tim's analysis of the map above.
It's pretty easy.
The question is how does one have to twist and turn to come up with a sensational bias, and the answer is becoming increasingly obvious that you have to twist and turn quite a lot.
Again, you say:
"However, Tim misleadingly suggests that it is zero for "their map"."
Again I say this is irrelevant and just a way for you to try and obscure the real question. Even you take Tim's second main street out, the bias still drops to something near 10-20% or well within the confidence interval in the Lancet paper, thus proving nothing at all.
Cheers
Aly wrote: "The real question is ... whether a reasonable selection of main streets will lead to a bias that is way outside the confidence levels (i.e. the estimated accumulation of inaccuracy) that the Lancet team calculated"
No Aly, the **real** question is: What on earth did the Iraq survey team actually do? What did Lafta, or Lafta's team, actually do on each survey day? More correctly...
**What did Lafta's team members do on each survey day to guarantee that there is no street bias? Apparently Burnham and Gilbert do not know (they have admitted they do not have the details) so how can they possibly make any valid statements dismissing street-biases?**
Not only do none of us know, but we are being told that **Gilbert doesn't know, and nor does Burnham**. And that is what all the fuss **should** be about in my humble opinion.
Now, you might (and have already) hazard a guess at what Lafta's team did -- and so might Tim et al. And so might the rest of us, including Gilbert and Burnham.
But here is the problem:
**Did Gilbert and Burnham put their name to a survey paper, as principal authors, where they did not know the exact survey methodology used?**
Whatever the rest of us on this thread are guilty of (e.g overstatements, moments of emotion etc.) none of the rest of us did that. So, therein lies your problem (as an L2 defender) and their problem as L2 authors. This is undoubtedly why they are being investigated, again in my humble opinion. The rest of this MSB topic is simply various academics attempts to make sense of little (and seemingly contradictory) information. Don't like it? Fine. Opinions are healthy, of course when backed up by a quantitative argument.
But how come you don't bother to ask yourself the questions in bold above?? I'll repeat them:
Question 1: **What did Lafta's team members do on each survey day to guarantee that there is no street bias? Apparently Burnham and Gilbert do not know (they have admitted they do not have the details) so how can they possibly make any valid statements dismissing street-biases?**
Question 2: **Did Gilbert and Burnham put their name to a survey paper, as principal authors, where they did not know the exact survey methodology used?**
OK Tim, you are a computer scientist: Give us all a working algorithm that can be implemented in a day, for the survey team, which guarantees R=1 plus/minus 20 percent, *but* also taking into account all 'ifs' and 'buts' related to adding in nearby streets etc. etc. (i.e. must be consistent with everything Burnham/Gilbert claims is there).
Oh, and write it in a way such that when translated from English it does not create problems, inconsistencies, ambiguities in interpretation.
Oh, and make it fit on one page such that it can be memorized.
Oh, and make it very simple so that it can be implemented without potential error.
... and I nearly forgot, make it work for every town, city etc. that is sampled throughout the country.
An interesting homework for you. We're all waiting....
(Of course, even in the infinitessimally remote chance that you come up with one and somehow prove in principle that it works, you have no idea if it is *the* one that was used..... Probably isn't ;-))
[BTW, there is no suggestion here that Lafta and his team did not do the best job they could. I am sure they did, and that they did something truly heroic. The issue is entirely different: If Burnham/Gilbert do not know the scheme used *exactly*, how can they make any claims about street bias being unimportant?]
LancetStudy I followed the scheme as described in paper on the map that the authors presented when arguing that n=10 was plausible and got n=0. Yes there are ambiguities in deciding what happens at complex intersections but they don't make much difference since there are usually multiple ways to get to a particular house. Even Robert Shone can't seem to get past n=0.2 on that map.
No-one is disputing that the L1 scheme using GPS gives a better sample, but this makes little difference. In order to conceal this the MSB people present a misleading map that they don't include in their paper but merely reference.
Tim, now you really are trying to be misleading. You say "I followed the scheme as described in paper on the map that the authors presented when arguing that n=10 was plausible "
Scheme? The MSB authors are *not* presenting a *scheme*, ... that is what L2 should have done. They are the samplers, after all. But they didn't -- so, I am afraid, MSB authors are within their rights to present what they see as plausible.
The JPR-related maps are not meant to be taken literally to the street since **nobody** knows what the actual streets are. They are just plausibility arguments -- and reasonable ones at that in my opinion.
I wonder what Lafta would say, if faced with the Google map? Probably not a lot, since maybe he didn't have a map. We just don't know....
So sorry, my homework assignment for you still stands. Instead of saying what didn't happen, tell us a plausible argument of what did happen. (PS Have you checked the danger-element on these maps, such that scheme needed to be adapted on-site as suggested by Burnham/Gilbert? No, unfair question. Impossible to know)
Again, if you want to argue that n=10 is plausible you need to present a map with n=10. You haven't.
Tim Lambert writes:
Misleading again, Tim - tut tut. By "that map", you mean the tiny portion of it which suits your particular needs.
Try working out the value of n for the whole map, keeping your first additional main street, but forgetting the second one (for reasons I gave in #116).
It'll be less than 10, obviously, but much higher than 0.1 or 0.2.
Tim writes: "Again, if you want to argue that n=10 is plausible you need to present a map with n=10. You haven't."
I haven't, since I haven't tried. But the JPR team did.
Since you define main street in particular way, you get your n. They define it in their way, estimate n=10 (which follows pretty much from the figure at the top of this thread by the way). But the big, big question is:
HOW DID LAFTA DEFINE IT TIM?
Get it? How... did... Lafta... define... it?
By the way, Tim, you have not checked that your 'main street scheme' is in principle feasible in terms of coverage for a survey team in the allotted time. To make your scheme plausible, you need to show that the extra Si that is introduced (as compared to the JPR choice) across all the survey areas, will still enable the team to make it round a random sampling of these Si's within the allotted time. Selecting what would be 'nice' for an Si, and then implementing it as a survey team in the allotted time, are two very different things. I think this is something JPR had in mind when thinking about the practicalities of the whole 'main street' scheme of L2 -- and is clearly(?) what US L2 and Lafta must surely have thought through (or not??). Saying that large areas are in Si, and then not actually sampling them, is equivalent of course to a Si which is small and biased.
Anyway, I notice you haven't answered my two questions (maybe because you are doing that homework I set you? Well, please drop it one minute, and just answer this:...):
Question 1: What did Lafta's team members do on each survey day to guarantee that there is no street bias? Apparently Burnham and Gilbert do not know (they have admitted they do not have the details) so how can they possibly make any valid statements dismissing street-biases?
Question 2: Did Gilbert and Burnham put their name to a survey paper, as principal authors, where they did not know the exact survey methodology used?
You can leave Question 1 to later, just answer 'Yes' or 'No' to Question 2. Please. Now....
Lee, Jody, Sod, Kevin D., anyone???
Another quick question for Tim, .....sort of a recap of the same old obvious, obvious question:
Take the map at the top that you pulled from the MSB site. Estimate n given *their* choice of main street. Though not cherry picked for any particular value, n=10 is certainly plausible.
You, as you have said, would have defined main streets differently, and got a smaller n.
What did Lafta do? (a), (b) or some unknown (to us) alternative (c)?
Answer: Not (a) in your opinion. So was it exactly (b)? And your proof that he did is...?. More likely (c), right Tim?
[PS. Thanks Tim, this thread is definitely fun... I was told it would be, and it is....]
Time wrote: "LancetStudy I followed the scheme as described in paper on the map that the authors presented when arguing that n=10 was plausible and got n=0. "
(Just saw this...)
Drivel, Tim. Pure drivel.
Got to hone those experimental skills. Computer science won't do that for you mate...
One more fucking time, LancetStudy, and all your various sockpuppets.
Lafta and the rest of the authors defined main streets and secondary streets such that every house in Iraq had an approximately equal chance of being surveyed.
They said so. That has been quoted to you. I've repeated it several times.
Only if one ignores this statement (or acknowledge it and assume Lancet2 to be either massively incompetent or fraudulent), and only if one ALSO goes out of one's way to exclude obvious main streets from main street category, and ALSO does so in a way that is clearly designed to exclude massive quantities of homes from the survey - only if one does all that, can one get a value for N that has any appreciable impact on the Lancet2 analysis.
It appears that the authors together designed a strategy. Lafta implemented the strategy. Ethical confidentiality makes it unethical to release all elements of the designed strategy. Ethical confidentiality makes it unethical to release any elements of the implementation.
Such ethical constraints are unfortunate reality when one is doing survey work that can bring a risk to the people being surveyed. They also open the authors up to completely unsupported and ludicrous charges such as the ones you dismal f***** are flogging to death here - because you all know that they cant defend themselves, because they are ethically constrained to not release the information you ratf***** are demanding.
It is despicable stuff you are doing - and of a kind with the sock puppetry y'all are engaging.
and to address yet another of LancetStudy (et al)'s scurrilous insinuations-but-not-actual-accusations.
I can get off the freeway and drive into a town of 5,000 people, and in 5 minutes identify and understand the layout of every main street in the town - with main street defined as any street with a reasonable probability of some kind of commercial activity on it. I know I can do it, because I do it routinely when trying to find those taco trucks I mentioned above - which are often parked on obscure but clearly commercially-identified 'main streets.'
I can do the same thing in 15-20 minutes in a town of 25-35k people. I know this because Ive done the experiment.
You on the other hand, LancetStudy (et al) are talking out your ass on this, and using your ass-wind to insinuate fraud or incompetence - and again, it is despicable behavior on your part.
As much fun as this thread is, we now know for a fact that Burnham/Lambert have been lying for more than two years about whether or not individual data was collected. (It was, they said it wasn't.) Given that, how can Tim (or any other Lancet supporter) rely on their description of the sampling plan?
Just curious.
By the way, Lee huffs about "insinuate fraud." We know that there was fraud. Hopkins just told us. The only remaining disagreement is over the extent of the fraud.
The expletives are out, so I am off this thread until someone provides a proper answer to my two questions (see above in this thread).
Until then, I just cannot resist replying to Lee's comment:
(1) "Lafta and the rest of the authors defined main streets and secondary streets such that every house in Iraq had an approximately equal chance of being surveyed."
Oh dear, back to square one: This is a *desired* outcome, not a statement of the actual selection probabilities following their choices of the samplable space.
Personally, I would bet a large amount of money that Lafta and his team did indeed do a great job subject to the constraints. And, if properly corrected for systematic bias, the numbers might be very, very useful. I repeat: A little analysis by the US L2 team along the lines of R estimate (or design whatever...) *could have* made his estimates really valuable. But they didn't. Or if they did, they are not telling how. I wish they would, since I for one do not see how it could be done accurately without knowing the main street selection that he used.
(2) Lee, I actually have something favourable to say about your taco truck scheme. At least it is a scheme. In other words, if we all knew the schedule of the taco trucks, and the day of the survey, we could then simulate what houses would have been sampled in a given realisation of the sampling in a given (portion of a) city. Then we could repeat the simulated sampling, and we could build up a profile of probability of being picked for each house. Not necessarily related to main streets (whatever they are) but at least well-defined, and repeatable.
To the extent that taco tricks move around randomly, we might then (on a good day) even remove bias tied to particular streets. And your scheme would translate into other languages pretty well as well. So well done.
However, it is irrelevant. It is the *actual* L2 scheme that we all need. You know, the one the survey team actually used. Actually Used Every Day In Iraq.....
I just read David Kane's message about the censure and then Tim's other page. Thank you for letting us know, David and Tim.
So now we know that at least that data existed post-survey. I do feel sorry for Lafta, he must feel let down. I don't know anything about the Burnham/Gilbert split, so I have no comment on any of that.
But what this probably means is that in principle some independent, assigned person (overviewed, of course) could reconstruct where the sampling actually occurred. With some funding etc. that would probably be useful. Conflict mortality estimates through epidemiology could be a very powerful scientific method, if properly applied with correct estimates of street-bias etc. It would be a shame not to push it forward properly. In fact, we probably all agree on this.... That way we would all really learn what MSB is or isn't about in practice, and a lot of things we don't even imagine possibly. Maybe MSB gets cancelled with some other effect? Or enhanced? Or essentially zero in certain places and large in others? etc.?
Who would fund it though, who would do it, and do enough people want it to happen. MSB or not MSB...that is the question.
My apologies: I have made the typo Burnham/Gilbert in several places instead of Burnham/Roberts....
Burnham/Gilbert ---> Burnham/Roberts under all my entries
LancetStudy(and various sock puppets) said:
"This is a desired outcome"
Bullshit. "There is no polite way to say it."
They didn't say it was their desired outcome. They said it was the criterion upon which they designed their scheme. Those ar ehugely different statements, and the fact that you keep misrepresenting it is telling. Especially while deflecting nearly every substantive point in my post.
Also LancetStudy et al:
"could reconstruct where the sampling actually occurred. With some funding etc. that would probably be useful." and "It would be a shame not to push it forward properly. In fact, we probably all agree on this."
So, Roberts was involved in a serious ethical mistake, and LancetStudy here proposes to amplify and extend that ethical mistake into an intentional greater violation of ethical constraints.
How -ethical- of him.
Given the admitted lack of information flow (for safety reasons) between members of the L2 team themselves, how can the US L2 team (i.e. Roberts and Burnham) be so sure that no MSB exists?
Surely the answer to this is the simple one that Tim has given:
"Because main street bias only exists in a small number of likely unrepresentative corner cases of sampling schemes, like the one that Spagat, Johnson & Gourley picked - for most definitions of "a main street" it is small or nonexistent".
In the end, if n=0, this means by definition that Si covers all of the governerate. So the survey team would have to be able to go everywhere.
Since they did not go everywhere, by their own admission for safety reasons and need for speed, we are back in the same position
This is an absolute 100% cast-iron howler, which frankly calls "Lancet Study"'s credibility into question on every single other point. Having every street included in the potential sample space (n=0) obviously does not mean that the survey team would need to "go everywhere". The survey team would sample exactly as many clusters, whatever the size of the sample space. If you don't understand this, you really don't know what you're talking about. If "Lancet Study" is actually one of the Spagat et al authors, then I profoundly hope this was a silly mistake born out of lateness and anger, because otherwise it's really embarrassing.
The consistent inability/refusal of the Spagat et al party to distinguish between "people killed on main streets" and "people kiled whose houses are on main streets" when making their arguments about the "topology of violence" is also embarrassing if it's an inability and irritating if it's a refusal.
As much fun as this thread is, we now know for a fact that Burnham/Lambert have been lying for more than two years about whether or not individual data was collected.
ah, David the defender.
But what this probably means is that in principle some independent, assigned person (overviewed, of course) could reconstruct where the sampling actually occurred.
again, for a person with access to the authors of the Spagat paper, your proposals are pretty illiterate.
Only if one ignores this statement (or acknowledge it and assume Lancet2 to be either massively incompetent or fraudulent), and only if one ALSO goes out of one's way to exclude obvious main streets from main street category, and ALSO does so in a way that is clearly designed to exclude massive quantities of homes from the survey - only if one does all that, can one get a value for N that has any appreciable impact on the Lancet2 analysis.
very good sum up, Lee.
i expect the Spagat supporters to prefer a discussion about the Burnham subject. it will allow them to continue to ignore, how adding a single street completely changes the outcome of the Spagat paper for an area. and as i have said repeatedly and as Lee supported with his "taco example", the problem is much smaller in smaller towns anyway.
LancetStudy
this is pretty much my last post on this, as we have gone round in circles about this for a while. I have certainly discussed this in the past ad nauseum and have no appetite to restart the whole discussion.
You say:
With respect I disagree. You are including a number of hidden assumptions in this question and I disagree with almost all of them. A couple of obvious assumptions are:
1. There would be street bias (I presume you mean MSB) be default. I disagree. There might or might not be, and this remains to be shown. As interesting as the paper by Johnson, Spagat et al. is, it is not proof that the thing called MSB exists in Iraq, i.e. there is no evidence that given the actual patterns of violence in Iraq, one has a greater chance of being killed by violence if one lives off a street that is close to a main street.
2. That Lafta et al. did not do what Burnham said they did, perhaps with some variation here and there. Again, I disagree. I think there is no evidence to suggest that Lafta and co. did more or less follow the instructions from Burnham, and unless there is some strong evidence to the contrary I see no reason to question that.
A question (which is essentially a rephrasing of the one I asked Robert Shone, and to which he did not really bother to respond) that one might then ask is:
If Lafta and his team deviated now and then from the methodology, and assuming that some streets were mis-characterised (whatever that might actually mean) as main streets, or not main streets, how likely is it, assuming that MSB actually exists that a large bias was introduced?
I submit that based on the sensitivity analysis it is quite unlikely, even in the best case for the MSB team, that is assuming that their model actually captures something of the reality of the situation in Iraq. By the way, and for complete disclosure, I don't accept that the MSB formula is even close to capturing the reality of the violence in Iraq, but I am assuming it does for the sake of argument.
Cheers
Aly, to address your point no. 1: Burnham/Roberts have specifically stated that they made efforts to "to reduce the selection bias that more busy streets would have." (see #13)
In other words, the "hidden assumption" you refer to (that there is a bias that needs to be reduced) is already accepted by Burnham/Roberts.
"... The survey team would sample exactly as many clusters, whatever the size of the sample space..."
Yes, yes, yes.. of course. Goodness! Not born out of lateness or anger, pure laziness in description. OK, if you want to be pedantic: Yes I agree, but after many realizations they would have found themselves at some stage penetrating deep away from busy streets. Like drawing numbers from a distribution that in principle goes to infinity, they would in practice draw houses tucked away deep in a neighborhood.... if Si is very large.
So when I say "they would have to go everywhere', then it means that after many realizations of their scheme, eventually they would have to penetrate deep to such a house. Hence 'everywhere' as opposed to just mainly next to main streets. Get it?
Based on their scheme of selection, this simply would never happen. Please lets not get into a pedantic debate about epsilon --> 0 in continuous probability distributions as opposed to likelihoods in finite realizations. I don't mind, but it is boring and obvious and I certainly ain't going to type out a whole lecture course for you.
So if Si is large, eventually after many realizations they would probe 'everywhere' (i.e. every type of house location).
So in practical terms of running a survey, do this throughout Iraq chosen clusters and you are out of time (and into areas they seemed to want to avoid for reasons X).
Robert Shone re-iterates a key point: "the "hidden assumption" you refer to (that there is a bias that needs to be reduced) is already accepted by Burnham/Roberts". Excellent point.
So L2 defenders, please quantify what exactly is the 'magic ingredient' that US L2 seem to have added to the algorithm in an ad hoc way?
(PS Tim doing the homework? And what about the responses to my two questions?? SIlence is golden guys......)
"..Roberts was involved in a serious ethical mistake, and LancetStudy here proposes to amplify and extend that ethical mistake into an intentional greater violation of ethical constraints...."
Nonsense.
I am sure that conflict epidemiology could be a quantitative scientific method, if (and only if) street-bias is quantified a priori for the particular street-selection protocol. Simple as that.
...homework...
How is the QGIS analysis goig on your side?
Jody write: "..How is the QGIS analysis goig on your side?
..."
Fine thanks. What about the simple yes/no answers to my questions?
Robert and LS
Let me make my points a little clearer.
1. You are both assuming that the mechanism by which the streets were chosen (as described by Burnham et al.) will lead to a street bias by default that needs to be further considered and reduced.
I disagree. So far I have seen no evidence this is the case and I think Tim's analysis of the maps above shows this pretty conclusively.
2. How large (in the case that I am wrong with point 1 above) is the street bias should it prove to exist and is characterised by the formula in the MSB paper likely to be?
By looking at the sensitivity of the model's parameter space, it seems that it will likely be small and hence not change the confidence intervals of the original study.
So far you have contributed nothing new to what we already could see from the L2 paper.
3. How likely is it that the MSB model accurately captures the dynamics of the actual violence in Iraq and is actually more than a meaningless toy example?
Until I see some attempt by the authors of the MSB paper to show, somehow or another that their model accurately captures some of what we are seeing on the ground, through calibration or analysis of patterns of violence or pretty much anything other than what amounts to accusatory hand waving, whilst cherry-picking parameter values that lead to sensational bias estimates, I consider their model to be an interesting but useless toy. Akin to the legendary "spherical cow" in my undergraduate physics lessons.
I hope this is clearer.
Cheers
Data recorded by Lafta is X. It then gets passed through a signal processing box Y by US L2 team, producing result Z (i.e. 650,000 plus/minus...). Hence Lancet article.
Process Y accounts for MSB, deduces R is near 1, and discards any possible effects of MSB.
What is Y?
Spherical cow models can be useful for highlighting issues. They will never explain where to hang a bell on a cow, the colour of a cow's eyes etc. But they can help give answers to other questions (e.g. rate of heat loss etc.). And at the very least they highlight issues, provide a basis for discussion, and pinpoint some parameters (not their values necessarily, but the embodiment of the parameter's meaning). And I guess some people just don't like those kind of models. Apollo 11 landed on the moon with spherical cow arguments. 747's fly with similar arguments. Etc. etc.
Actually, personally I would have classified JPR as one step beyond mean-field theory (i.e. first level beyond an assumption of uniformity). That's all. No more, and no less. Certainly limited, but not 'flawed'. Just one level beyond the uniformity assumption which (unless Y is a magical algorithm) is what L2 implicitly assumed or hoped.
Robert in #154: "the 'hidden assumption' you refer to (that there is a bias that needs to be reduced) is already accepted by Burnham/Roberts."
I alluded to this much earlier when I pointed out that one of the goals of the study design was to eliminate geographic bias. Jody also pointed it out, noting that surveys must be designed in such a way as to account for "design effect."
It is your assumption, Robert, that surveying on main streets would increase the death toll. As sod and others have pointed out repeatedly, that may or may not be the case. Burnham et al made no such assumptions about how it would bias the results. You are implying that their results are probably inaccurate, because they attempted to design a scheme that would make them accurate. Do you see the problem with that argument?
Bruce wrote: " surveys must be designed in such a way as to account for "design effect.""
Yes. Design is statement about what is planned to happen: W
Then there is what actually happened: X
This is fine *if* there is then a clever algorithm Y on the US L2 side, which can discount ways in which X might differ from W (e.g. possible street bias in X).
So Y must be able to assess it, and quantify any difference approximately, before being able to discard it to produce inference Z.
So what is Y?
Bruce Sharp writes:
No. I'm saying they claimed they used (unpublished) procedures which reduced MSB to zero, and should disclose the details - regardless of my (or any one else's) speculation over the value of q.
Robert, if the selection of main streets was sensible, then the amount of space that cannot be surveyed is neglible, and q -- which you cannot quantify anyway -- doesn't even matter.
As far as I can tell, all the fuss over "MSB" can be reduced to: "The L2 team made poor main street choices, perhaps because they were clueless, or perhaps because they are dishonest. Here is a long paper with some equations where you can insert arbitrary parameters to change the L2 estimates to a number that you'll like better."
Everything else is window dressing.
Based on their scheme of selection, this simply would never happen
No, as Tim has explained. Based on Spagat et al's proposed scheme of selection, this would never happen. Based on other selection schemes very similar to Spagat et al's (like the one Tim proposes), it would happen. In other words, in all except a small number of seemingly unprepresentative cases, it would happen. This is the point of the map examples above. They show that the JHU team would have had to have been very unlucky to have happened upon a scheme that would have excluded large proportions of houses from the sample space.
So if Si is large, eventually after many realizations they would probe 'everywhere' (i.e. every type of house location).
So in practical terms of running a survey, do this throughout Iraq chosen clusters and you are out of time (and into areas they seemed to want to avoid for reasons X).
This is not very comprehensible. The first paragraph is simply the statement (admission?) that if a sensible scheme is devised, all houses in Iraq are in the sample space. "Many realisations" is irrelevant here - there was only one survey carried out.
The second paragraph I don't understand at all. "[D]o this throughout Iraq chosen clusters and you are out of time" just describes the fact that for obvious resource reasons, the JHU team carried out one survey, rather than thousands. "[A]nd into areas they seemed to want to avoid for reasons X" just seems to assert that the JHU team would have intentionally designed a sample scheme so as to avoid houses far from main streets. Which a) they didn't, and b) the idea that houses a long way from main streets were more dangerous to sample than houses next to main streets, is obviously inconsistent with the central proposition of the Spagat et al paper, which is that houses a long way from main streets are less dangerous than houses next to main streets.
Bruce at 165:
You left out a critical part in your summary. I'll add it here in bold:
Based on no evidence whatsoever, we are certain that... "The L2 team made poor main street choices, perhaps because they were clueless, or perhaps because they are dishonest. Here is a long paper with some equations where you can insert arbitrary parameters to change the L2 estimates to a number that you'll like better."
LancetStudy
Agreed. They can be useful, but not when they are simply used as Spagat et al. did to cherry pick a particular set of parameters and come up with a sensational bias. In that case they are only useful to piggyback off an actual good and well known study to get extra publishing points for their respective university departments. They are useful to try and talk about what a reasonable set of parameters might mean, and how likely these are. This is not what has been done so far, more's the pity.
This is rubbish and you know it. To compare the MSB model with the same models that fly aircraft and got Apollo 11 to the moon is fantasy.
Aircraft manufacturers and designers run thousands of simulations and tests and sensitivity analyses (plural) and callibrations and and and and...
This is how they can be sure that their models represent something more than nonsense combined with hot air. This is what separates an actual model from a meaningless toy (MSB).
If the authors of this paper were serious about this (and I hope you are not one) then they need to do some kind of work to show that their model callibrates to reality somehow. There are a hundred different ways they could do this, and they have done precisely nothing except draw random marks on random maps.
Poor effort.
Aly wrote: "...they have done precisely nothing except draw random marks on random maps...."
Oh dear, Aly. Like your friends, I see that your logic is full of the waste that a spherical cow emits......
...OK Aly: I want to be kind to you: We now know that you don't really have much of an idea what a model is. Maybe you have never done any research, never been to graduate school, not even some kind of research project?? Anyway, for your education/information, MSB is typical of a first-approximation model in a problem.
But the good news is that you can redeem yourself, here and now in 10 minutes. Just answer a simple question, which of course you know the answer to (otherwise, how could you be so sure of yourself?). Here it is:
Data recorded by Lafta is X. It then gets passed through a signal processing box Y by US L2 team, producing result Z (i.e. 650,000 plus/minus...). But somehow process Y accounts for MSB, allows the deduction that there is no street bias in X, and hence allows L2 to discard any possible effects of street bias. So what on earth is this marvellous Y data procedure?
Lancet Study ( and your various sock puppets):
You keep asking what "marvellous Y data procedure" ... "allows the deduction that there is no street bias."
That is a deeply disengenious and dishonest question. The fact is that THERE IS NO EVIDENCE THAT THERE IS ANY STREET BIAS!!!!!
You guys are making this shit up.
N does not equal 10. Or anything close to it. As I've aid over and over now - the lancet authors stated as a design criteria (not a desired goal, but a criteria upon which they based their sampling design) that every house in Iraq should, to use your language, be in the sampling frame.
Only if one ignores this statement (or acknowledge it and assume Lancet2 to be either massively incompetent or fraudulent), and only if one ALSO goes out of one's way to exclude obvious main streets from main street category, and ALSO does so in a way that is clearly designed to exclude massive quantities of homes from the survey - only if one does all that, can one get a value for N that has any appreciable impact on the Lancet2 analysis.
Stop being despicable.
To Tim and friends. I have a new question:
**Data recorded by Lafta is X. It then gets passed through a signal processing box Y by US L2 team, producing result Z (i.e. 650,000 plus/minus...). But somehow, this mysterious process Y allows the deduction that there is no possible street bias in X, by quantifying it as negligible using some unknown quantitative model, and hence allows L2 to ignore even mentioning the possibility of street bias in the Lancet article. So what on earth is this miraculous Y data procedure?**
Oops, it is the same question as always. You never answer it, and yet it addresses what makes L2 valid or not scientifically. So maybe you should quit snooping around IP addresses and address this apparently simple question. Or will the Lancet erratum to do so?
Lee claims there is no evidence for street bias. There is certainly no evidence that there is elephant bias, or taco bias etc... But that is because elephants and tacos were not involved either in the design, survey implementation, or the violence itself, as far as I know.
However, streets were involved in all 3. Jody tells us that the design would have incorporated any STREET bias effect (in some way that she does not explain, or know..), the survey involved people walking on a STREET to houses on one STREET or another, and the violence occurred on a STREET. And the common word is...? Yes, STREET.
So the doubt about street bias is reasonable, and the issue should be addressed by the scientists which dismiss it. Just saying 'no, it doesn't exist' is not a scientific argument.
...possibility of street bias...
So presumably as I'm not an epidemiologist I am incorrect when I read:
'The SE for mortality rates were calculated with robust variance estimation that took into account the correlation between rates of death within the same cluster over time. The log-linear regression model assumed that the variation in mortality rates across clusters is proportional to the average mortality rate; to assess the effect of this assumption we also obtained non-parametric CI's by use of bootstrapping. As an additional sensitivity analysis we assessed the effect of differences across clusters by extending models to allow the baseline mortality rate to vary by cluster'.
As being some statement of the efforts spent trying to discern systematic error?
So presumably the difference between rural, small town and city clusters generated by 'main street bias' is not detectable in this way, and further 'main street bias' leads to a level of intra-cluster correlation that is also undetectable by the above process?
Jody writes: "so presumably as I'm not an epidemiologist I am incorrect when I read:..."
1. Not incorrect when you read it. But incorrect to link that phrase to streets. It talks about sampling on the scale of clusters, not the streets within a cluster.
2. ".. street bias' leads to a level of intra-cluster correlation that is also undetectable by the above process..."
Yes, correct.
**What quantitative analysis did Burnham/Roberts do, either post-survey or pre-survey, to correct for possible street bias during the survey?**
Again....
What quantitative analysis did Burnham/Roberts do, either post-survey or pre-survey, to correct for possible street bias during the survey?
street, **street**, _street_, STREET....
Surely the answer to Lancet Sock's question is:
"The inherent unlikelihood of any such bias existing in the first place".
Now that it's answered, perhaps you could post under your real name please?
dsquared wrote: "...inherent unlikelihood ..."
What???
One again, LancetStudy (and your army of mousy sock puppets) - bullcrap!
"Only if one ignores this statement (or acknowledge it and assume Lancet2 to be either massively incompetent or fraudulent), and only if one ALSO goes out of one's way to exclude obvious main streets from main street category, and ALSO does so in a way that is clearly designed to exclude massive quantities of homes from the survey - only if one does all that, can one get a value for N that has any appreciable impact on the Lancet2 analysis."
See that? Ive posted it several times now in response to your spammed-all-over straw man question.
If N has a value much smaller than 10, then street bias does not alter the Lancet2 conclusions in any appreciable way.
This is true also of several other of their parameters, which have been adressed in other threads.
Lambert's post that started this thread is an answer to your oft-spammed question. Lancet 2's authors statement that a sampling design parameter was that every house in Iraq be in the sample frame, ADDRESSES YOUR F****** QUESTION!!!!!!!!!!!!
What???
Are you slow or something, LancetSock? As has been shown above, main street bias only exists in a small number of seemingly cherry-picked and for the most part obviously incorrect schemes for identifying "main streets", and even then only in large towns. As Tim showed above, even in the best-case example chosen by Spagat et al. to illustrate their point, it only existed thanks to a really quite perverse decision to fail to identify a particular road as a main street. From this I conclude that the vast majority of road-identification schemes which a competent researcher on the ground would choose, would not lead to material street bias. Hence, the fact that it is very unlikely that the sampling space was distorted by main street bias, is good grounds for assuming that the sample is not distorted by main street bias.
Now could you use your real name please?
You guys get personal very quickly don't you. So be it...
Since Lee likes cross-postings, here is mine:
Lee writes: "If N has a value much smaller than 10, then street bias does not alter the Lancet2 conclusions in any appreciable way..."
Putting Tim's incorrect analysis to the side for one moment, the question is: Why didn't Burnham/Roberts do a street-bias analysis? Where did they do it? Do they reach the same conclusions? If so, how did they manage to do that prior to JPR, if they didn't have the JPR formula? Did they derive the JPR formula and plug in numbers? If not, what formula? What are the assumptions? What numbers did they plug in? Did they use a map? Which map? Which streets did they pick as main streets? The streets that Lafta generally picked?
Questions, questions.... 'irrelevant' questions? No.
Lee, my friend, invite dquared to grab a taco from one of your main-street-defining-taco-trucks, and read the following out loud, together, very, very slowly while munching...
The survey involved people walking on a STREET to houses on one STREET or another, and the violence occurred on a STREET. So how, where and when did Burnham/Roberts manage to quantify and discard any street-bias effect?
... and that concludes my side of the proceedings.
**Outcomes**
1. possibility of street bias is reasonable, and given the lack of details from L2 and apparent communication issues within L2, sounding more likely to me every day
2. JPR model reasonable. A good first-approximation model.
3. JPR estimates are plausible, reasonable even. Above all, their invite to readers (in the JPR abstract) to suggest their own values is honourable and honest
4. L2 does not contain a quantitative consideration of street-bias in the paper
5. The opponents of JPR, actually use the JPR model framework to discuss street bias through its parameters. A classic proof of the usefulness of a model!
6. Tim's title is incorrect, ... and may be seen as slanderous
You guys get personal very quickly don't you
Sock puppets aren't people. If you want to get treated like a person, be a person.
Why didn't Burnham/Roberts do a street-bias analysis?
Because there's no such thing as a "street-bias analysis". There is no such thing, because, absent cherry-picked cases like the one in the badly flawed JPR paper, the "contiguous houses down secondary street" method samples the space effectively.
In answer to the six new silly assertions which have sprung up as a result of my answering one silly question:
1. No; it has been established that only rare and cherry-picked street choices would result in measurable truncation of the sample space, which is in itself a long leap from "Bias", as nobody has established that any such censoring was informative.
2. No, clearly a bad approximation, as it presents only a single set of parameters which are not robust. As a model of "bias", even worse because its discussion of whether the hypothetical censoring was informative is nugatory.
3. No, the suggested values are unreasonable (as demonstrated above). Since nearly all peturbations of the street identification code give parameter values close to zero, it is neither honourable nor useful to ignore this.
4. Nor does it give consideration of a million and one other almost certainly fictitious phenomena. Not the Lancet's job to do Spagat et al's work for them in inventing the hitherto unknown "Main street bias" phenomenon, which remains wholly speculative.
5. Clearly silly - a model whose only use is to prove that the issues it raises are not important, is not a useful model.
6. No it isn't, and (how it pains me to mention this), if it is true that "LancetStudy" is a pseudonym for one of the authors, they should perhaps look closer to home; Michael Spagat has considerable past history of making slanderous statements about researchers with whom he disagrees, and attracted an open letter of complaint from Human Rights Watch Colombia (IIRC) for doing so.
Sock puppet,
Try for just one second to think like a scientist. Suppose JUST for the sake of argument that people who lived on main streets did have a higher death rate. What Tim has shown is that it is inconceivable that all (or even a significant majority) of the clusters would be on the main streets. There is simply no way to rationally choose main streets without getting many of the clusters off the main streets.
These clusters would have very different rates of death than the clusters on the main streets and this would show up in the analysis and as #174 point out
âGAME OVER MANâ. Since this did not happen, there is no MSB. The end.
elspi wrote: "..The end.."
Au contraire, it is just the beginning...
oh god
LancetStudy :"So how, where and when did Burnham/Roberts manage to quantify and discard any street-bias effect?"
They didn't. They didn't have to - BECAUSE THEY FUCKING DESIGNED IT OUT UP FRONT!!!!!!!!!
Get it yet?
The taco truck example was an answer to your dishonest insinuation that the sampling teams didn't have enough time to identify main streets. Misrepresent much?
They also didn't quantify and discard possible time of day effects, possible scary interviewer effects, possible 'nice interviewer I want to please' effects, possible biased question effects, and on to a multitude of possible effects. They design to minimize all of these - including skewed sampling effects - as in EVERY FUCKING SURVEY EVER TAKEN!!!!
Stop being despicable, LancetSTudy.
Neat..
Invent a sort of incomplete data question where you cant even know how incomplete the data are, and demand the set is complete before believing the result.
Presumably even if we had the full set of household addresses lets say from a true random sample there would still be some clustering, and you could pull some post hoc 'ah but look they cluster near this or that, there's bias'
Anyhow I'm quite interested in how 'main street bias' works without creating increased intra over inter cluster correlation given that the clusters were in different places in different sized towns etc.
They also didn't quantify and discard possible time of day effects, possible scary interviewer effects, possible 'nice interviewer I want to please' effects, possible biased question effects, and on to a multitude of possible effects. They design to minimize all of these - including skewed sampling effects - as in EVERY FUCKING SURVEY EVER TAKEN!!!!
they also did ignore all positive effects on the polling process on mainstreets (Taco truck next to cluster: TTntC-effect, reducing massively the time needed for lunch breaks)
neither did they quantify effects that give a too low deathrate (wiped out or fled households) down to the third digit behind the comma!
Interesting, of course, that the potentially quite serious problem of "Didn't Visit Anbar Because It Was Too Dangerous Bias" in the UN IFHS survey has attracted so much less attention.
LancetStudy.
This may appear to be a nit-pick, but to me it speaks volumes...
At #173 you refer to Jody as 'she'. Given the number of times in this thread that you have referenced Jody I am surprised that you remain ignorant of said person's identity.
Google Jody's name, add 'Dr' and 'London' to the string, and find out why your oversight is rather glaring.
If I was extensively debating with someone who hadn't done even a cursory amount of research on basic background such as this, I would be very sceptical of this person's capacity to construct a cogent argument.
Of course, that's just my bias.
nteresting, of course, that the potentially quite serious problem of "Didn't Visit Anbar Because It Was Too Dangerous Bias" in the UN IFHS survey has attracted so much less attention.
i think i have an explanation for this: as they did not go there, they did not have to travel on roads, so there can not be any street bias.
on a more serious note, the IFHS study replaced the numbers for those dangerous regions with numbers calculated from the IBC project. and the IBC take their numbers from newspaper reports.
the serious bias in this approach is, that the number of reports is certainly linked to the number of journalists in Iraq.
and this number gets reduced [during times of high violence](http://www.nytimes.com/2004/10/02/international/middleeast/02media.html)
and even more with the [recent withdrawal of journalists](http://www.brookings.edu/opinions/2008/1015_iraq_media_nessen.aspx)
so let us check some numbers:
he New York Times, Los Angeles Times, and Washington Postâwhich all continue to staff small Baghdad bureausâpublished a total of 858 front-page stories from Iraq in 2003, but only 138 in the first nine months of this year.
ooops, this gives us a JWB (journalist withdrawal bias) of 6.2!!!
dsquared: a model whose only use is to prove that the issues it raises are not important, is not a useful model.
True, if you are looking at the matter from a scientific perspective. But from a troll's point of view, if you can annoy people and get them arguing and lavishing attention on you - even if the attention mostly takes the form of scorn - that's success.
In the previous thread on this subject a professor of demography wrote a lengthy comment on Burnham et al. and where it may have gone wrong; he didn't assume that it did of course. There was practically no response (not that I suppose the author particularly wanted responses). A sock puppet, whose sock-puppetry was obvious from the similar quality of writing and reasoning even before Tim confirmed the shared IP address, got umpteen responses.
If a troll was a scientist, the MSB paper is just the sort of paper a troll would want to write. Whether or not this particular troll is one of the authors, it's easy to see why s/he admires their work.
i think the use of the Lancet iraq study as the main example of their model is extremely dubious.
if you had a good theoretical model, would you use an example from a paper, that you consider to be completely flawed in many aspects? one which you consider to consists of faked, incomplete and unavailable data? a paper that, according to you, does not accurately describe its methodology? one that forces you to guess all parameters, with no way to check them?
it just doesn t make any sense, unless you consider other motives. (mainly publicity)
A good test of the model would be to apply it to something like traffic fatalities, or pedestrians injured by cars. We'd expect more accidents on busy streets. If we applied the L2 sampling scheme, would our results match the actual accident rate, as reflected in official statistics? And if they didn't match, would applying the formulas from the MSB paper give us a correction factor?
My guess is that we'd find that the L2 sampling scheme would result in pretty accurate numbers to begin with, and the utility of an MSB algorithm to adjust those numbers would be negligible.
Well LancetStudy
Touchy aren't you? I guess you are one of the MSB authors after all... what a shame.
I'll leave you to your spherical cow to do what comes naturally. Oh yes, and to speculate about whether or not I have a graduate degree...
cheers, and goodbye