Unfortunately, the Journal of Peace Research has published the badly flawed “Main Street Bias” paper. My earlier criticisms still apply, so I’m reposting them. Consider this the first draft of a reply to their paper.

The authors argue that main street bias could reasonably produce a factor of 3 difference.

How did they get such a big number? Well, they made a simple model in which the bias depends on four numbers:

  • q, how much more deadly the areas near main street that were sampled are than the other areas that allegedly were not sampled. They speculate that this number might be 5 (ie those areas are five times as dangerous). This is plausible — terrorist attacks are going to made where the people are in order to cause the most damage.

  • n, the size of the unsampled population over the size of the sampled population. The Lancet authors say that this number is 0, but Johnson et al speculate that it might be 10. This is utterly ridiculous. They expect us to believe that Riyadh Lafta, while trying to make sure that all households could be sampled, came up with a scheme that excluded 91% of households and was so incompetent that he didn’t notice how completely hopeless the scheme was. To support their n=10 speculation they show that if you pick a very small number of main streets you can get n=10, but no-one was trying to sample from all households would pick such a small set. If you use n=0.5 (saying that they missed a huge chunk of Iraq) and use their other three numbers, you get a bias of just 30%.

  • fi, the probability that someone who lived in the sampled area is in the sampled area and fo the probability that someone who lived outside the sampled area is outside the sampled area. They guess that both of these numbers are 15/16. This too is ridiculous. The great majority of the deaths were of males, so it’s clear that the great majority were outside the home. So the relevant probabilities for f are for the times when folks are outside the home. And when they are outside the home, people from both the unsampled area and the sampled area will be on the main streets because that is where the shops, markets, cafes and restaurants are. Hence a reasonable estimate for fo is not 15/16 but 2/16. If use this number along with their other three numbers (including their ridiculous estimate for n) you get a bias of just 5%.

In summary, the only way Johnson et al were able to make “main street bias” a significant source of bias was by making several absurd assumptions about the sampling and the behaviour of Iraqis.

Comments

  1. #1 ron
    February 14, 2009

    Here is **a challenge** to the L2 defenders and/or MSB attackers, which is potentially very useful for quantifying MSB:

    Walk out onto your street and ask the first 5 people you meet if they know the color of your car (or house etc.). Multiply this fraction who know, up to the population level of your country. Is the result reasonable, or larger than reasonable? I doubt if it is smaller than reasonable. Hence R>1.

  2. #2 ron
    February 14, 2009

    **!! CHALLENGE TO TL and others… !!**

    Walk out onto your street and ask the first 5 people you meet if they know the color of your car (or house etc.). Multiply this fraction who know, up to the population level of your country. Is the result reasonable for the number of people in your country who know the color of your car, or larger than reasonable? I doubt if it is smaller than reasonable. Hence R>1. Hence TL should, as an apparent scientist, agree to change this blog title (or open a new one) titled:

    ** Lancet publishes badly flawed paper (i.e. L2) **

  3. #3 luminous beauty
    February 14, 2009

    ron,

    It would only be applicable if you asked those people the color of their own car. Burnham, et al. polled households for deaths in their own households not their neighbors.

  4. #4 ron
    February 14, 2009

    Luminous beauty wrote: “ron,
    It would only be applicable if you asked those people the color of their own car….”

    No.

  5. #5 luminous beauty
    February 14, 2009

    ron,

    You can’t tell shit from shinola.

    The stupid, it burns!

  6. #6 Bruce Sharp
    February 14, 2009

    Ron, a few people have taken the time to respond to you on the good-faith assumption that you are genuinely interested in this discussion. Your messages have become so absurd that I have to conclude that you are either a troll, or you are very, very badly confused, not only about L2, but about the MSB paper as well. (Longtime Deltoid readers: does this remind you of that time when that guy from Chicago Boyz was here, arguing that the high crude mortality in some clusters could be offset by negative crude mortality in others???)

    Setting that aside, Ron, I’m happy to take your challenge. But remember… we are testing “main street bias,” and I don’t live on a main street. So to see whether or not asking on a main street will bias the results, let’s do this: I’ll first pick a random neighborhood of my city, and ask 5 people if they know the color of my car. Then I’ll go to one of the main streets, and ask if they know the color of my car.

    I am willing to bet that in the random neighborhood, exactly zero people will know the color of my car. And I am also willing to bet that on the largest street in my city, exactly zero people are going to know the color of my car.

    I am also willing to bet that in a complete nationwide census, the number of people who know the color of my car will be somewhere around… oh, I dunno, maybe .000003%.

    In which case, I think we could say that my random survey was fairly accurate. We could also say that main street bias didn’t significantly affect the results.

  7. #7 ron
    February 14, 2009

    LB wrote: “It would only be applicable if you asked those people the color of their own car. Burnham, et al. polled households for deaths in their own households not their neighbors.”

    OK, I just did it: For various reasons I have seen 3 people outside of my household in the last few hours. I asked them to name the color of their car. Two of the people couldn’t even remember if they had a car, while one named the color of his car. Why? The first two live in a warden-controlled gated community for people needing 24/7 care because of dementia. So do 2/3=66.7% of the population not know the color of their car? I don’t think so.

    My point is: Bias is the norm, not the exception. Every finite study is susceptible to implicit biases. These biases have to be analyzed and quantified. MSB does this for one type of bias, albeit in a simplistic way, and produces a general formula (Eq. (4) for 2-subsystems, and EPL for a general number). You’ve done a good job, JPR authors.

  8. #8 ron
    February 14, 2009

    LB and BS (.. telling initials me-thinks…) wrote various things which are non-scientific.
    Simply point me to the exact statements that I made that are incorrect and *prove* they are incorrect, and then we have a basis for discussion. Otherwise, just carry on being rude. You are getting a D-grade so far in the debate…

    BTW, a question to LB: What is the precise methodology underlying the phrase “The stupid, it burns!”. It seems to have syntax errors, with a bias….

  9. #9 ron
    February 14, 2009

    Bruce Sharp wrote:
    ” I’ll first pick a random neighborhood of my city…”

    THAT IS THE POINT!!! L2 didn’t pick a RANDOM neighborhood, in the sense that random should mean unbiased.

    You got it Bruce, good on you!!! At last!!!
    You have now officially understood the problem with L2, and simultaneously the value of JPR. Case closed. I am leaving….

  10. #10 John
    February 14, 2009

    **Kevin Donoghue**: You wrote in your reply to sod and myself regarding probabilities of death: “Workmen in Si: 0.075 Workmen in So: 0.015 Others in Si: 0.005 Others in So: 0.001″ I’m glad you did because I hope this clarifies the gender discussion. It is clear that the model as presented in EPL could be augmented to incorporate gender, but that would double the number of model parameters, and would probably lead to even more controversy. My point really is that the way the model has been put forward in the JPR article simply does not deal with gender. Therefore, any attempt to compare “predicted” male and female casualties to empirical data is simply misleading. To get to the gender issue in the first place, you need to make your own assumptions about gender specific death probabilities, as you did above. That is fine per se, but what is misleading is to say that the MSB papers make assumptions about male and female death probabilities when, in fact, they don’t. Therefore, there really is no contradiction with any kind of evidence regarding male and female deaths, simply because the model does not make any statements about male and female deaths. It only operates at the level of two distinguishable sub-populations of individuals (samplable vs. non-samplable).

    **Sod**: After reading the EPL paper carefully, it is clear that the death probabilities qi and qo are, in fact, population averages, just like in the disease example that I presented. This is given explicitly in the EPL paper, at the very bottom of page 2 and very top of page 3 (granted, they are a bit difficult to see in the text). If one allows for Ni subsystems in the samplable space, i.e. as many subsystems as there are people, then qi is simply the average of death probabilities for everyone in the samplable system, men, women, children, etc. Please see also my comment to Kevin above.

    About some of the criticism directed against the parameter values. Perhaps the draft of the JPR paper is different from the published version, but in the actual paper there is Table I, in which the values of all of the parameters, q, n, fi, and fo are varied. There are altogether 225 different parameter combinations listed there. In the EPL paper, there are nine different surface plots that explore the parameter space. Alternatively, one can of course also just plug in any values in eq. 1 in the JPR paper and see what comes out. The parameter values in the text are considered “plausible” in the JPR paper, and in the EPL they are based on “plausibility arguments”. What is plausible to one person may not be that to another, but it doesn’t sound like the authors are suggesting that these parameter values are carved in stone. They provide a simple, transparent equation, carry out sensitivity analysis, and write in the abstract of the JPR paper: “The authors provide a sensitivity analysis to help readers to tune their own judgements on the extent of this bias by varying the parameter values.”

    John

  11. #11 sod
    February 14, 2009

    Sod: After reading the EPL paper carefully, it is clear that the death probabilities qi and qo are, in fact, population averages, just like in the disease example that I presented. This is given explicitly in the EPL paper, at the very bottom of page 2 and very top of page 3 (granted, they are a bit difficult to see in the text). If one allows for Ni subsystems in the samplable space, i.e. as many subsystems as there are people, then qi is simply the average of death probabilities for everyone in the samplable system, men, women, children, etc. Please see also my comment to Kevin above.

    you have it wrong John. they are explicitly using gender to calculate fi and fo.
    an extreme example will show this easily: if their working males would spend ALL their time outside their homezone. (just a slightly different approach to their 6 h version..)
    the outcome would be clear: Lancet 2 shouldn t have found a SINGLE working age violent death of a male in their sample.

    Sod: After reading the EPL paper carefully, it is clear that the death probabilities qi and qo are, in fact, population averages, just like in the disease example that I presented. This is given explicitly in the EPL paper, at the very bottom of page 2 and very top of page 3 (granted, they are a bit difficult to see in the text). If one allows for Ni subsystems in the samplable space, i.e. as many subsystems as there are people, then qi is simply the average of death probabilities for everyone in the samplable system, men, women, children, etc. Please see also my comment to Kevin above.

    i think you don t understand me. i don t oppose the idea of sampling bias. their formula is ok.

    just everything about using the lancet paper as an example is wrong. all their chosen values are wrong. their modelling of the “mainstreet” is wrong. (lancet actually has a low probability to poll any of their chosen “mainstreets”) taking Lancet as an example is wrong, as it was a method that was chosen for security reasons and is unlikely to be repeated. all the attacks on Lancet are wrong. (data release. they simply don t have a place in such a theoretical work)

    and their results are wrong, as shown for example by the gender issue, or by comment #4, and and and..

  12. #12 Kevin Donoghue
    February 14, 2009

    John: what is misleading is to say that the MSB papers make assumptions about male and female death probabilities when, in fact, they don’t.

    I don’t see how you get to that conclusion. The probability that a female resident in the survey space dies outside it is zero, because Johnson et al. explicitly assume that she never leaves. Similarly a female non-resident can’t die in the survey space. Other restrictions follow from the fact that working-age males spend a specified amount of their time outside their own zones. So, for example, if you know the probability that a working-age male from Si dies (anywhere), you also know the conditional probability that a working age male dies in So specifically. And so on. When you take these things into account, there isn’t much elbow-room in the model.

    It’s because of these restrictions that I had trouble finding the four probabilities which (I think) meet the objection sod has been making throughout this thread. The assumptions n=10, q=5, f(for working-age males)=0.75, f(for all others)=1 impose horribly tight restrictions on the probabilities.

    Johnson et al. want their model to justify their claim that Burnham et al. estimate three violent deaths for every one that actually takes place. If the assumptions make any sense they must be consistent with the composition of the death-toll as well as the total – there has to be some disaggregated version of the model which generates R=3 as desired. After some doubts, engendered by sod’s protests, I’ve come to the tentative conclusion that the MSB approach can handle that problem, but only if we take all aspects of the MSB squad’s worldview on board.

    In short, although I think they are quite wrong about Iraq and I don’t think much of their model, it’s not impossible to reconcile the gender-composition of deaths in Burnham et al. with their assumptions. But if you think it’s easy, I urge you to put some numbers in a spreadsheet and see if you can come up with an alternative to my four probabilities.

  13. #13 Kevin Donoghue
    February 14, 2009

    Sorry John, this was a bit cryptic:

    …if you know the probability that a working-age male from Si dies (anywhere), you also know the conditional probability that a working age male dies in So specifically.

    In fact I’m not sure that’s strictly true. But what I have in mind is that (1) your f gives you the weights to apply to the two (unknown) probabilities and (2) your q gives you their ratio. It may be true that males and females could have distinct q values, which complicates things, but you’re stuck with the given q for the total. (I can only say it may be true because I haven’t been able to make an example, with distinct q values, work in practice.)

    If I was a bit more diligent I’d set up the whole system of equations in a systematic way in order to see just what’s possible and what isn’t. But I’m a bit bored with it at this stage. Nobody seems to want to defend the model, complete with the actual parameter values the authors propose. And if we can all choose the parameter values we like, what’s the point?

  14. #14 Bruce Sharp
    February 14, 2009

    Ron in #209: Having thus vanquished King Arthur by bleeding all over him, the Black Knight declared victory and wobbled away on his legless, armless torso, to bask in the warm glow of triumph.

  15. #15 Bruce Sharp
    February 15, 2009

    Looking at the draft version of the paper, I’m confused by one of the assumptions that the authors make in support of their fractions of time spent in or outside the survey space. Perhaps I am misinterpreting something? The draft says:

    “Assuming that there are two working-age males per average household of seven (Burnham et al., 2006), with each spending six hours per 24-hour day outside their own zone, yields fi = fo = 5 / 7 + 2 / 7 × 18 / 24 = 13 /14 .”

    Does that mean they are assuming that the working males are always working inside the survey space? That would imply that every factory, market, office, warehouse, garage and so on is in the survey space. For every work location to be on a “main street,” you’d have to be willing to use a pretty liberal definition of “main street.”

    I’m also having a problem with the derivation of q. The authors’ argument that the main streets are going to be prime targets for IEDs, car bombs, etc., makes sense, but on the other hand, I would think that other types of violence (abductions, for example) would be more likely to take place in more isolated areas. So how do you quantify the relative levels of violence? Obviously, several people have already commented on the way the parameters are pulled more or less out of thin air, but the support for q=5 in the draft seems bizarre. Is this the same in the published version? In the draft, they write:

    “Given the extent and frequency of such attacks, a value of q = 5 is plausible. Indeed, many cities worldwide have homicide rates which vary by factors of ten or more between adjacent neighbourhoods (Gourley et al., 2006).” The link to Gourley 2006 in the draft is no longer valid, so I can only guess what that document says. What I find striking, however, is the idea that a drastic difference in homicide rates in other cites would have any relevance at all in this context. A tenfold difference in violence could, I suppose, be used as justification to say that we know that values as high as q=10 are plausible… but then, depending on whether violence was higher inside or outside the survey space, we could also say that it means that values as low as q=1/10 are also plausiible. In other words, it doesn’t tell us anything at all.

    I don’t see any problems with the logic of the model, but I’m with Kevin: without accurate parameters, I don’t see how it could ever be useful.

  16. #16 Robert Shone
    February 15, 2009

    I’ve commented briefly on this debate at my blog:
    http://dissident93.wordpress.com/2009/02/15/pope-of-debunkers/

  17. #17 Bruce Sharp
    February 15, 2009

    Hi Robert –

    Thanks for posting the link. However, I don’t understand your first objection, concerning the parameters. You say that nobody knows the correct parameters “partly because the Lancet authors won’t release necessary information.” What could they possibly release that would allow you to determine an accurate value for q? Maybe I’m entirely confused (always a possibility), but I’m not sure how you are supposed to determine the possibility of death outside the survey space without… you know… conducting some sort of survey.

    Regards,
    Bruce

  18. #18 Tim Lambert
    February 15, 2009

    Oh, so Shone reckons I’m the Pope now? What are the odds that David Kane will describe his nonsense as a tour de force?

  19. #19 Jody Aberdein
    February 16, 2009

    I just stumbled upon the [now classic CMR thread](http://scienceblogs.com/deltoid/2007/08/robert_chung_on_david_kane.php).

    Highly likely I should say.

  20. #20 Ozzy
    February 16, 2009

    _Bruce S. …What could [Lancet study authors] possibly release that would allow you to determine an accurate value for q?_

    If the Lancet authors released the information of what streets were in the samplable region (i.e. not necessarily the sampled streets, just the candidates during the selection process) then by superimposing maps such as the BBC’s one of Baghdad bombings and attacks, which are at street level resolution, an estimate of the q-values (in particular, qi/qo) could be made. Better still, if they released the actual streets surveyed (i.e. not necessarily the exact houses) this would improve the estimates of q-values.

    _So yes, they could release very important information to pin down the MSB parameters. Without this information, there is no point in asking questions about market-specific scenarios. Unfortunately, the Lancet authors are preventing any advancement of this topic._

  21. #21 Bruce Sharp
    February 16, 2009

    Ozzy, thanks, that helps a bit. I hadn’t seen a map of attacks. That would certainly be an improvement over the s.w.a.g. (scientific wild-ass guess) algorithm. But wouldn’t the reporting of violent incidents suffer from just as much main street bias the violence itself? I’d think that a map of violent incidents from a media outlet would represent essentially the same incidents as those cataloged by Iraqi Body Count. I’m not necessarily sold on the Lancet figures, but I think virtually everyone here would agree that the IBC figures represent only a subset of the actual violence. Wouldn’t it be reasonable to assume that at least part of that discrepancy between the IBC figures and other estimates comes from incidents that occur in relatively isolated areas?

    Regards,
    Bruce

  22. #22 Ozzy
    February 16, 2009

    _Bruce S….yields fi = fo = 5 / 7 + 2 / 7 × 18 / 24 = 13 /14 .”
    Does that mean they are assuming that the working males are always working inside the survey space? That would imply that every factory, market, office, warehouse, garage and so on is in the survey space. …._

    Some points:

    1. Their theory does not assume fi=fo. See Eq. (4) of their published JPR article.

    2. The closer fi and fo are to 1, the higher the bias R since the more localization of people in their own q-specific zones (i.e. qi or qo). So if you would have preferred that males from So be assigned less time in Si, then this makes R even larger and hence MSB even bigger.

    3. There is no assumption about ‘main streets’ per se being the root of the bias. The words ‘main street bias’ does not mean that ‘there is a bias for people living/working on main streets’. Instead it means ‘a bias which has to do with the use of main streets as an initial identifier in the Lancet sampling selection process’. Since that is a bit of a mouthful, they shortened it to ‘main-street-bias’. This is what I remember at the time, and this is what one of them personally told me.

    Following up on this last point, I notice a very common misconception by people attacking the JPR on this thread, in which they interpret the JPR team’s ‘main street bias’ as having something to do with living on a main street or working/walking on a main street. It is not. Main street bias just means ‘a bias which has to do with the heterogeneity of streets in terms of where car bombs, street markets and their attacks are likely to occur’. It was simply the fact that the Lancet team started with main streets as the reference point for selecting cross streets, that led to the name. If the Lancet team had used ‘unpaved streets’ as the initial reference point for choosing cross-streets, then this would have *also* led to a bias (of possibly very different value, but of similar cause: i.e. street heterogeneity in terms of casualties-on-the-street).

    The key point is that attacks etc. are not evenly distributed across a city in the same way as throwing darts blindly at a map of Baghdad (for example). Attacks are localized in areas around cross street to main streets (see the BBC maps, among others). Main streets in Baghdad, for example, tend to have relatively few actual residents — they are essentially dual carriageways. People tend to walk around on cross streets — and that is where markets are, and where cars can easily park. Hence market bombs, car bombs, sniper attacks, convoys etc. are concentrated there. It does not mean that no attacks, kidnapping etc. occur elsewhere, or that other back-alleys do not contain people living in fear. It just means that casualties are concentrated around cross-streets to main streets. Hence MSB, hence R>1.

    Because of this possibility, the Lancet study’s street selection should have been designed such that there was no street heterogeneity — or if there was, then it should have been corrected or quantified by the Lancet authors. They didn’t do this, or even mention this in their original paper — hence the whole JPR story….

  23. #23 Bruce Sharp
    February 16, 2009

    Ozzy, on point 1, my apologies: as I noted, I have only seen the draft, and not the published version. On point 2, yes, I realized that less time in Si makes MSB larger. I didn’t raise the question because I want to send the results in one direction or another; I raised the question simply because it seems like a bizarre assumption, and a poor model for the real world. On point 3, thanks… that’s a helpful clarification.

  24. #24 sod
    February 16, 2009

    If the Lancet authors released the information of what streets were in the samplable region (i.e. not necessarily the sampled streets, just the candidates during the selection process) then by superimposing maps such as the BBC’s one of Baghdad bombings and attacks, which are at street level resolution, an estimate of the q-values (in particular, qi/qo) could be made. Better still, if they released the actual streets surveyed (i.e. not necessarily the exact houses) this would improve the estimates of q-values.

    all of this is false. the method was chosen, because road names and accurate maps were NOT available.

    the BBC map is useless. the red dots are the size of a cityquater. incidents that kill 10+ people are far from forming a majority of kills.
    and even then, these dots only show where the killing happened, NOT where the people involved lived!

  25. #25 Kevin Donoghue
    February 16, 2009

    Their theory does not assume fi=fo. See Eq. (4) of their published JPR article.

    Eq. (4) is in an appendix in the draft and, I presume, in the published paper also. Wherever it is presented, Eq. (4) is completely uncontroversial. It is no more a model of survey bias than “Trade surplus = Value of Exports – Value of Imports” is a model of international trade.

    To obtain a model you have to go beyond definitions and tautologies. You have to make some behavioural assumptions. That’s what Johnson et al. do; Eq (4) is merely their point of departure. So yes, their theory does assume fo=fi.

    But even then, although their R(q,f,n) function is unsatisfactory in that regard, nobody would bother arguing with them if they weren’t pushing for the parameter values Tim rightly objects to, and relying on question-begging devices such as their BBC map. (If media sources are provide a reasonably complete picture of the violence then it’s obvious that Burnham et al. are way off; the whole argument turns on whether there is a lot of violence that we don’t see reported.)

  26. #26 ozzy
    February 16, 2009

    _Sod…all of this is false. the method was chosen, because road names and accurate maps were NOT available.
    the BBC map is useless. the red dots are the size of a cityquater. incidents that kill 10+ people are far from forming a majority of kills. and even then, these dots only show where the killing happened, NOT where the people involved lived!…._

    Actually, Sod is not correct on various points. First, some identifier of main streets (and cross streets) needed to be made in order for the cross-street-to-a-main-street to be randomly selected. This information is what is needed for estimate of q to be made.

    The BBC map is quite useful. See the authors EPL article, which demonstrates that size of dots is adequate for distinguishing, at least in an approximate way, information about relative frequency of attacks near cross-streets as compared to other street topologies.

    Most importantly, Sod, the q parameter does not depend on where people live. It is simply the rate of attacks in that subspace. So qi is the probability of attacks in subspace Si,and qo is the probability of attacks in subspace So. This is easily generated using the BBC map, as long as some rough idea of where the main-streets and cross-streets are — at least, which are the ones that could have been picked (and hence define Si, and by extension So).

  27. #27 ozzy
    February 16, 2009

    _Kevin Donoghue….Eq. (4) is completely uncontroversial. It is no more a model of survey bias than “Trade surplus = Value of Exports – Value of Imports” is a model of international trade. To obtain a model you have to go beyond definitions and tautologies. You have to make some behavioural assumptions. That’s what Johnson et al. do; Eq (4) is merely their point of departure. So yes, their theory does assume fo=fi…._

    It is very nice to see that Mr. Donoghue agrees that the JPR’s main result, an analytic formula for bias R based on street heterogeneity, is completely uncontroversial. Let’s just pause and take that news in!

    Now, the remaining points are again, with all due respect, wrong. To compare Eq. (4) and the analysis that produced it, to “”Trade surplus = Value of Exports – Value of Imports” is a model of international trade”, is bizarre and misguided.

    Next point, their theory does not assume f0=fi. The theory of superconductivity does not assume a given implementation of the parameter values (e.g. phonon frequencies), nor does the theory of gravitational attraction assume certain distance separation of objects. One obtains a theory, with parameters, and then (as the JPR authors nicely put it in their introduction) they put in candidate values leaving readers to choose their own.

    Unless you cherry-pick a vanishingly small set of possible parameter values (i.e vanishingly small compared to the space of possible parameter values) you will not get R=1. And as soon as you reach that straightforward conclusion that R is not 1, you are led to ask the Lancet authors to address the issue by clarifying the R value. Whether R is actually near 3, smaller, or larger, remains to be seen.

  28. #28 ozzy
    February 16, 2009

    I apologise to my colleagues for multiple postings, but I missed this comment:

    _Kevin Donoghue….If media sources are provide a reasonably complete picture of the violence then it’s obvious that Burnham et al. are way off; the whole argument turns on whether there is a lot of violence that we don’t see reported….._

    Rather incorrect. All that is needed is the relative danger of samplable Si region compared to the non-samplable So region. This will give the q. The fact that the media may have under-reported each of those red circles by factors of 5, 10, 100 etc. is irrelevant for the estimation of q since q is simply a ratio.

  29. #29 ozzy
    February 16, 2009

    My above comment “…The fact that the media may have under-reported each of those red circles by factors of 5, 10, 100 etc. is irrelevant for the estimation of q since q is simply a ratio…” was rather loose. It is correct in substance, but those factors would of course remove practically all attacks from ever taking place. What I meant was: any under-reporting by some reasonable factor (e.g. 3) would not matter for the purposes of estimating q.

  30. #30 Jody Aberdein
    February 16, 2009

    ..’Lancet authors to address the issue by clarifying the R value’..

    Sorry. The Lancet Authors should go provide empirical evidence for n, q and f (which would most likely now have changed considerably) because someone can colour in some google maps and has some odd ideas about womens place in the home?

  31. #31 ozzy
    February 16, 2009

    Dear Jody. You wrote: “.The Lancet Authors should go provide empirical evidence for n, q and f (which would most likely now have changed considerably) because someone can colour in some google maps and has some odd ideas about womens place in the home?.”

    All they need to do, is declare what the samplable space was at the time of the survey (i.e. Si, and hence So by default). If they are willing to provide more than this, yes it would help pin down the q values more — but not necessary. Just Si and So will do. We can all do the rest together….

  32. #32 Jody Aberdein
    February 16, 2009

    How would that help us with N0, Q0 and f0 and with explaining the gender distribution in the presented results? What would we then do with the exact street addresses of each survey point to find out these values?

  33. #33 ozzy
    February 16, 2009

    “…declare what the samplable space was at the time of the survey …”

    Just to be crystal clear: This requires simply a list (or even simpler, a Google map with lines) identifying the main streets and cross streets used in the Si pool, from which the actual sample selection was then drawn. Then everything not included, is So by definition.

    No names of households needed, nor the specific names of streets used as starting points. Just the initial *pool* which made up Si.

    Obviously, giving the specific streets would be a real bonus to getting accurate q values, and then better estimates of the f’s and n’s based on other information. But I guess this type of release will never happen. So to get going, all one needs is Si.

  34. #34 ozzy
    February 16, 2009

    Jody write: “How would that help us with N0, Q0 and f0 ?”

    Hmm, now I see you don’t really understand what these parameters are — and hence what the model is. In that case, I don’t know what to say other than repeat: Yes, it will help with q immediately. Then from access to information about housing density and distribution of commercial activity at the time from other sources, we can get at better estimates of Ni, No, fi, and fo. Then R. Job done. Even gender labels can be added, by generalizing f etc. to carry the gender label. ANd the EPL result generalizes the R calculation. Etc…

    But without Si and So, no further progress beyond JPR can be made. The whole conversation becomes pointless.

  35. #35 Jody Aberdein
    February 16, 2009

    So how exactly would you work out S0, Q0 and F0, F1 then? Just for it to be clear in my mind.

  36. #36 ozzy
    February 16, 2009

    I have to run off, but quickly:

    “how exactly would you work out S0, Q0 and F0, F1 then? Just for it to be clear in my mind…”

    So is the rest of the streets that are not in Si. So knowing Si, and knowing all the streets that exist, So=All-Si.
    Knowing So, and taking e,g, the BBC map, one can simply count the red circles in Si and So, and the ration is qi/qo. Then the f’s can be deduced roughly by knowing something about the employment and markets inside and outside of So. This is available, very roughly, from lists of businesses and markets. Apologies for rush: Got to go, back later….

  37. #37 Jody Aberdein
    February 16, 2009

    So for when you’re back..

    Again just to be clear it is not that we just need Si from Burnham et al, but that we also need to estimate qo using some mortality data that by definition cannot have come from Burnham et al.

    For this we choose the BBC’s map of media reporting of deaths, saying that under reporting is not an issue for a ratio.

    So we are to adjust a result because we believe the sampling method was not random, and we are to adjust this result by a factor generated from media reported deaths.

    Anyone wish to collaborate on a paper? I propose a factor, let’s call it R, due to Media Street Bias. Intuitively I think this should be proportional in some way to the amount of whisky available locally, the other events in the news that day , and whether or not the military drive you to one area or another to report on what they think you should. We could probably get it into a journal if we make the formula look complex enough.

  38. #38 Bruce Sharp
    February 16, 2009

    Ozzy, in #228, I think you are missing the point that both Kevin and I tried to make: if you truly believe that mainstreet bias affects the level of violence, why would it not also affect the reporting of violence? Are we supposed to believe that although terrorists don’t stray from the main roads, journalists do? I don’t know how to make the point any more clearly than I did the first time: there is clearly a discrepancy between the toll reflected in media reports (i.e., the IBC count) and the actual toll. That discrepancy is deaths unreported in the media. You can’t use the media reports to provide a baseline figure for the deaths that the media doesn’t report… and that is figure you need to determine realistic ratios.

  39. #39 Kevin Donoghue
    February 16, 2009

    ozzy: It is very nice to see that Mr. Donoghue agrees that the JPR’s main result, an analytic formula for bias R based on street heterogeneity, is completely uncontroversial.

    Eq. (4) is not based “based on street heterogeneity” whatever that may mean. It’s based on the assumption that we have two sets Si and So and we make inferences based on a sample taken from one of these only.

    ozzy: their theory does not assume f0=fi.

    My theory:

    ozzy = ron = troll

    I could be wrong of course. Tim, has Tim Blair or somebody of that ilk linked to this post? I’m seeing a pattern in the comments.

  40. #40 Eli Rabett
    February 16, 2009

    Ozzie writes:

    Next point, their theory does not assume f0=fi. The theory of superconductivity does not assume a given implementation of the parameter values (e.g. phonon frequencies), nor does the theory of gravitational attraction assume certain distance separation of objects. One obtains a theory, with parameters, and then (as the JPR authors nicely put it in their introduction) they put in candidate values leaving readers to choose their own.

    Nonesense. If someone takes your guesses and model and gets that niobium is superconducting at 400 K you get laughed at. At best you then adjust the model to account for bounds on the parameter space, at worst, if the guesses are realistic your model gets tossed on the trash heap.

  41. #41 luminous beauty
    February 16, 2009

    …nor does the theory of gravitational attraction assume certain distance separation of objects.

    What? Is this the noodly appendage theory of gravitational attraction?

  42. #42 John
    February 16, 2009

    **Jody**: I would ask you to take a look at my earlier postings on the gender issue. There is little point in repeating the argument here, but the main point is that the model in its present form makes no statements about male and female deaths. Therefore, there is no discrepancy with “reality”.

    **Bruce**: I would be interested in hearing your view on the following. It seems to me, and I believe everyone here would more or less agree, that the media only reports some fraction of the deaths, i.e. underestimates the actual death toll. In particular, they probably report only the most devastating cases with most casualties. So for example, the BBC map only shows cases with more than 10 deaths. But I would have assumed that whether these incidents get reported is independent of, and hence uncorrelated with, their location. In other words, whether incident X (with 10 or more deaths) taking place in Y gets reported is independend of Y. Do you see a problem with this assumption?

    John

  43. #43 Kevin Donoghue
    February 16, 2009

    John: … I would have assumed that whether these incidents get reported is independent of, and hence uncorrelated with, their location.

    That seems very wrong to me. Numerous reporters have made it clear that they had very little freedom of movement in Iraq at the height of the conflict (it’s probably better now). Whether something got reported had a lot to do with whether the scene could be reached safely. Whatever about L2, there is no doubt at all that camera crews had a main street bias.

  44. #44 ozzy
    February 16, 2009

    I am back, and was very disappointed to see the level of thought in the responses.
    In the last few exchanges I had before I left, I thought some element of reasonable progress was being made. Now I come back, and see that all the people against MSB papers have resorted to simplistic, trite comments without substance or focus. You really don’t like a proper debate do you?

    I think it is a waste of my time trying to engage you with responses. I suspect my comments are uncomfortably reasonable for you. I had seen some others drift away from this site, and now I see why — with childish comments like “Media Street Bias”, and others, you guys’ thoughts are worthless.

  45. #45 Bruce Sharp
    February 16, 2009

    Hi John –

    I think that assumption is reasonable enough. However, at best it only address part of the problem. Incidents with high death tolls are a subset of the total deaths, and again, the problem is not the known incidents. The problem is the unknown incidents. Abduction and murder, for example, would represent a different subset of deaths. Do we have any reason to believe that this subset occurs with the same relative frequency inside and outside the survey space? As noted above, I think you could make a reasonable argument that these incidents would be more likely in isolated areas. Does a map of high-fatality incidents tell us anything at all about these deaths?

  46. #46 sod
    February 16, 2009

    Jody: I would ask you to take a look at my earlier postings on the gender issue. There is little point in repeating the argument here, but the main point is that the model in its present form makes no statements about male and female deaths. Therefore, there is no discrepancy with “reality”.

    if this is so simple, why don t you simply explain, how and why all those men in the “mainstreet zone” get killed, while their women, spending MORE TIME there are left unharmed.

    there might be possibilities, but they are not very likely.

    Just to be crystal clear: This requires simply a list (or even simpler, a Google map with lines) identifying the main streets and cross streets used in the Si pool, from which the actual sample selection was then drawn. Then everything not included, is So by definition.

    i love it, when people are giving homework to others. i love it even more, when you know in advance, that whatever result you will get, they wont accept it.
    if i was part of the lancet team, the last thing on my mind would be coloring some maps for you…

    so here is my counter offer:
    why don t you provide fully colored maps of all towns in Iraq, that would give a n=10 number?

    what you and others seem to miss is this: even the smallest village has (a) mainstreet(s). there isn t one “mainstreet concept” that will fit all places of Iraq.
    you would also need population density data. in a country, that can t account for its total population…

    again my advice: waste some serious research time on this (6 months and some money?) and you will learn about futility…

  47. #47 sod
    February 16, 2009

    with childish comments like “Media Street Bias”, and others, you guys’ thoughts are worthless.

    media street bias of course is real. sunni areas? sadr city?

    in a place with streets without names, journalists will move events to the closets landmark.

    sorry, but just because you don t understand it, its neither childish nor worthless.

    it is simply bizarre: people have doubts about tiny details in the Lancet paper, but accept the IBC numbers as a gospel truth.

  48. #48 Jody Aberdein
    February 16, 2009

    ‘Media Street Bias’ … worthless

    So an explicit, albeit contested sampling schema is debated for some 246 comments in this blog, and yet we are to qualify this with a correction factor based upon a BBC map of media reported deaths. Whilst ‘Media Street Bias’ might be considered a childish comment, perhaps we could have some explanation of how the media’s sampling protocol is more robust
    ?

  49. #49 David Kane
    February 16, 2009

    Tim: I see you more of a Martin Luther myself, so Shone and I will just have to agree to disagree on this.

    By the way, if you still think you criticism of Johnson et al (2008) stands, then you ought to write it up clearly (shouldn’t take more than a couple of pages), post it and start a new thread. You can hardly expect the authors to bother with this endless thread, especially since the criticisms you make in later entries (like #55) have nothing to do with the criticism you make in the initial post.

  50. #50 Lancet Debunker
    February 16, 2009

    Sod’s “…tiny details in the Lancet paper…”

    Tiny details?? TINY details?? The non-random sampling (i.e. bias toward cross streets with higher numbers of car bombs, market bombs, sniper and convoy attacks) is a tiny detail???

    Idiot.

  51. #51 Tim Lambert
    February 18, 2009

    David, of course I think my criticisms of the MSB paper stand. I’m not sure what I wrote that might make you think otherwise. I will collect things together and make a new post, but I want to draw some maps so it might take a little while.

  52. #52 Tell
    February 18, 2009

    Tim wrote: “…I want to draw some maps…”
    Without knowing the main streets?? Get this everyone: Tim is going to draw conclusions (with the undoubted predetermined goal of criticising JPR) based on MAPS in which he will ASSUME he knows the L2 list of possible MAIN STREETS!!
    This is priceless….

    Tim: don’t you realise by know, that to go beyond JPR you need to know about Si space? In other words, unless you know something about what MAIN STREETS were included, and hence what cross streets were samplable, your time is completely wasted on this.
    I repeat: you need to know what is a MAIN STREET in Baghdad etc. How many are there, therefore what type of streets are cross streets. etc. etc. Then you can start….

    Anything you do is, a priori, invalidated without knowing this — it is immediately worthless. So do us all a favour, and ask the L2 authors about their samplable space Si???

    Tim wrote “…of course I think my criticisms of the MSB paper stand…”
    Note the use of ‘of course’: In other words, *irrespective* of the fact that the ensuing discussion debunked Tim’s JPR criticism, he will (and always will) blindly criticize JPR. Basically, it ticks him off that JPR makes a valid point, and that they might indeed be right….

  53. #53 Jody Aberdein
    February 18, 2009

    I agree with Tell, Clearly it would be totally impossible to do any kind of analysis without this detail. Certainly picking one’s own main street or several and comparing with a grid based random selection or some such wouldn’t in any way set any kind of bound for this problem.

  54. #54 Eli Rabett
    February 19, 2009

    For what it is worth, there appears to be an infestation of Roshes here, or perhaps that is Roshii, in any case a Lott.

    Eli and several others here were involved years ago in a very long thread on USENET about McIntyre and McKitrick’s complaints over the Mann, Bradley and Hughes papers. One of the energetic M&M defenders was a certain Nigel Persaud, who, it later turned out, was McIntyre and who in disguise showed up a few other places including Deltoid. Now far be it from Eli to moan about pseudonyms, but in that case, and the bunny strongly suspects this, there are Mary Roshii amongst us. On re-reading the old thread, it is obvious that a lot more progress could have been made if Nigel had stood up, simply because he was defending his own work and if he had not had to pretend that he didn’t know about a bunch of stuff we could have all contributed more. FWIW

    The bunnies can all now play pin the tail on the Rosh

  55. #55 Eli Rabett
    February 19, 2009

    You don’t have to have the same list of main streets, you just have to have a list of main streets. You divide them up into subsets, run the calculation on the subsets and compare the ensemble result. Surely one does not have to tell a condensed matter physicist that?

  56. #56 Kevin Donoghue
    February 19, 2009

    Since the “gender argument” has cropped up a lot in this thread, it may be worthwhile to try to wrap it up here. The crux of the matter is that the assumptions made by Johnson et al. do have significant implications for the distribution of deaths. In what follows let “men” be shorthand for working-age males (of whom there are 2 per household of 7) and “others” be shorthand for women, children and old men.

    To avoid too many messy fractions, let the population, N=154 (each one standing for a little under 0.2m Iraqis). On the assumptions made by Johnson et al. this population is divided up (by residence) as follows: 4 men in Si; 40 men in So; 10 others in Si; 100 others in So. Since men spend a quarter of their time outside their own zones, while others spend all their time in their own zones, the average numbers in the zones are: 13 men in Si; 31 men in So; 10 others in Si; 100 others in So.

    With the population divided into men and others, we have four probabilities to consider instead of the two (qo and qi) in the aggregate model. To avoid messing with subscripts, let the probabilities of death be as follows: for men in Si: w; for men in So: x; for others in Si: y; for others in So: z.

    It turns out that we are quite restricted in our choice of these probabilities. What looks like a four-dimensional space actually has only one dimension. Firstly, the probability of death of randomly selected unisex individuals in Si has to be q times the corresponding probability in So. Hence:

    (13/23)w + (10/23)y = q[(31/131)x + (100/131)z]

    A second equation ties the (w,x,y,z) vector down further. The expected number of deaths among residents of the survey space Si is 3w + x + 10y. When scaled up by a factor of 11 to provide the Burnham (L2) estimate of deaths for the whole population this becomes 33w + 11x + 110y. This corresponds to the numerator in Eq (4) of the Johnson et al. paper. The denominator is the expected (actual) total deaths: 13w + 31x + 10y +100z. The ratio of these two numbers is the bias, R, hence:

    33w + 11x + 110y = R(13w + 31x + 10y + 100z)

    A third equation is given by the fact that the L2 estimate of deaths is about 2.3% of the Iraqi population so that:

    33w + 11x + 110y = 154(0.023) = 3.542.

    With three equations to tie down the four unknown probabilities, we have some flexibility in choosing (w,x,y,z) but we may not have a lot, bearing in mind that being probabilities they must be in the interval [0,1]. Do the values proposed by Johnson et al. for their parameters, and hence for R, leave us with any plausible-looking probabilities? The handiest way to summarise the options is to insert their proposed values (q=5 & R= 2.9512) into the above equations and express w, x, y and z as functions of t, where 0 < t < 1. For the sake of readability I’ve multiplied all coeeficients by 1,000 so these are the expressions for expected deaths per thousand:

    Men while in the survey space Si: w(t) = 25.968 – 2.062t.

    Men while in So, outside the survey space: x(t) = 20.617 – 20.616t.

    Others in the survey space Si: y(t) = 22.348 + 2.68t.

    Others in So, outside the survey space: z(t) = 6.391t.

    WARNING: I am very prone to errors in calculations of this kind, as I’ve already demonstrated a couple of times in this thread. I’ve checked this a couple of times but basically it’s worth what you paid for it.

  57. #57 Tim Lambert
    February 19, 2009

    Kevin Donoghue writes:

    >My theory: ozzy = ron

    Correct!

    But don’t stop there.

  58. #58 Bruce Sharp
    February 19, 2009

    Go figure. I thought Ozzy was somebody else.

    I was thinking: Ron = Lancet Debunker = Tell

    By the way.. Kevin, you might want to repost #256 in the new thread, since this one has dropped below the fold.

    Regards,
    Bruce

  59. #59 Tim Lambert
    February 19, 2009

    >I was thinking: Ron = Lancet Debunker = Tell

    Also correct.

    Keep going.