Comments

1. #1 David Kane

   February 9, 2009

   Note that this page of clarifications from the authors may be helpful to understanding the paper. Highlights:

   Some people who have commented on our work hold the erroneous view that we assume that people are killed at home or that they must be killed at home for our model to make sense. In the model there is a zone of households that can be reached by the sampling methodology, the “surveyable” zone, and a zone of households that cannot be reached by the survey, the “unsurveyable” zone. People can be killed anywhere. Below we consider pertinent cases. Bear in mind that both zones consist of many irregular pieces all over town.

   …

   It is important to bear in mind that the f values are averages over many individuals. Some people will spend more and others will spend less than the average fraction of time out of zone. We have also averaged over two populations: the population inside the surveyable zone and the population outside the surveyable zone. In the model this assumption corresponds to setting fi = f0 = f. Although this is clearly a simplification, a lack of information about the detailed implementation of the recent Iraq study prevents a more precise estimate to be made at this stage.

   Neither of these points gets directly to the issue of Tim’s counter-example, but I thought that some readers might find them helpful. The link is via footnote 2 in the paper.

2. #2 Crust

   February 9, 2009

   David Kane (or anyone else), did they address at all the much higher violent death rate for non-elderly adult males than for any other group? As Tim and others have pointed out that’s a major problem for their MSB theory.

3. #3 Robert Shone

   February 9, 2009

   David Kane writes:

   [Tim] is just creating an example in which we know R is 1

   In the MSB formula R cannot be 1 unless n=0, q=1 or f=1/2. That makes sense when you think about it – the only way to avoid bias completely is an all-inclusive sample, spatially uniform violence or perfect diffusion of people among zones.

   So, by that definition, Tim’s definition of R=1 in his hypothetical construction is wrong. But we “know” that R=1 in Tim’s example. Or do we? How do we know? Because it’s defined as part of Tim’s “construction”. But it’s a Platonic spook which exists in no real-world equivalent of Tim’s artificial set-up.

4. #4 sod

   February 9, 2009

   i think it is pretty funny to watch you guys dance around the gender issue. if only Iraqis were as good at avoidance as you guys are….

5. #5 Kevin Donoghue

   February 9, 2009

   Tim Lambert: Consider this the first draft of a reply to their paper.

   Tim, I take that to mean that you are going to publish your criticisms, by way of a response in the Journal of Peace Research or something of that sort? Until now my feeling was that the MSB was so crappy it was best ignored, but the fact that a journal saw fit to publish it is a scandal. There ought to be a response.

   When it comes to the sort of arguments which I think might carry weight with journal editors (as opposed to normal people) one of the worst features of the paper is the glib way the authors introduce the assumption fi=fo. In the draft I’m looking at they make no real effort to justify it. I take it the published paper is no better in that regard? But your “all deaths are in the marketplace” story highlights the fact that, in this model, there is one way and only one way that Iraqis can enhance their survival prospects. That is by spending as much time as possible in the relatively safe region outside the survey space. And how do the Iraqis avail of that possibility? According to the authors, they don’t!

   The fi=fo assumption, smuggled in with hardly a shred of justification, is tantamount to an assumption that Iraqis are stupid.

   Now, what happens if we credit the Iraqis with a bit of intelligence? That’s not such an easy question. If those who live in the back-streets have jobs on the main streets then they have little option but to accept the risks, as you note in your post. But quite a few seem to be unemployed so I wouldn’t want to be dogmatic about this. It may even be the case that, on some reasonable assumptions, rational behaviour increases the bias. However the authors certainly ought to have addressed the issue.

   There’s another thing about the fi=fo assumption that I think should have set off alarm bells. In Appendix 2 where the “no-bias” conditions are explored, they set fi=fo just before they examine the cases where R=1 identically. By doing that they obscured the fact that fi+fo=1 is a perfectly respectable no-bias locus. It corresponds to the case where everybody, regardless of residence, spends the same amount of time in the survey space. It’s far-fetched of course but it’s certainly no less worthy of consideration than n=0 or q=1. In fact they do mention f=0.5, saying this is the case where everybody spends 12 hours per day in the survey space. But it’s not a matter of the number of hours. Really, in a mathematical discussion, isn’t it a bit odd to mention that there are 3 points in the domain of the R function such that R=1, while passing over the fact that one of those points is the sole survivor of the infinite set we just vaporised? Methinks they are trying to gloss over the fact that most urban dwellers, if employed, spend their working day on or near a main street.

6. #6 Kevin Donoghue

   February 9, 2009

   Come to think of it n=0 and q=1 are infinite sets too. But as Jonah Goldberg would say, I believe my point is unaffected.

7. #7 David Kane

   February 9, 2009

   cruft: I don’t know if they address that point. (I have not studied the paper and associated materials closely.) But, I don’t see how it matters. It does not effect the math. It does not (at least as I can see) effect the parameter estimates. Now, of course, one can always make a model better by adding more details. Perhaps, in the next iteration, one might allow for the parameters to vary by gender or age. (See also the quote in #101.)

   sod: Putting your comments in bold does not make them more persuasive.

   Kevin writes:

   Until now my feeling was that the MSB was so crappy it was best ignored, but the fact that a journal saw fit to publish it is a scandal.

   This sort of hyperbole does not serve you well. Your claim now is that the authors, the editors and the prize committee are all idiots? That, for some reason, they can’t see what is obvious to you? All these people with fancy Ph.D.’s and university appointments?

   As always, I am a fan of challenging the tenured, but you need to be realistic about the number of people that have looked closely at it. They might all be wrong, but you should provide some evidence.

   And, bad news, more journal editors are persuaded that this approach has merit. See:

   J.-P. Onnela, N. F. Johnson, S. Gourley, G. Reinert, and M. Spagat, Sampling bias due to structural heterogeneity and limited internal diffusion pdf, submitted to Europhysics Letters 85, 28001 (2009).

   Better alert the editors/referees of Europhysics Letters lest they engage in scandalous behavior as well!

   By the way, your point about fi and fo is interesting. Isn’t it best if we focus the discussion on these substantive points?

8. #8 sod

   February 9, 2009

   cruft: I don’t know if they address that point. (I have not studied the paper and associated materials closely.) But, I don’t see how it matters. It does not effect the math. It does not (at least as I can see) effect the parameter estimates. Now, of course, one can always make a model better by adding more details. Perhaps, in the next iteration, one might allow for the parameters to vary by gender or age. (See also the quote in #101.)

   look David, you have been avoiding this point over 100 posts now. the model wouldn t be “better” or “improved” when it got gender right, but it would be NOT wrong.

   as it is, this model suggests that the majority of violence victims found by Lancet should be women. many more elderly or kids. young men should be a tiny minority among those killed by violence! (and polled)

   but instead young men are the overwhelming majority among those killed by violence found by the Lancet study! (and in reality, btw…)

9. #9 sod

   February 9, 2009

   J.-P. Onnela, N. F. Johnson, S. Gourley, G. Reinert, and M. Spagat, Sampling bias due to structural heterogeneity and limited internal diffusion pdf, submitted to Europhysics Letters 85, 28001 (2009).

   nice, Spagat recycled the Baghdad bombing graph again. the one, on which he made the completely insane claim, that incidents that kill over 10 people “almost certainly cover over half of all deaths.”

   http://sod-iraq.blogspot.com/2008/04/spagat-and-kane.html

   looking at sampling bias due to different distribution of violence makes sense. that is, why you do as many clusters as you can.
   but the Spagat version of “mainstreet bias” is obviously false, and a seriously flawed attack on the Lancet study.

10. #10 Jody Aberdein

    February 9, 2009

    Somewhat amusingly, despite framing the paper for ‘systems with structural heterogeneity’ the one system they choose is of course the Iraq conflict. But so many other structurally heterogenous systems to choose from one would think.

    In any case when it comes to parameterisation, we are referred to the learned reference 13, Johnson et al J Peace Res 2008, and thence to Spagat’s coloured in google maps.

11. #11 Nick

    February 10, 2009

    I have spent some time studying the MSB papers (i.e JPR and EPL) and the arguments here. I just wanted to add my own thoughts before anyone goes further with hypothetical analysis of cases etc. I have also studied this entire blog, and am quite shocked by the level of aggression and rudeness of some of the discussions. But, lets set that aside and turn to the math. As I explain below, there is nothing wrong with the JPR analysis:

    Burnham and co-authors’ analysis of the Iraq survey results (call this L2) implicitly assumes that within each survey set (i.e. cross street algorithm) there is no possible bias, i.e. R=1 in the Journal of Peace Research paper (JPR) language. In JPR language, this would indeed be the case if, for example, the samplable and non-samplable regions have either n=0, or q=1, or f=1/2 in the case that f_i=f_0. In my opinion, the point of the JPR article is to ask what happens if this assumption of implicit homogeneity does not hold. It considers arguably the simplest generalization of full homogeneity, by allowing the samplable and non-samplable regions to have different values for the model parameters. This could correspond, for example, to a case of different q values. The JPR authors’ latest published paper, the EPL, generalizes this approach to allow for multiple subsystems (i.e. more than 2). Some of them may be in the samplable region, and some may not. They may even be male and female labels (i.e. gender). For this general case, the parameters now contain multi-valued indices to numerate these many possible subsystems, i.e. they are not just ‘o’ and ‘i’ hence the harder notation in the authors’ EPL. This EPL formalism would allow, in principle, for a near exact calculation of R using exact details of markets etc. *provided* that the exact details of the samplable region are known. Without the details of what streets are sampled, however (or at least, what the samplable region is) any such calculation would be a waste of time. In other words, in the lack of any information about S_i details (e.g. whether markets are present or not) such a detailed calculation would also be unnecessary: Having just two subsystems is sufficient to show there is a large potential bias R. So this seems to be what the JPR proposed, and their EPL generalizes this approach for an arbitrary numbers of subsystems.

    By adding in a single highly heterogeneous feature (i.e. a single market) Tim Lambert (TL) is implicitly asking for consideration of a *3* subsystem hypothetical example. There are now in principle *three* possibly distinct killing probabilities q, i.e. for the region S_i not including the market, for the market itself, and for the unsamplable region S_o. (TL assumes that the market is in the samplable region). To calculate the bias, a three-subsystem formula is required which can be obtained from the authors’ EPL paper’s formalism in exactly the same way as the JPR two-subsystem formula is derived. But why stop at a three-subsystem calculation? Indeed, stopping at third-order expansions is, as any mathematician knows, often unreliable. One should go further, to all higher orders: Indeed, if the JPR authors knew the actual streets sampled, then a very accurate multi-subsystem calculation could be carried out. But they do not, since L2 authors do not seem to have released it. Hence they have to stop their calculation at the first generalization level beyond L2′s assumptions, i.e. the two-subsystem model of the JPR. To go further, i.e. to include specific markets, would be to add a specificity to the model which is not justified by the lack of detailed information provided by the L2 survey team. Moral of the story? L2 team should provide details of the surveyed streets! (or at least, the samplable region S_i). Failing that, the only way forward seems to be the one taken by JPR and EPL.

    So what about the hypothetical TL market example? Can the two-subsystem JPR result still be used to calculate the specific bias for the TL setup, without resorting to the more general formalism of the EPL? Yes, but the person plugging in the numbers (TL?) needs to fully understand the meaning of the terms and the steps in the algebra. The key is in JPR Appendix B, where the final formula is derived. Start from the introduction: The samplable region, which is where TL wants the market situated, now has an implicit heterogeneity meaning that it cannot be represented by a single q. In other words, there is a probability of being killed q=0 for everywhere in the samplable region S_i *except* in the market which has some high value q=qm. (The following holds, by the way, irrespective of N_i and N_0, thereby addressing the concerns of a later blogger who had other suggested values). Reading through the first paragraph of JPR, for the specific example of TL, you can see that:
    - The probability that during one day, for example, a randomly chosen person is resident in S_i and gets killed in S_i is q_i . f_i . N_i/(N_i+N_o) as given by the JPR. But remember from the JPR that q_i is the *averaged* killing rate in S_i, where the average is over the entire S_i subsystem (i.e. market and non-market). It would be a weighted average of qm for the market region of S_i, and 0 for the non-market region of S_i, weighted by the time spent in the market with respect to time spent in the rest of S_i. As an example, we can set f_i=1 as TL requests, just to say that the S_i resident never leaves S_i.
    - The probability that during one day, for example, a randomly chosen person is resident in S_o and gets killed in S_i is q*_i . (1-f_o) . N_o/(N_i+N_o). Because of TL’s desire to add a third subsystem, i.e. the market which adds heterogeneity to S_i, the killing rate probability q*_i corresponds to the killing rate in the market only. I emphasize that this 3-subsystem example should be treated with the EPL formalism — but no worries, we can do that using the JPR equations by simply recognizing that q*_i is qm, since all the time the S_o resident spends outside S_o (i.e. all his time in S_i) is spent in the market.
    - The probability that during one day, for example, a randomly chosen person is resident in S_o and gets killed in S_o is 0.
    - The probability that during one day, for example, a randomly chosen person is resident in S_i and gets killed in S_o is 0.
    The rest of the derivation then follows, but the expressions contain the new unknown parameter qm due to the additional heterogeneity introduced by TL. The eventual result? An expression which yields R=1 for the TL example. It is essentially just Eq. (4) of JPR, but with the appearance of an additional qm due to the additional heterogeneity of the market. No mystery, and no surprises. It all works.

    Dare I offer some advice without getting abuse? I could say “Read through carefully the first paragraph of Appendix B, and allow for the additional heterogeneity that you have forced into the problem. Alternatively, just work through the general case in EPL.”
    But what I will also dare to say is: “Let’s all ask the L2 authors to release information about the samplable areas. Ideally, houses surveyed — if not, the name of streets surveyed — if not, the name of streets in the samplable space S_i. Then we can all forget this endless hypothesizing, get out some more accurate parameter values, and see what the JPR authors predict for R. Maybe it is near 1? Maybe it isn’t. Maybe there are aliens, maybe there aren’t etc. etc. (you get the drift…).

    To summarize: L2 assumes complete homogeneity within the pool of possible streets, which is a massive and unjustified assumption. In effect, they assume R=1. JPR generalizes this to allow for two-subsystem heterogeneity, showing that an R value larger than 1 can arise. EPL generalizes this to allow for n-subsystem heterogeneity. Both of these are of course also assumptions, but less so than assuming that everything is homogenous as in L2. TL wants even more heterogeneity, in his hypothetical case of a single market. However, the reality is that there is (at least) as much heterogeneity as there is number of people and streets in Baghdad. In principle this could be modeled using the EPL and JPR approach, but *only* if more information about the street sampling is known. Let’s all ask them — then maybe all this unpleasant discussion would disappear, and everyone would be freed up to do something much more productive for the world.

    P.S. Gender could be incorporated by allowing the parameters in the JPR article to carry subscripts (f) and (m) for female and male. This is just the same as adding another subsystem label (see EPL). The equations would all then have double the number of unknown parameters, and be far more complex. Without additional details about the surveys, is there much point in doing this?

12. #12 bi -- IJI

    February 10, 2009

    > I have also studied this entire blog, and am quite shocked by the level of aggression and rudeness of some of the discussions. But, lets set that aside

    You know, it’ll really helps the discussion if you don’t preface your comment with a quick note on how you expect to get shouted down and bullied. Really, how do you expect any reasonable person to respond to this type of thing? Do you expect reasonable people to simply overlook your claims of persecution while you continue to scream them?

    > Maybe there are aliens, maybe there aren’t etc. etc.

    If your criticism of Burnham et al. is based on the possibility of there being aliens in Iraq, then you have a huge problem.

13. #13 Kevin Donoghue

    February 10, 2009

    Nick: Burnham and co-authors’ analysis of the Iraq survey results (call this L2) implicitly assumes that within each survey set (i.e. cross street algorithm) there is no possible bias, i.e. R=1 in the Journal of Peace Research paper (JPR) language.

    It might be an idea to read L2, Nick. Burnham et al don’t “implicitly assume” that there is no bias. They discuss possible sources of bias at some length. In MSB language they consider both R<1 and R>1 since either is possible. The MSB authors brush one of these possibilities aside for no good reason. When it comes to presenting the estimates L2 is quite explicit about the estimators used and no reader could possibly be fooled into thinking that they have incorporated any bias adjustments.

    …get out some more accurate parameter values, and see what the JPR authors predict for R.

    My remarks about the business model of the Underpants Gnomes refer. One of the parameters you require, for example, is the risk of death outside the survey space. Burnham can’t give you that. Try God.

    I really don’t think you’ve thought this thing through.

14. #14 Robert Shone

    February 10, 2009

    Nick writes:

    However, the reality is that there is (at least) as much heterogeneity as there is number of people and streets in Baghdad.

    Well put – this seems an obvious but key point wrt Tim Lambert’s Platonic construction. Many thanks, Nick, for that detailed post which fills in a lot of gaps for me.

15. #15 Robert Shone

    February 10, 2009

    Kevin Donoghue writes:

    Burnham et al don’t “implicitly assume” that there is no bias. They discuss possible sources of bias at some length.

    Kevin is only half right here, at best. They may discuss “possible sources of bias”, but they assume no bias in their study. In comments made by Burnham et al after MSB was suggested, they very explicitly claim no bias for their study.

16. #16 sod

    February 10, 2009

    As I explain below, there is nothing wrong with the JPR analysis:

    you mean a model that predicts the majority of (polled) victims to be females is not wrong? when Lancet found the majority to be male?

    But what I will also dare to say is: “Let’s all ask the L2 authors to release information about the samplable areas. Ideally, houses surveyed — if not, the name of streets surveyed — if not, the name of streets in the samplable space S_i.

    well, i ll limit my shouting to most basic part:
    you will NEVER release the streets that you polled! this demand is insane!

    P.S. Gender could be incorporated by allowing the parameters in the JPR article to carry subscripts (f) and (m) for female and male. This is just the same as adding another subsystem label (see EPL). The equations would all then have double the number of unknown parameters, and be far more complex. Without additional details about the surveys, is there much point in doing this?

    is there a reason to look at a factor (“mainstreets”), when there is a different other factor (gender) that is massively more important?
    the authors own assumption (males spend significant time outside their homezone) completely RUINS any attempt of a combined “mainstreet – gender” analysis. one of those two points (mainstreet) simply is irrelevant to the result!

17. #17 Nick

    February 10, 2009

    Sod wrote: “…NEVER release the streets that you polled!”

    Why?

    What are they hiding? And why are you scared?
    Remember, no identities required. At the bare minimum, Burnham et al. just need to tell the MSB team what streets are in the samplable region, and hence what streets are in Si (and by default So). All I can conclude from their refusal, is that (1) they don’t know, or (2) they are hiding known weaknesses in the surveying. Or both, since (1) is just a specific case of (2).

18. #18 Kevin Donoghue

    February 10, 2009

    Discussion of what Burnham has and whether he should or shouldn’t give it to this or that individual is a diversion from the topic of this thread. However I will note that he has been quoted as saying: “Our goal was to reduce any type of risk to the community and the participants. While we have much of the raw data, we requested that anything designating the interviewers or the location of the neighborhoods visited not be sent to us.”

    Could anyone wishing to pursue this aspect of the Iraq mortality controversy please do so in an appropriate thread? Thank you.

19. #19 Robert Shone

    February 10, 2009

    Kevin Donoghue writes:

    Discussion of what Burnham has and whether he should or shouldn’t give it to this or that individual is a diversion from the topic of this thread.

    Actually it goes to the heart of this thread. We wouldn’t be discussing some nonsensical Platonic abstraction (Tim’s market) if we had real data on the topic to look at.

20. #20 Nick

    February 10, 2009

    Kevin Donoghue writes:
    “…is a diversion from the topic of this thread”

    Did you read my earlier long post, explaining in detail why TL’s claims of invalidity of JPR article, are themselves invalid? Thread topic over.
    (By the way, which bit didn’t you understand? Sounds like you understood none of it.)

    I can see that this is not a discussion forum in the true sense, so I wish you all the best of luck in your future adventures and will now bow out from future posts here. Bye bye….

21. #21 Crust

    February 10, 2009

    David Kane:

    I don’t know if they address that point [much higher mortality for non-elderly adult males than for others].

    Well, they didn’t address it in the “Clarifications” piece you linked to FWIW.

    But, I don’t see how it matters.

    Huh? As the authors themselves point out, men from the szmpled zone likely spend more time on average in the unsampled zone than others living in the sampled zone. Similarly, males from the unsampled zone likely spend more time in the sampled zone. In other words, you would expect to see a greater sex/age differential in the unsampled zone than in the szmpled zone. But to get their numbers to work out, they would need a dramatically smaller differential there. It just doesn’t add up.

22. #22 sod

    February 10, 2009

    What are they hiding? And why are you scared? Remember, no identities required.

    you don t understand this. there are rules. you can t simply break them and publish data that will make people traceable.

    Spagat needs to fix the problems in his paper, BEFORE any additional data is released. and don t expect miracles from such data.
    what would some more information about the location change?

23. #23 Kevin Donoghue

    February 10, 2009

    Nick: Did you read my earlier long post, explaining in detail why TL’s claims of invalidity of JPR article, are themselves invalid?

    Yes, it was long wasn’t it? For the benefit of readers who might be in a hurry, this is the gist:

    You people are rude, Tim is all wrong about the MSB paper because the authors wrote another paper which is much better and I’m going to be brave and tell you again how rude you are.

    Which, if true, leaves the editor of the JPR looking a bit foolish. Not only did he publish the inferior product, but a jury said it was the best paper he’d got all year.

24. #24 frankis

    February 10, 2009

    Hopefully Nick has only left to go and do something “much more productive for the world”. Iraq could probably use further surveying work if he’s up for that; his raw data might be offered to the awarded MSB team for some comfy-chaired admonishment from them. Vale Nick!

25. #25 Gator

    February 10, 2009

    What is the actual equation that the paper derives for the bias? The actual paper seems to be behind a paywall, and the draft paper equations seem to assume f0=fi=f.

    In any case, it is a perfectly valid thing to pick test cases where one thinks they know the answer and see what your “model” predicts. This is just what Tim did in #55. The fact that the equation is so far off in such a simple limiting case pretty much kills the model in my mind.

26. #26 David Kane

    February 10, 2009

    1) Thanks to Nick for taking the time to explain in #111. For those who don’t have the time to read it, the key insight (which I had missed before) is that Johnson et al (2008) explicitly assume that qi and qo are constant throughout there respective areas. So, Tim’s example is not a counter-example to the model since q in not constant across the sampled region. Unless Tim wants to continue this aspect of the conversation (presumably in a new thread), I would say we can conclude that, the math (at least) in Johnson et al is correct.

    2) Gator: The draft pdf (linked to above) is, for the purposes of this discussion, equivalent to the published version. You claim that “The fact that the equation is so far off in such a simple limiting case pretty much kills the model in my mind.” Well, please read Nick’s response. Tim’s example does not work because it does not meet the assumptions of the model. Any example that does meet those assumptions will produce the right answer.

27. #27 Gator

    February 10, 2009

    David Kane and Nick. Tim describes a situation where the market is the dangerous place, and all deaths occur in the market. However, since the market is in the sampled area, one can just as well say that all deaths occur in the sampled area, and none in the unsampled. This is what Tim’s example says. So the model should be valid in either case. The fact remains that for this simple limit of equal population in and out, and highly skewed risk factors, the model is wrong.

    Anyway, “Nick’s model”?? I don’t see a Nick on the paper’s author list. I just saw Nick making a new model where one has to identify and separately quantize “dangerous” areas of the sampled area. This is a whole new thing and apparently not captured in the paper under discussion. It certainly wasn’t in the draft paper.

28. #28 Gator

    February 10, 2009

    After a bit more thinking and reading, I think Nick has perhaps explained why the model in the paper does not work. (I admit I didn’t read his explanation too closely because it was so longwinded…) The paper explicitly states that qi is the risk over the entire sampled area. If the sampled area includes safer areas (back from the main street) and unsafe (the main street) then the model will overestimate the risk of death for people living in the sampled area because they are assumed to spend more time there. In fact, they may spend the same amount of time in the actual risk areas (as in Tim’s model) as the people from the unsampled areas. I.e. everyone goes to the market once a day, but other than that they stay away from the dangerous areas.

    As Nick points out this model is too simple. But he is criticizing the Johnson et al model, not Tim’s test of the model.

29. #29 Gator

    February 10, 2009

    OK, my last comment in a bit…

    One must also see that this simple risk model is likely to be wrong by realizing the violence is not randomly occurring in time or space. Presumably a “smart” terrorist will detonate their car bomb when the market is crowded, or when it is time to pray at the mosque. Assuming that the risk of death is proportional to the time in the sampled area is a gross oversimplification. This is in effect assuming the violence is random in time and space over the sampling area.

30. #30 Tim Lambert

    February 10, 2009

    David, my example shows that the model used in the MSB paper is wrong. Their model assumes that there is no main street bias, so is only going to give the correct answer in the case of R=1.

31. #31 Robert Shone

    February 11, 2009

    Gator writes:

    it is a perfectly valid thing to pick test cases where one thinks they know the answer and see what your “model” predicts. This is just what Tim did in #55.

    Yes, but it’s not “valid” to do so by completely ignoring premises which have been spelt out for that model. The MSB authors were pretty clear in explaining where they were drawing the lines in terms of what Nick calls “two-subsystem heterogeneity”. Lambert simply ignored that.

32. #32 Robert Shone

    February 11, 2009

    To put it another way (let’s say if I were adopting Nick’s precise-sounding jargon, which I can’t take credit for): In a two subsystem framework, the market has to be located either in S_i or S_o. If the market is located within S_i, the samplable subsystem, then we have q_i > q_o, since the risk of death pertaining to the market has been absorbed in q_i. If A spends more time in S_i than B does, then A also clearly has a higher risk of death. Therefore, the risk of death is not the same for A and B as claimed by Tim, and sampling from S_i will introduce an upward bias. Note that there is no distinction between where within S_i the person spends his or her time, which is why why q_i is defined as the probability of death when present within S_i, not as the probability of death when present within some subset of S_i. How large this bias is depends on q = q_i / q_o, where one implicitly assumes that q_i > 0 and q_o > 0, i.e. there is always a non-zero probability of death in a war-zone. One could introduce a different three subsystem framework, in which there is a third subsystem, say, the market. In this case, however, the equations would need to be derived again, since they are based on a two subsystem model.

33. #33 sod

    February 11, 2009

    it is now 130 posts and not a single real reply to the gender problem. i can t really blame you for the avoidance of the subject, it is a critical error in the study.

    so let us look at some other points:

    Si, the samplable subsystem, then we have qi > qo, since the risk of death pertaining to the market has been absorbed in qi. If A spends more time in Si than B does, then A also clearly has a higher risk of death. Therefore, the risk of death is not the same for A and B as claimed by Tim, and sampling from Si

    i don t think you understood Tim s point, and i think you assume the Spagat model to be more flexible than it really is.

    a simple question should make the problem obvious. even to you!
    how much time do you spend in the supermarket every week? how much more/less time would you spend there, if it was further away (closer)?

    people go to the market to by stuff. they don t spend significant more time there, when it is closer to their home. so while you could model it, it wouldn t give a significant bias for the Lancet study.

    on an other important point:
    what use does Spagat et al see in their model?

    I haven t seen the lancet guys claiming that their method of cluster choice is a model to follow. instead it was the most practical solution under EXTREMELY difficult circumstances.

    with GPS systems getting tiny and extremely common and perfect aerial maps being readily available, i can t see many repetitions of the Lancet cluster choice in the future!

    other problems with cluster choices (different violence in different regions) are well known and always taken care of in cluster sampling.

    i simply can t shake the feeling, that Spagat is working on a theoretical paper (always popular, as few people can do math..) that is abusing the popularity of the Lancet paper.

34. #34 David Kane

    February 11, 2009

    Tim: I believe that Robert Shone’s #132 answers your #130.

    But, I could be wrong! If so, the best way forward is not to continue the discussion in this thread but to start a new thread in which you provide a precise description of your counter-example. Then, we could focus on precisely that debate.

    So, if you still think that you have found a flaw which invalidates two published papers, please start a new thread devoted to just that flaw. I bet that Johnson et al might even respond. You can hardly expect them to respond to a point that didn’t arise until comment #55 in this thread.

35. #35 Kevin Donoghue

    February 11, 2009

    Robert Shone: In a two subsystem framework, the market has to be located either in Si or So.

    It’s not clear that we can say this. The draft I’m looking at is vague on that point. The survey space Si is a set of households. The marketplace is not a household. Is it outside the survey space by definition?

    I’m not sure what to make of Tim’s argument. For now I cling to my belief that, although the MSB model is unsatisfactory in just about every way a model can be unsatisfactory, it is logically valid. But maybe the way the thing is set up breaks rules I’m not acquainted with. Certainly the premises on which the model rests could have been more clearly stated.

    I think the authors intended that all deaths take place in either Si or So. Nobody is allowed to die in the marketplace.

36. #36 Tim Lambert

    February 11, 2009

    DK:
    >I believe that Robert Shone’s #132 answers your #130.

    No, it doesn’t. Their model assumes that the risk of death is the same everywhere in the sampled region. Which is main streets and cross streets. That is, it assumes that there is no main street bias in the death rate.

    If you believe that main streets are more dangerous, then their model is wrong.

    If you believe that main streets are not more dangerous, their model shows that the L2 sampling scheme is unbiased.

    Choose one.

37. #37 Robert Shone

    February 11, 2009

    Tim Lambert writes:

    Their model assumes that the risk of death is the same everywhere in the sampled region. Which is main streets and cross streets. That is, it assumes that there is no main street bias in the death rate.

    Could you elaborate on the last sentence in that paragraph. It looks like a non sequitur to me (it also looks like nonsense). What do you denote by “main street bias”?

38. #38 Gator

    February 11, 2009

    There is an old physics joke that ends with the punchline “First, assume a spherical cow.” That is what this model has done. They have made a very simple model. Now they need to show it is applicable to the real world. A simple model like this should be either blindingly obvious or shown to match real world data.

    The model is not blindingly obvious. It is too easy to come up with real-world situations that are not described by this model. Car bombs in the market, or the mosque. q is not well described as an average value over space and time. What about a Sunni that has to go into a Shite area? I.e., q is population dependent and not simply area dependent. Why would main streets be so much more dangerous? What about sectarian violence as various militias try to control neighborhoods and slums?

    The authors present no evidence from the real world that might be used to convince us that this overly simplified model is nonetheless useful. They could look at data in the US. They could send a team into Iraq. They could do lots of things, but have done nothing to show this model is applicable to anywhere in the world. They have done nothing to show that the parameters they chose have any connection to the real world.

39. #39 sod

    February 11, 2009

    Could you elaborate on the last sentence in that paragraph. It looks like a non sequitur to me (it also looks like nonsense). What do you denote by “main street bias”?

    Tim (an unimportant people like me) thinks that there is a higher risk of attacks in the REAL mainstreets.
    but those have a lot of “traffic” from people who do not live nearby. and people living directly in the mainstreet actually have a tiny chance of being polled under the lancet methodology.

    In a two subsystem framework, the market has to be located either in Si or So.

    it could stretch from one into the other.

    What about a Sunni that has to go into a Shite area?

    it is obvious, that religious heterogeneity of an area would be a very important factor, when looking for violence.

    but it has no place in their model, as it might not follow mainstreets…

40. #40 Robert Shone

    February 12, 2009

    sod writes:

    Tim (an unimportant people like me) thinks that there is a higher risk of attacks in the REAL mainstreets. but those have a lot of “traffic” from people who do not live nearby. and people living directly in the mainstreet actually have a tiny chance of being polled under the lancet methodology.

    That’s a fairly banal observation, and doesn’t invalidate MSB at all, no matter how you dress it up in Tim’s oddly-worded assertions.

41. #41 sod

    February 12, 2009

    That’s a fairly banal observation, and doesn’t invalidate MSB at all, no matter how you dress it up in Tim’s oddly-worded assertions.

    if you claim so, it must be true….

    you (again) don t get the problem. at all.

    let me help you out:

    if you chose a high n value (a huge part of the country wasn t possibly polled) and focus on “real big” mainstreets, then you get streets (or markets, if you prefer) that attract A LOT of visitors/traffic from outside its neighborhood. this is a problem for their model, as actually the casualty ratio for out/insiders will be very different from the one, that is based on where you live.

    if you consider mainstreets to be “little mainstreets”, with mainly local traffic/visitors, you will get a better out/insider ratio of casualties. but this is a problem for the study as well, as it means that more streets are considered mainstreets, bigger regions got polled and n is actually smaller than they assume it to be.

    and of course, all of these problems are independent from the gender problem, that you folks still prefer to ignore…

42. #42 Robert Shone

    February 12, 2009

    sod writes:

    this is a problem for their model, as actually the casualty ratio for out/insiders will be very different from the one, that is based on where you live.

    It’s not a problem for the model – rather it’s a matter of debate for the parameters that you plug into the model. (Same for your “problem” with the n-value). See comment #3.

43. #43 Bernard J.

    February 12, 2009

    David Kane and Robert Shone.

    Could you address the ‘gender problem’ that sod has repeatedly drawn attention to? I am curious to understand how you resolve this within the overall employment of the model in the paper.

44. #44 Tim Lambert

    February 12, 2009

    Since people are having trouble following sod’s gender argument, let’s make it quantitative using the parameters given in the paper.

    They have q=5 i.e sampled area is 5 times as violent as unsampled area. Working age males spend 1/4 of the time in the unsampled area, so their death rate is only 5×3/4 + 1×1/4 = 4 times the rate of the unsampled area. 2/7 of the population in the sampled area is working age males, so (4×2/7)/(4×2/7+5×5/7) = 24% of the violent deaths in the sampled area will be working-age males. But L2 found that 82% of the violent deaths were working-age males. That’s a pretty big difference, don’t you think?

    Conclusion: the model and/or parameters bear no relation at all to the Lancet study.

45. #45 Tim Lambert

    February 12, 2009

    We can also ask ourselves: what parameters do we need to plug into their model to match the observed distribution of violent deaths in L2?

    Well, n has no effect and the 2/7 working-age males seems to come from the Lancet study, so the only parameter we can choose is q. Solve for the q value that gives you 82% of deaths amongst working-age males and you get 1/42 (i.e unsampled area is 42 times as violent as sampled area). Plug that into their model along with n=10 and f=(13/14) and you get R = 0.11, implying that there 5.4 million violent deaths in Iraq. Or maybe, just maybe, something is wrong with their model.

46. #46 David Kane

    February 12, 2009

    Bernard J: I address the gender issue here. You may find other entries in that thread to be of interest.

    Tim: If the model were designed to predict the gender ratios of deaths than you might have a point. But the model does not do that. Nor does it predict the daily temperature in Baghdad. That doesn’t mean that the model is wrong, it just means that you are misusing it, as Robert explains in #142.

    Now you are, of course, free to argue that the parameter value for q = 5 is wrong and that, given higher mortality for men, a value like q = 10 would be more accurate. Perhaps you are right! If you are, then R (the bias) would be even higher. Is that the case you want to make?

    Again, the purpose of the paper is not to argue that q must be 5 or n must be 10 or any specific set of parameter values. The purpose is to provide a model which can be used — Have you finally given up the ghost in arguing that the derivation of the formula is wrong? — in a situation with non-universal sampling and to calculate the bias for different ranges of the various parameters.

47. #47 Tim Lambert

    February 12, 2009

    Err David, with their model the higher mortality rate for men implies that q is lower than 5, in fact it would have to be 1/42.

    And no, I’m not misusing their model. It purports to say something about deaths in Iraq and where they occur. It doesn’t say anything about temperature.

    I have never argued that the derivation of the formula was wrong. The formula is wrong because their model is wrong. The model is wrong because it assumes that there is no main street bias.

48. #48 Kevin Donoghue

    February 12, 2009

    David Kane: If the model were designed to predict the gender ratios of deaths than you might have a point. But the model does not do that. Nor does it predict the daily temperature in Baghdad.

    It is my understanding that total deaths = male deaths + female deaths. David, can you tell us what assumptions we need to make about the distribution of female deaths to rescue the MSB theory from sod’s critique?

    Tim Lambert: Solve for the q value that gives you 82% of deaths amongst working-age males and you get 1/42 (i.e unsampled area is 42 times as violent as sampled area).

    Ah, but that’s just for men! Maybe for women, children and the elderly the sampled area is 7 times as violent as as the unsampled area! Wouldn’t that give us a both-sexes all-age-groups weighted average q of 5? See, there’s always a way. (Although I’m not going to swear to it that even that fix works.)

    Coming soon from Johnson et al: Age and Gender Bias in Epidemiological Studies of Conflict Mortality.

49. #49 Gator

    February 12, 2009

    “All models are wrong, some models are useful.” This model is clearly not useful. The authors show no link of model predictions to reality, and it is too easy to think of realistic situations where the model is clearly wrong.

50. #50 Robert Shone

    February 12, 2009

    Tim Lambert writes:

    2/7 of the population in the sampled area is working age males, so (4×2/7)/(4×2/7+5×5/7) = 24% of the violent deaths in the sampled area will be working-age males.

    You forget that young males are more likely to be victims of violence for other reasons (this seems to hold in peaceful societies as well as in war-zones – for example, in the UK, males aged 16-24 are much more likely to be victims of violence than any other group). And given the inclusion of combatants in L2, this factor will probably be even more pronounced.

    Unless you introduce a lot more data and many more assumptions, the issue of gender is just another red herring, along with Tim’s hermetically-sealed Platonic market. Please see Nick’s post #111 again for comments on introducing further “subsystems”.

51. #51 Robert Shone

    February 12, 2009

    To address sod’s many posts which express the following on gender:

    as it is, this model suggests that the majority of violence victims found by Lancet should be women. many more elderly or kids. young men should be a tiny minority among those killed by violence!

    It simply does not follow, for the reason I’ve just given in #150

52. #52 Eli Rabett

    February 12, 2009

    Many of the violent deaths in Iraq took place in mosque bombings. The sexes are separated in mosques.

    The 3Ms mosques, marketplaces and mainstreets all have something to do with this, however, the entire mainstreet argument appears wrong to me. While bombings will be concentrated on mainstreets, death squads would be concentrated on back streets and in private houses. Abductions are a lot easier on narrow lanes than wide streets with lots of traffic. The whole Spagat thing appears to be a flight from reality.

53. #53 John

    February 12, 2009

    I would like to present my take on the mathematics of the gender issue (male vs. female deaths). I’ll start with an example.

    Assume that we are dealing with a population of 50 males (nm=50) and 50 females (nf=50), which together constitute a population of size 100 (n = nm+nf = 100). Let us assume that males are exposed to some disease with a probability qm=0.03 and females with a probability qf=0.01. How many males are expected to develop the disease? This is given by nmxqm = 50×0.03 = 1.5. The corresponding number for females is nfxqf = 50×0.01 = 0.5. In the total population of 100, altogether 1.5+0.5 = 2 people are expected to fall ill.

    What if, instead of having the gender specific probabilities, we are given the probability q that a person, regardless of gender, falls ill? This probability is given by weighted population average of the corresponding male and female probabilities, resulting in q = 1/(nm+nf)x(nmxqm+nfxqf) = 1/100x(1.5+0.5) = 0.02. So what can we do with this number? Since it is a population specific probability, we can use it to calculate the expected number of people in this population who will develop the disease, which is given by nxq =100×0.02 = 2. This of agrees with our above result, as it of course should. We cannot, however, use q to calculate gender specific expectation values. We can calculate qxnm = qxnf = 0.02×50 = 1 but this is meaningless at the level of the male population, or female population, by itself. This calculation will certainly not yield the expected numbers of males or females falling ill (1.5 and 0.5, respectively), since for that we need the gender specific probabilities as above. It can only be used to calculate the expected number of people falling ill disregarding their gender, or equivalently averaging over the gender.

    Let’s put this in the context of the JPR paper. There qi is defined as the probability of a violence related death to anyone present in the samplable region. It does not distinguish between males and females and represents a population average (or sub-population, in this case, as it is limited to individuals, males and females, in the samplable region). The same argument holds for qo pertaining to the non-samplable region. Here, then, qi plays an analogous role to q in my example. We cannot therefore use qi in the present case to calculate the gender specific expectation values any more than we can do so in the above example. To do that, we would need to know the gender specific probabilities (qm and qf in the example) and the numbers of males or females in the population (nm and nf in the example). In this case we have neither of these. If it is men who are sent out to take care of the potentially dangerous tasks, their probability of death could be significantly higher than its counterpart for women, and one would expect a higher number of male casualties than female casualties. Also, note that even if we know the gender composition of a typical family, it does not allow us to infer the underlying number of men and women in the population, because we don’t know the gender compositions of the other families. Since the JPR model does not distinguish between the genders but uses a population-wide probability in its formulation, it cannot be used to calculate the expected number of male or female casualties.

54. #54 sod

    February 12, 2009

    What if, instead of having the gender specific probabilities, we are given the probability q that a person, regardless of gender, falls ill?

    John, you didn t get to the core of this problem.
    we aren t “given” any “probabilities that a person regardless of gender falls ill”.

    the model is very clear:

    1. people who live in a mainstreet zone, had a higher probability of being polled.

    2. your risk of death increases significantly, ba the time that you spend in the mainstreet zone.

    3. females (from the polled region) spend significantly more time in the mainstreet zone, than their males do.

    conclusion: we would expect to find more females killed by violence in lancet poll, than males.

    if you think that male and female have a different risk, that is fine (as it is obviously true). but it is in contradiction to their claim, that where you live (and spend your time) is THE important factor for your risk to get killed.

    if you think that males do the more dangerous “jobs”, that is fine as well. but again it is contradicting their claim, that the location of your home determines your risk of death. (if males from the polled regions and those from outside do the risky job at the market, their death risk will be similar/same)

55. #55 sod

    February 12, 2009

    You forget that young males are more likely to be victims of violence for other reasons (this seems to hold in peaceful societies as well as in war-zones – for example, in the UK, males aged 16-24 are much more likely to be victims of violence than any other group). And given the inclusion of combatants in L2, this factor will probably be even more pronounced.

    true. but i haven t heard a lot about a “mainstreet bias” among british victims of violence.

    obviously more violence will happen on mainstreets. but how much time a young male spends there, has little or nothing to do, with how close to the mainstreet he lives. (at least not in a “one street further” sense, like in the Spagat version of mainstreet bias)

    and please could you guys stop pretending that the gender issue isn t par of the paper. Spagat makes EXPLICIT assumptions about gender. he is using it, to calculate how much time people spend in the danger zone!

56. #56 Kevin Donoghue

    February 12, 2009

    Certainly Johnson et al. are quite specific about the fact that they are assuming two working-age males in a seven-person household. These males spend six hours per 24-hour day outside their own zone. The other five members of the household spend all their time in their own zone. Also, the survey space (Si) has one-tenth the population of the unsurveyable space (So).

    Sorry folks, but complaining about sod’s taste in fonts won’t get you around this problem. (Although FWIW, I would like my reading made a little easier too.) Talk of adding subsystems won’t get you around it either. We are faced with certain facts of arithmetic here:

    Total population = working-age males + others, and

    Total deaths = working-age male deaths + other deaths.

    If anyone can generate aggregate numbers consistent with these facts, and the parameter values suggested in the JPR paper, please share them. I’ve tried and I can’t seem to avoid the kind of absurdities Tim comes up with.

57. #57 Kevin Donoghue

    February 12, 2009

    In #148 above I wrote, facetiously: “Maybe for women, children and the elderly the sampled area is 7 times as violent as as the unsampled area! Wouldn’t that give us a both-sexes all-age-groups weighted average q of 5? See, there’s always a way. (Although I’m not going to swear to it that even that fix works.)”

    Just as well I wasn’t going to swear to it. That fix doesn’t work either. (Because the aggregate q isn’t a weighted average of the subset qs.)

    So, what does the defence have to say?

58. #58 Tim Lambert

    February 12, 2009

    Note also that in their model working-age males have a higher violent death-rate because of working-age males from the unsampled area visiting the sampled area — with their parameters they would make up 40% of the deaths despite being only 29% of the population. But their model predicts that the Lancet survey would be so strongly biased that it would find that just 24% of the deaths were working-age males.

59. #59 Robert Shone

    February 13, 2009

    Tim Lambert writes:

    Note also that in their model working-age males have a higher violent death-rate because…

    No. In your misreading of their model, males have a higher risk. They make no such claim, and so such prediction follows from their gender assumptions for example parameter values for f. (See my comment #150).

60. #60 Robert Shone

    February 13, 2009

    sod writes:

    Spagat makes EXPLICIT assumptions about gender. he is using it, to calculate how much time people spend in the danger zone!

    Yes, but it doesn’t follow from this that your predictions (and Tim’s) regarding male/female violent death ratio can be imputed to the MSB model. (See my comment #150)

61. #61 Robert Shone

    February 13, 2009

    To summarise, on the gender question:

    1. The MSB authors suggest parameter values for f based on a few gender-related assumptions.

    2. Tim Lambert and sod impute to the MSB model, based solely on 1., predictions regarding male/female violent-death ratio.

    3. This is invalid for reasons which should be obvious (see my comment #150).

62. #62 Kevin Donoghue

    February 13, 2009

    Robert Shone accuses Tim: In your misreading of their model, males have a higher risk.

    From the draft paper: “Given the nature of the violence, travel is limited; women, children and the elderly tend to stay close to home.”

    The lit. crit. approach to these discussions is always entertaining, but statistics is not a literary sort of field. It is subject to what Alwyn Young called the tyranny of numbers. Is any defender of the paper prepared to face the arithmetic?

63. #63 Robert Shone

    February 13, 2009

    Well, Tim does misread the MSB model, since it doesn’t claim that “working-age males have a higher violent death-rate because of working-age males from the unsampled area visiting the sampled area…” (which is the comment of Tim’s that I was referring to, as Kevin Donoghue well knows).

64. #64 Kevin Donoghue

    February 13, 2009

    But you misread Tim – he didn’t say that the paper claims that. He said that “in their model working-age males have a higher violent death-rate because of working-age males from the unsampled area visiting the sampled area” – it’s not an explicit claim made in the paper but it is a logical implication of the assumptions made. There are more males in the unsampled area (So) because it’s ten times as populous. The working-age males visit the more dangerous survey space (Si) while everyone else stays at home. QED, pretty much, unless you want a proof written in symbolic logic.

65. #65 Robert Shone

    February 13, 2009

    Kevin Donoghue writes:

    it’s not an explicit claim made in the paper but it is a logical implication of the assumptions made.

    It’s not a “logical implication” that the MSB authors have “in their model” (to use Tim’s words). It’s Tim’s own problematic logic, which he imputes to the MSB paper. (Why problematic? Pretty obvious – see #150).

66. #66 sod

    February 13, 2009

    Well, Tim does misread the MSB model, since it doesn’t claim that “working-age males have a higher violent death-rate because of working-age males from the unsampled area visiting the sampled area…” (which is the comment of Tim’s that I was referring to, as Kevin Donoghue well knows).

    Robert S., you did not understand this. again. basically everything that you write is either a pretty wild claim or simply false. this one is the latter.

    the values of n and fi and fo, plus what they write about males and females in their paper, makes it a simple task, to calculate the ratio of males inside the mainstreet zone.

    Assuming that there are two working-age males per average household of seven (Burnham et al., 2006), with each spending six hours per 24-hour day outside their own zone,

    a man living inside the zone spends three times as much time in the mainstreet zone,as one living outside of it. but there are many more man, living outside this zone! (n=10)

    if Lancet had polled males on the (main)street, they would have found the majority of them not living nearby!

    *******

    the Spagat paper makes two claims, that contradict each other, when you factor in a simple fact:

    1. Claim: where you live is very important for your risk of being a victim of violence.

    2. Claim: males travel around a lot. they spend 1/4th of a day outside their home zone.

    3. FACT: the majority of victims of violence in iraq are male.

    if the killing (of males) in the mainstreet area was happening indiscriminating on the street, it would kill more men who live outside the zone, than those living inside. (see my calculations above) this would lead to a much smaller bias, than the paper claims.

    if the killing (of males) in the mainstreet zone is targeting them at their homes (not well supported by evidence, btw), the question is, why their females don t suffer a similar toll of violence.

67. #67 Robert Shone

    February 13, 2009

    sod writes:

    Robert S., you did not understand this. again.

    Sod, with respect, I think I do understand. Tim and yourself are attempting to impute to the MSB paper claims (or logical implications) about male/female violent-death ratio, based on that model’s gender-related assumptions for suggesting values for f.

    What I’m saying is that other important factors besides space-time location affect the male/female violent-death ratio (for example: is a male combatant more likely to be killed; are men more likely to be targeted by gunfire, etc? – see my earlier comment #150).

    Obviously space-time location in itself is an important factor in the risk of death, and it raises the whole possibility of bias (which is what the MSB paper tackles). However, you can’t impute specific predictions of male/female violent-death ratio to the MSB model based solely on the gender-related assumptions for suggesting f.

68. #68 Ron

    February 13, 2009

    Sod wrote: “if the killing (of males) in the mainstreet area was happening indiscriminating on the street, it would kill more men who live outside the zone, than those living inside. (see my calculations above)”

    What?? Have you seriously read and thought about this statement. It is ridiculous.
    The L2 analysis makes an implicit assumption that there is *no* main street bias. The JPR paper takes the first step beyond that, which is to look at what happens if attacks are not completely random in terms of types of urban layout (which is a reasonable question to ask). But somehow out of this, you have managed to tie yourself up into knots to such a degree that your criticism of JPR is based on details which are way beyond the scope of either JPR or the L2 implicit assumptions. If you worry so much about gender and location issues being wrong in JPR (and hence important), why don’t you attack the L2 in the same way? When they backed out totals, no corrections were made there for such street-location/neighborhood/gender mobility biases? Why not? Can one assume that they don’t exist? Are they obviously exactly zero? No, they are not — and that is the point of the JPR paper, to show that such biases are not obviously zero.

    Would you demand your money back from a model train that you purchased, just because it didn’t have real seats in it when you unpacked it at home? A model is a model, and its purpose is to show the effect of what might lie hidden in reality. ‘Reality’ here is the L2 study, and the model in JPR is aimed at showing what might be missing from this. End of story.

69. #69 Kevin Donoghue

    February 13, 2009

    sod,

    You don’t have a hope in hell of persuading Robert Shone to look at the numbers. That’s not his scene. It’s all about words. In this case, is it okay to say that something happens in the model when what you mean is that the assumptions clearly imply that it happens? If you’re fond of reading Heidegger and Derrida you can maybe have an interesting conversation with him. Then again, maybe not. I’m not so I wouldn’t know.

70. #70 Kevin Donoghue

    February 13, 2009

    Would you demand your money back from a model train that you purchased, just because it didn’t have real seats in it when you unpacked it at home?

    No, but if the wheels didn’t fit the tracks I would. There is a concept in law: fitness for purpose. As anyone who has read this thread will surely understand by now, Johnson et al. is not of merchantable quality.

71. #71 sod

    February 13, 2009

    A model is a model, and its purpose is to show the effect of what might lie hidden in reality. ‘Reality’ here is the L2 study, and the model in JPR is aimed at showing what might be missing from this. End of story.

    Ron, you missed the main part of the story. both Lancet and reality agree, on males being the majority of victims. but the Spagat model makes such an outcome extremely unlikely!

    all my comments above are based on the assumptions of the Spagat model.

    your train seats example was the most stupid one, among several pretty moronic “examples” that were given by those who support the Spagat paper.

    i am really curious:
    please give an explanation: what is killing those males who live in the mainstreet zone, but leaves visiting males and their households untouched?

    shouldn t this explanation be in the paper? why isn t this MORE important fact part of the name of the “bias”?
    mainstreet males get killed when they carry the trash outside-bias” might be a start…

72. #72 sod

    February 13, 2009

    What I’m saying is that other important factors besides space-time location affect the male/female violent-death ratio (for example: is a male combatant more likely to be killed; are men more likely to be targeted by gunfire, etc? – see my earlier comment #150).

    well, i agree with Kevin: numbers possibly simply aren t your thing.

    so i ll try three simple questions:

    1. (as above) if there is a factor that is more important than “mainstreet bias”, then why talk about “mainstreet bias”?

    2. do you really believe, that the majority of combatants in Iraq stay in the road that they live in?!?
    (hint: attackers AND defenders may take casualties occasionally…)

    3. how does the gunfire pick out local men among the majority of non-local men on a mainstreet?

73. #73 Robert Shone

    February 13, 2009

    Kevin Donoghue writes:

    You don’t have a hope in hell of persuading Robert Shone to look at the numbers. That’s not his scene. It’s all about words.

    Which numbers are we talking about? Tim Lambert’s? Here they are again:

    Note also that in their model working-age males have a higher violent death-rate because of working-age males from the unsampled area visiting the sampled area — with their parameters they would make up 40% of the deaths despite being only 29% of the population. But their model predicts that the Lancet survey would be so strongly biased that it would find that just 24% of the deaths were working-age males.

    What is wrong with Lambert’s numbers? They’re derived based on the false assumption that the only factor affecting male/female violent-death ratio is space-time location. Not surprisingly, the MSB paper doesn’t make that assumption. Lambert is falsely imputing stupid assumptions to the MSB paper.

74. #74 Ron

    February 13, 2009

    Sod wrote “if there is a factor that is more important than “mainstreet bias”, then why talk about “mainstreet bias”?”

    The JPR paper looks at a potential bias due to a type of street bias. It does not look at potential bias due to other possibilities — that was not their job. But it *was* the job of the L2 authors!!! This is an obvious possible bias — now you are raising others. So do us all a favour, get on to the L2 authors and ask them how they managed to disregard these all these plausible potential biases. The L2 authors are either so very smart that they know how to estimate them and disregard them (… and yet, not so smart as to add this to the publications??) Or maybe they were smart enough to know that there are many potential biases that they haven’t accounted for, but hoping that noone would notice?

    The L2 authors, having made a claim of a result based on surveys, have the scientific duty to show that potential biases that many people would think of as reasonable, are not in fact present.

    Do you get this? Let me repeat….. *The L2 authors, having made a claim of a result based on surveys, have the scientific duty to show that potential biases that many people would think of as reasonable, are not in fact present.*

    You bunch of L2 defenders are sad, sad cases ….. God knows what you do for a career….

75. #75 Kevin Donoghue

    February 13, 2009

    …get on to the L2 authors and ask them how they managed to disregard these all these plausible potential biases.

    To which they will reply: we discussed potential biases at great length; please read the paper.

76. #76 Kevin Donoghue

    February 13, 2009

    Which numbers are we talking about? Tim Lambert’s?

    No. The numbers in Johnson et al.; q=5, n=10, f=5/7 + 2/7 x 18/24=13/14 etc.

    Assumptions have consequences.

77. #77 Gator

    February 13, 2009

    Robert Shone is really arguing that it is not fair to compare the Spagat model to reality. It is a model and the model does what it does. All bow to the model. Treat the model on its own terms.

    I’m wondering though, in this framework, how RS figures the paper has anything to do with the actual situation in Iraq? Can he connect anything in the model with any facts?

78. #78 Ron

    February 13, 2009

    Kevin Donoghue: “To which they will reply: we discussed potential biases at great length….”

    Wrong. Their discussion of potential biases is not complete.
    ‘At great length’ is neither a correct summary of their paper, nor does it mean such a discussion is complete.

79. #79 Ron

    February 13, 2009

    Gator write: “I’m wondering though, in this framework, how RS figures the paper has anything to do with the actual situation in Iraq? Can he connect anything in the model with any facts?”

    Can you connect the L2 inference to facts? i.e. L2 sampled households in particular types of street environment, and made the huge unjustified leap of multiplying that up to the level of the entire population.

    If L2 had said “We sampled households in ‘X’ type of street environments, and found that Z percent of them had known casualties” then that is one thing. But to then say “and therefore multiplying up to the level of the population, and ignoring the restriction Y, we will also get Z percentage” — that is just folly…..

80. #80 John

    February 13, 2009

    After reading some of the posts it is clear that the “gender problem” arises from an erroneous interpretation of the model and in that respect I agree with the point raised in 150 and 167 by Robert Shone, which are closely related my comment in 153. You really need to make assumptions about male-specific and female-specific probabilities before you can make any statements about the number of male and female casualties. One may assume that males and females have identical probabilities to die, but then the entire gender issue is contingent on this assumption. Exploring this assumption may be interesting, but it is not an assumption that is made in the MSB papers. Again, I’m talking about the “probability of a male to be killed” and the “probability of a female to be killed”. The mixing of people is conceptually separate from death probabilities and is mathematically governed by an independent parameter. To estimate mixing, one may or may not used gender based arguments, but in either case the male and female death probabilities need be specified separately *if* one wishes to make statements about male and female casualties.

    I would ask those of you who believe the gender problem to be genuine to provide *mathematical expressions* for the male death probability and for the female death probability, as well as for the expected numbers of male and female casualties. I’m not asking for an example, but an expression using the notation of the paper. If we are discussing the model as is, you will not need to introduce any additional notation nor make any additional assumptions, just the expression will be sufficient.

81. #81 Ron

    February 13, 2009

    Gator writes: “…All bow to the model”

    So Gator, what is your model for estimating potential biases? Merely to say “there aren’t any biases, beyond the ones that L2 dismissed in words” is not a scientific argument.

    If you are a scientist, you will know that systematic errors have to be estimated in order to be discounted. If you can think of a potential error, then estimate its potential effect. That is how science works.

    “Bow to L2″ is what you really wanted to say — right?

82. #82 Ron

    February 13, 2009

    My above post should have said:

    If L2 had said “We sampled households in X type of street environment, and found that Z percent of them had known casualties” then that is one thing. But to then say “and therefore multiplying up to the level of the population, and ignoring the restriction X, we will also get Z percentage” — that is just folly…..

83. #83 Kevin Donoghue

    February 13, 2009

    Ron,

    This thread is about the merits of Johnson et al. If you want to make the case that it’s a good paper, castigating Burnham et al. doesn’t do it. Even if Burnham issues a press release tonight saying “Sorry, I made the whole thing up”, that won’t make Johnson et al. a good paper. For the reasons why it isn’t, read Tim’s original post and the comments in the thread.

84. #84 Bruce Sharp

    February 13, 2009

    Ron,

    If you look at the messages in this thread, you’ll notice that there is actually very little discussion of L2. I don’t believe anyone here has claimed that L2 is totally free of bias, or that its estimates are beyond reproach.

    The sampling design of L2 was intended to eliminate mainstreet bias. Johnson et al are not arguing that the L2 authors were ignorant of the phenomenon; they are arguing that the measures taken were insufficient, and they present a model which purports to prove this point.

    Many of the people here find the assumptions underlying that model to be dubious, and further argue that the distribution of the casualties inferred by the model are entirely unrealistic. If the model is wrong, however, that doesn’t mean that L2 is correct; after all, there is more than one way to be wrong.

    You might want to look back at post #4 if you want an example of a potential problem in L2, which does not involve mainstreet bias. And since you were wondering what some of the people here do for a career, I’ll help you out a little: the guy who wrote post #4 is a professor of demography.

    Regards,
    Bruce

85. #85 Ron

    February 13, 2009

    Bruce wrote: “The sampling design of L2 was intended to eliminate mainstreet bias.”

    With all due respect, this is not the case. It was intended to eliminate heterogeneity at a larger scale (e.g. towns, directorates etc.) but not the fine-grain heterogeneity at the level of streets.

    Bruce write: ” Johnson et al are not arguing that the L2 authors were ignorant of the phenomenon…”

    They *are* indirectly arguing this. Since the words main-street-bias were coined by them apparently, and no mention is made in any of the Burnham et al. team’s previous publications about such possible heterogeneity at the street-level, I think that the Johnson et al. article is precisely flagging up something which L2 did not address. Whether it is a large or small bias can then be debated — but the fact that prior to JPR it was not mentioned at all, makes the JPR paper interesting in itself. The fact that the JPR authors then produce a quantitative expression for the mean-field correction (i.e. heterogeneity) at the lowest order in order to capture this possible missing L2 street-level heterogeneity (i.e. two distinct subsystems rather than one) makes the JPR article very worthy. The JPR article then *suggests* some parameter values, inviting the L2 authors to provide more information to modify these estimates. All reasonable in my view.

    Bruce wrote: “..the guy who wrote post #4 is a professor of demography.”

    This is very, very worrying. It is just as worrying as when several famous epidemiologists initially rushed to state there is nothing wrong with the L2 approach. How can he, or others, vouch for something without even estimating the possible biases?
    I happily agree that there may be several other sorts of biases which are equally possible, and some may prove to be more important than the JPR bias. But that remains to be seen when L2 release more information. Before that it is impossible to prove one way or the other).
    To the professor of demography: What happened to the healthy skepticism of academics?
    Is that the kind of critical thinking that is taught in demography?

    In short: Johnson et al. are looking at one potential source of bias in generic sampling within systems with heterogeneity. They then suggest an application in Iraq. The JPR stands by itself in terms of an abstract model. No problem. To get truly Iraq-specific model would require Iraq-specific knowledge about what to model — which comes back to requiring Iraq-specific information about street heterogeneity during sampling.

    Finally, it is clear that Tim Lambert’s criticisms are merely intended to deflect critical attention away from L2. Let’s talk about L2 — oh I remember, it is to all practical purposes, bias free. No need to ever question its assumptions….

86. #86 ron

    February 13, 2009

    I just randomly walked out onto my street and asked the first 5 people I met if they know the color of my car. 4 did. Multiplying this 80% record up to the population level, approximately 50 million people know the color of my car. (Or maybe, I ought to mention that 4 of these 5 people happen to live a few houses away…)

    Still sure that MSB is way off the mark? Why don’t you try this simple exercise yourselves?

    I know that there will be posts objecting to this experiment. I am sure that these criticisms will point out how my ‘simple case to test L2′ implicitly mis-applies the L2 sampling procedure — yet this is *exactly* what Tim Lambert did with his supposed ‘simple case to test JPR’ earlier on this page.

87. #87 sod

    February 13, 2009

    What is wrong with Lambert’s numbers? They’re derived based on the false assumption that the only factor affecting male/female violent-death ratio is space-time location. Not surprisingly, the MSB paper doesn’t make that assumption. Lambert is falsely imputing stupid assumptions to the MSB paper.

    i doubt that you understand their model. the calculation of R is dependent of the fi and fo values. those are simply defined by their assumptions on male and female behaviour. fi is very high, because females are assumed to stay at home. fo is very high, because only males are assumed to leave their area.

    here is a simply thing to do: (weird, that i didn t have this idea before)

    we know that the vast majority of victims of violence are male. so why not simply restrict the model to male persons? n stays at 10 (stupid, but ok for the moment), so does q. fi is lowered to 3/4 8male leave the mainstreet area only for 6 hours, fo is reduced to 3/4 as well (see Spagat paper) lo and behold, we nearly managed to reduce R to 1.83!!!

    The JPR paper looks at a potential bias due to a type of street bias. It does not look at potential bias due to other possibilities — that was not their job. But it was the job of the L2 authors!!! This is an obvious possible bias — now you are raising others.

    ahm, no. the “male bias” isn t a bias. it is a RESULT of the study. and a fact, confirmed by all other data sources…

    The L2 authors, having made a claim of a result based on surveys, have the scientific duty to show that potential biases that many people would think of as reasonable, are not in fact present.

    no. actually those attacking the paper are supposed to find evidence for an error in the paper. Spagat hasn t done that.

    the Lancet paper makes a pretty strong point: they used this method, because they thought other methods to be to risky for the doctors doing the polling.
    i think that those of us who didn t walk the streets of iraq after the Samarra bombing should think hard, before they make accusations.

88. #88 Gator

    February 13, 2009

    Ron: Wow, what a strawman.

    The fact remains that Spagat et al have done nothing to connect their model to anything in the real world. Until they do it is useless to comment about what it says about the real world when it is so easy to come up with real world situations that contradict the implications of the model.

89. #89 Bruce Sharp

    February 13, 2009

    Hi Ron –

    The term “mainstreet bias” may have originated with Johnson et al, but the concept certainly isn’t new; it’s just a variation of geographic bias. And whether or not it is a large bias or a small bias is directly relevant here. There is no way to reflect every aspect of heterogeneity in a survey… but we don’t need to reflect every aspect of differentiation. We only need to reflect those which will affect our results.

    If you find the concept of the paper to be very worthy, that’s fine. The concept could be sound, but that does not mean that is is relevant to Iraq. A large part of the dispute here centers on the parameter values that the authors “suggest.” Garbage in, garbage out: even if the model is sound, invalid parameters will yield invalid results.

    Regarding your comment that the professor of demography’s comments are “very, very worrying,” I’m not sure why you are worried, or what you think Robert was “vouching” for. I’m also not sure whether you meant that Robert was being insufficiently skeptical, or whether I was being insufficiently skeptical. In either case, it seems that what bothers you isn’t a lack of skepticism: it’s the fact that the skepticism is directed toward a model that you’ve chosen to accept at face value.

    As to Tim’s intentions, it seems to me that he is addressing the criticisms of L1 and L2. That is rather different than deflecting them.

    Regards,
    Bruce

90. #90 sod

    February 13, 2009

    You might want to look back at post #4 if you want an example of a potential problem in L2, which does not involve mainstreet bias. And since you were wondering what some of the people here do for a career, I’ll help you out a little: the guy who wrote post #4 is a professor of demography.

    what (the other) Robert wrote in comment #4 is perfect, but has a major flaw for this discussion:
    all the people arguing for the Spagat paper, are 100% convinced, that both Lancet papers are complete fabrications. that the total deathtoll comparison between the two contradicts the Spagat paper has ZERO value in a discusSion with them.
    just keep the argument in mind, in case you ever happen to talk to some rationale people..

    I just randomly walked out onto my street and asked the first 5 people I met if they know the color of my car. 4 did. Multiplying this 80% record up to the population level, approximately 50 million people know the color of my car. (Or maybe, I ought to mention that 4 of these 5 people happen to live a few houses away…) Still sure that MSB is way off the mark? Why don’t you try this simple exercise yourselves?

    oh, i just did. i numbered the two “mainstreets” of my small home town. (actually i live in the mainstreet bias zone..) thewn chose one of them randomly. numbered the roads departing from it, choosing one at random again. on that road i chose 5 (random houses).
    i ll ask them for the colour of my car tomorrow…

    I know that there will be posts objecting to this experiment. I am sure that these criticisms will point out how my ‘simple case to test L2′ implicitly mis-applies the L2 sampling procedure — yet this is exactly what Tim Lambert did with his supposed ‘simple case to test JPR’ earlier on this page.

    no. you still don t understand what he did.

91. #91 Kevin Donoghue

    February 13, 2009

    sod: lo and behold, we nearly managed to reduce R to 1.83!

    But if I’m understanding you correctly the bias for non-workmen is then 3.67 because f=1 for them. It’s the bizarre assumption that fi=fo that produces most of the craziness, I think. They could have told a more plausible story with fo>fi, the idea being that people avoid the dangerous survey space as much as possible. But then they would have had to think about the underpinnings of the model, which they preferred to leave vague. Is a mosque in Si, or in So, and how do you tell? The paper doesn’t say.

92. #92 ron

    February 13, 2009

    Kevin Donoghue said: “….It’s the bizarre assumption that fi=fo that produces most of the craziness, I think.”

    The JPR paper does not need to make this assumption. See Equation (4) of JPR — it is general. As in all theory papers, it is then interesting to look at certain special cases of the parameters to get a feel for what is going on. But Eq. (4) and the whole JPR theory does *not* have this fi=fo limit applied.

93. #93 ron

    February 13, 2009

    Sod wrote: “…no. you still don t understand what he did.”

    Err…. I don’t understand what L2 actually did (not what they say they did, what the team actually did).

    As for TL? Yes, I fully understand what he said/did. His criticisms are misguided.

    PS. I would place good money on your hometown layout being quite a bit different from Baghdad….

94. #94 ron

    February 13, 2009

    Sod said: “…small home town…”

    Wait a minute, you are a genius. So small home town maps to Baghdad in terms of layout? Even though your hometown has only two mainstreets… and is small.

    Errr…. look down and check the hole you just shot in your foot..

95. #95 ron

    February 13, 2009

    Sod wrote: “…all the people arguing for the Spagat paper, are 100% convinced, that both Lancet papers are complete fabrications. that the total deathtoll comparison between the two contradicts the Spagat paper has ZERO value in a discusSion with them….”

    Incorrect about the percentage. About 90+….

96. #96 sod

    February 14, 2009

    As in all theory papers, it is then interesting to look at certain special cases of the parameters to get a feel for what is going on. But Eq. (4) and the whole JPR theory does not have this fi=fo limit applied.

    the problem is, you change the parameters so that they make sense, and the paper stops contradicting the Lancet results.

    Err…. I don’t understand what L2 actually did (not what they say they did, what the team actually did).

    feel free to replicate their results. nobody is holding you back!

    Wait a minute, you are a genius. So small home town maps to Baghdad in terms of layout? Even though your hometown has only two mainstreets… and is small.

    so mainstreet bias does only apply to towns the size of Baghdad? that point is interesting. where does Spagat say this?

    and you might notice that my method is much more similar to the lancet one, than what you proposed..

    As for TL? Yes, I fully understand what he said/did. His criticisms are misguided.

    no, you didn t understand what Tim did. if you were interested in car colours, he would advice you to look closely at gas stations.

97. #97 Kevin Donoghue

    February 14, 2009

    sod,

    I think the following probabilities of death will get you results which, though implausible, are not completely insane:

    Workmen in Si: 0.075
    Workmen in So: 0.015
    Others in Si: 0.005
    Others in So: 0.001

    Even these values don’t quite work, but I think that’s because the model is constructed in such a way that it can’t be sensibly disaggregated into sub-models. Obviously that, if true, is a drawback of the model; but it’s hardly the worst.

    John,

    Thanks for pointing me in (I hope) the right direction. I’m sure sod and Tim don’t need to be reminded of the basics but I do, now and then. Not sure if my comment to sod answers your question but it’s the best I can do for now.

    Ron,

    Equation (4) is just a scaled-up ratio of two expected values. I think you’ll find even demography professors can come up with things like that. But it took three physicists, an economist and a statistician to rig the parameters so as to get the results Tim quite rightly objects to. Equation (1) wouldn’t bother anyone very much if it were not for the specific numbers Johnson et al. are pushing. However I wouldn’t like it myself, since even in the general case it assumes Iraqis are stupid (fi=fo and independent of q). I don’t usually like models which assume such extreme irrationality.

98. #98 ron

    February 14, 2009

    Sod wrote: “feel free to replicate their results. nobody is holding you back!”

    First step: Get L2 to tell us exactly what their survey team did.

    Sod wrote: “…mainstreet bias does only apply to towns the size of Baghdad?”

    JPR applies as a general theory (i.e. Eq. (4)) to any system with heterogeneity of two subsystems. EPL to two or more. The L2 street bias will have varied from town to town in its value. Is that so hard to understand for you?

    Sod wrote: “…no, you didn t understand what Tim did”

    Yes I did.

    KD wrote: “..Equation (4) is just a scaled-up ratio of two expected values. I think you’ll find even demography professors can come up with things like that.”

    Good, then have any of them proved that Eq. (4) is mathematically incorrect? Not just that “one can do a better approximation” — I mean they have to show that it is mathematically incorrect??

99. #99 ron

    February 14, 2009

    KD wrote: “….Equation (4) is just a scaled-up ratio of two expected values. I think you’ll find even demography professors can come up with things like that. But it took three physicists, an economist and a statistician to rig the parameters so as to get the results Tim quite rightly objects to.”

    Where in the literature does Equation (4) appear? It doesn’t. So the three physicists, an economist and a statistician did a little more than you are willing to give them credit for…

    Equation (4) does *not* assume fi=fo. Is that clear?

    Want it to be q-dependent? Derive an appropriate form yourself…. What non-linear dependence will you chose? Hmmm, I bet you wish you knew more about the specific L2 neighborhoods surveyed, then you could pin down the q-dependence. Aha, now you agree with me — L2 need to release info about this in order to make a proper evaluation about whether MSB is a major bias.

100. #100 ron

     February 14, 2009

     Sod wrote: “…no, you didn t understand what Tim did. if you were interested in car colours, he would advice you to look closely at gas stations.”

     You cannot ask a gas station questions. And since all this is about asking people questions (i.e. a survey) then you have problem. You are actually admitting that where you ask people questions can affect the result. QED.

     Go back to my earlier color-of-car survey example. To me, it sums up the essence of MSB. And it is reasonable, simple — and beyond L2′s considerations of potential bias…

     It encapsulates a major, major flaw in L2. There may be many others, but MSB lives as a possibility…… It will do until L2 clarify exactly what was and wasn’t surveyed. Of course, they don’t want to release this data (irrespective of security issues) since it may invalidate their study….