Unfortunately, the Journal of Peace Research has published the badly flawed "Main Street Bias" paper. My earlier criticisms still apply, so I'm reposting them. Consider this the first draft of a reply to their paper.
The authors argue that main street bias could reasonably produce a factor of 3 difference.
How did they get such a big number? Well, they made a simple model in which the bias depends on four numbers:
-
q, how much more deadly the areas near main street that were sampled are than the other areas that allegedly were not sampled. They speculate that this number might be 5 (ie those areas are five times as dangerous). This is plausible -- terrorist attacks are going to made where the people are in order to cause the most damage.
-
n, the size of the unsampled population over the size of the sampled population. The Lancet authors say that this number is 0, but Johnson et al speculate that it might be 10. This is utterly ridiculous. They expect us to believe that Riyadh Lafta, while trying to make sure that all households could be sampled, came up with a scheme that excluded 91% of households and was so incompetent that he didn't notice how completely hopeless the scheme was. To support their n=10 speculation they show that if you pick a very small number of main streets you can get n=10, but no-one was trying to sample from all households would pick such a small set. If you use n=0.5 (saying that they missed a huge chunk of Iraq) and use their other three numbers, you get a bias of just 30%.
-
fi, the probability that someone who lived in the sampled area is in the sampled area and fo the probability that someone who lived outside the sampled area is outside the sampled area. They guess that both of these numbers are 15/16. This too is ridiculous. The great majority of the deaths were of males, so it's clear that the great majority were outside the home. So the relevant probabilities for f are for the times when folks are outside the home. And when they are outside the home, people from both the unsampled area and the sampled area will be on the main streets because that is where the shops, markets, cafes and restaurants are. Hence a reasonable estimate for fo is not 15/16 but 2/16. If use this number along with their other three numbers (including their ridiculous estimate for n) you get a bias of just 5%.
In summary, the only way Johnson et al were able to make "main street bias" a significant source of bias was by making several absurd assumptions about the sampling and the behaviour of Iraqis.
- Log in to post comments
They guess that both of these numbers [fi and fo] are 15/16.
Not that it matters much, but in the last draft I saw they were both 13/14. Did they go with 15/16 finally or are you just pasting your early criticism exactly as you wrote it?
Since Robert Shone hasn't shown up yet, I will make his point for him: you are just making up your parameter values. The obvious response is of course that that's all the authors themselves are doing.
First, I think it's important to note that Tim Lambert's original post opened with a misrepresentation of Jon Pedersen's views (taken from Stephen Soldz) on main street bias. I pointed this out at the time, but Lambert still hasn't corrected it. I emailed Pedersen about it back in December 2006, and he responded as follows:
"Yes, probably Stephen Soldz confused the issue somewhat here. There are actual several issues:
1) I very much agree with the MSB-team that there is some main stream bias, and that this is certainly an important problem for many surveys - not only the Iraq Lancet one.
2) I am unsure about how large that problem is in the Iraq case - I find it difficult to separate that problem from a number of other problems in the study. A main street bias of the scale that we are talking about here, is very, very large, and I do not think that it can be the sole culprit.
3) The MSB people have come up with some intriguing analysis of these issues."
(Jon Pedersen, email to me, 4/12/06)
Second, the criticism from Tim Lambert is directed at one set of parameter values which was presented only as an illustrative example by the msb authors (who later added an exploration of the parameter space).
In other words the criticism misses the point that the actual bias could be determined only as a result of disclosure by the Lancet authors on basics such as sampling procedures and main streets selected as starting points, etc.
So this brings us back, in a way, to the AAPOR thing. The Lancet authors still haven't disclosed the basic level of information which is obviously necessary to assess how their claim of giving all households an equal chance of selection holds up.
If you're extrapolating from 300 actual violent deaths to 601,000 estimated violent deaths, based on this claimed sample-randomness, then it would seem pretty important that the sampling scheme could be assessed in some way. Currently it can't be, because nobody outside the Lancet team knows what that sampling scheme entailed.
I haven't read the published version so perhaps my criticism of it has been handled. However, on the chance it hasn't been dealt with, I'll repeat it.
The main methodological difference between the 2004 Roberts paper and the 2006 Burnham paper was the way the starting point was selected. Johnson et al. proposed "main street bias" as the explanation for the difference between the two papers in the number of estimated violent deaths. But one of the differences in the findings (as opposed to the method) was that there was a corresponding decrease in non-violent deaths. MSB doesn't explain that.
A simpler explanation that addresses both the increase in violent deaths and the balancing decrease in non-violent deaths without resorting to a putative MSB is that there was a problem with attribution of cause of death. This explanation is consistent with my observations in other surveys and registries: there is often much more ambiguity about the cause of death than that a death occurred at all.
We can suggest various parameter values to plug into the msb formula. Tim claims that the value originally suggested as an example for f (15/16) was "ridiculous". I'd argue that Tim's own suggested value (2/16) was actually the really ridiculous suggestion, since it implied that the average Iraqi (including women, children and the elderly) spends only 3 hours out of each 24-hr day in their own home/zone (presumably sleeping), and spends the other 21 hours outside their zone.
(Since this is clearly ludicrous, I asked Tim if he was redefining "f" in an unspecified way, thus changing the whole equation, in a manner unknown to us. Tim replied that he was indeed redefining f, but he hasn't explained how anyone could take his redefinition and his assumptions and arrive at the value of 2/16.)
And does anyone take seriously the claim of the Lancet authors that the value for n is zero?
One other thing. Given the amount of effort that Tim Lambert has put into attempts to discredit the msb authors, it really is sad that he failed to mention, above, that the Journal of Peace Research didn't just publish the msb paper, but awarded it the best article of the year.
So, the research is not only peer-reviewed, like that other peer-reviewed study (the one we're supposed to elevate to Holy Writ status on account of its being reviewed by peers) - it's also prize-winning.
(Yes, I'd already mentioned that it received the award, but way, way down in some other thread, where nobody except hardcore "science" and epidemiology geeks read).
i don t have access to the paper or the older version at the moment. (anyone got a link that is still working?)
but Robert Shone gave this explanation of the number in another topic:
Moving on, what "wild assumptions" underlie "f=15/16"? The MSB team make the assumption that women, children and the elderly stay close to home, whilst allowing for two working-age males per average household of eight, with each spending six hours per 24-hour day outside their own zone. This yields f=6/8+(2/8x18/24)=15/16. Any "wild assumptions" here?
http://tinyurl.com/cczz4k
let me see: the number is huge, because they use the assumption that women stay at home all the time?
this isn t just wild, it is outright moronic!
we know already, that the majority of violent deaths in Iraq are young male. but those spend a significant time of the day OUTSIDE their "mainstreet bias homezone"
Kudos to Tim for starting a new thread devoted to this topic. I think that this will lead to productive discussion. Actions like this demonstrate why Tim/Deltoid are great a host/location for all things Lancet/Iraq.
Robert Shone wrote:
Really? Actually, I think it was a kindness. Serious journals don't usually award "best article of the year" (except sometimes for student papers). If the Johnson et al. paper was mostly unaltered from the version we've seen before, that they judged it "best" makes me wonder about their other articles.
Robert claims that "Serious journals don't usually award "best article of the year" (except sometimes for student papers)."
False. The Journal of Financial Economics is one of the premier journals in economics. It is currently soliciting votes for the 2008 Paper of the Year.
The reason that the award is important is because Tim (and other critics) can't just claim that a flawed paper got by one or two incompetent reviewers. This is clearly a paper that the editors of the journal are ready to stand behind. Which of those editors would you like to accuse of incompetence first?
Just to be clear on what the disagreement is about, here is the abstract for the paper.
Would Tim or Robert or anyone else take issue with these claims? (You are still free to maintain your other criticisms as well. But it us hard to describe a paper whose abstract you agree with as "badly flawed," whatever issues you might have with the details.
And, by the way, this paper (working draft published well in advance of IFHS) does a great job of predicting that a better survey without main street bias (i.e., IFHS) would estimate only a small percentage of violent deaths. Too bad we didn't bet on the IFHS outcome before we saw their estimate. Spagat et al (and I) would have won that bet.
like all papers that produce an underestimate will agree on low numbers, David?
the IFHS paper does not show an increase in violence after the samarra bombing. yes, that is the incident that caused the INCREASE of violence, that caused the "surge" of US troops.
the paper doesn t show a change. you don t think that is a problem?
you also simply ignore the massive increase in excesss non-violent deaths. no interest in looking more deeply into that?
sod, you can download a draft version from Mike Spagatâs page.
David Kane quotes the abstract: "...the authors develop a mathematical model to gauge the size of the bias and use the existing evidence to propose values for the parameters that underlie the model."
As Tim points out they didn't use the existing evidence; he's too polite to say it, but the fact is they plucked the parameters out of their arses to get the result they were aiming for.
Intriguingly, this has been spun for the physics audience as well, in Europhysics Letters EPL:
'Sampling Bias in Systems with Structural Heterogeneity and Limited Internal Diffusion', JP Onella et al, EPL (85) 2009, 28001
David Kane: The Journal of Financial Economics is one of the premier journals in economics.
Not that it matters but thatâs untrue, unless by âone of the premier journalsâ you mean itâs in the top 50. Keele ranks it 45th in fact â which is highly respectable but certainly not stellar. And as dsquared remarked in an earlier thread, the fact that you consider the AER more reliable than The Lancet is in itself enough to cast doubt on your sanity.
But frankly David, you would be better off forgetting all this pecking-order shit. It clouds your thinking. If the MSB paper was published on Red State you would do a better job of judging it on its merits.
Itâs odd, too, that you would think of financial economics as a field which Robert might be expected to regard as serious. I would think financial economists currently enjoy about as much esteem in the scientific community as astrologers. Clicking on your link I was amused to see that one of the JFE prizes is named in honour of Eugene Fama of all people.
Kevin, in the original draft of some 3 years ago they plump for 15/16, whereas in the published version this is 13/14. The reason is that they decide there are 2 working age males per 8 person household originally, but 2 per 7 in the published paper.
Why? I don't know.
David Kane wrote:
Thanks for the heads-up! I'll move it into the category of "not a serious journal."
Kevin,
I don't want to hassle you too much. You are smart guy and highly knowledgeable about the Lancet. But, please! Stuff like this --- "Keele ranks it 45th in fact" --- does you little good. Did you notice how that spreadsheet you linked to has things listed alphabetically (sort of) within the 4 Keele ranks? The 45th row does not mean what you think it means.
And, if you demand a journal ranked in the top category by Keele, check out the prizes award by the Journal of Finance.
Anyone who doesn't agree that the Journal of Financial Economics (and the Journal of Finance) are premier journals in economics/finance does not know what they are talking about.
You write: "the fact is they plucked the parameters out of their arses to get the result they were aiming for." Well, it doesn't really matter where the estimates come from, does it? They argue that the estimates are reasonable. If you disagree, be specific as to why. And then they show the answer for a wide range of parameter estimates. Do you disagree with that calculation?
Again, I think it would be great to focus on what the paper actual says. Quote a portion that you disagree with and explain why you disagree. That is how we are going to make progress.
If Robert does not think that there are any serious journals in finance, then he should make that case. He strikes as more serious than that.
Tim writes:
Tim: I have asked you this question before and you have refused to answer it. I will try again.
Were all the houses in Iraq (main streets, side streets, back alleys) included in the sampling frame?
Once you explain to us what you think Lafta did, then we can explain why you are wrong. 10 is a perfectly reasonable estimate for n.
Tim,
Perhaps I am misreading things, but your critique of f demonstrates a fundamental confusion. You write:
Correct. The key here is that "sampled area" means, not just the home, but the "survey space", i.e., all those areas that are within the area that the interviewers might have visited. So, for example, the park or market across the street from your house is a a part of the "sampled area." f is then the proportion of time that a random Iraqi spends in that sampled area. Women, children and the elderly (during these times of extreme violence) obviously spend the vast majority of their time in the sampled area. Many men did as well. But, of course, many men spent hours outside of the sampled area, mainly at work. What is f? Well, reasonable people might differ and the authors make their pitch. You then write:
This is wrong on several dimensions. First, you have no idea how many deaths (whether of men or otherwise) were "outside the home." The location of the death was never released (or collected?) by the Lancet team. Second, even if you magically knew what percentage of deaths were outside the home, that tells you little about f. If someone is killed at the neighborhood park, but they killed outside the home but still within the "sampled area". Third, even if you knew how many deaths were inside and outside the sample area, that knowledge has no direct bearing on f, which is just the amount of time spent in the sampled area.
This is just gibberish. Again, some markets/shops/cafes/etc are in the sampled region. Also, you don't think that people sleep in their houses? Assuming that people sleep for 8 hours (or sleep for 6 hours and spend two hours dressing/cleaning/eating/whatever), then the minimum value for f (both fi and fo) is more than 5/16. I suspect that you are just confused about the definition of fo.
Assuming of course that women, children and men of above working age never stray into the surveyable area.
Oops no. Apologies David. I'm confusing the original assumption with your minimum, which assumes everyone leaves the area in which they reside. Of course this minimum is comfortably below the 8/16 level to give no effect of place of residence on violence exposure. Bed time me thinks.
It's probably safe to assume that almost all the Iraq denialists around here are also climate change denialists and some are also evolution denialists. Some of them would probably deny that Dick Cheney throws like a girl and Rush Limbaugh is a fat fool. "What are you in denial about Johnny?" - "What have you got?!"
Any of them (it may be all of them) who think that the attribution of cause of death as reported to interviewers is the thing, rather than the fact of a family member being now undeniably dead, is delusional as well.
Is delusion worse than denialism? Denialism is more morally bankrupt and shameful but at least it's curable.
David Kane wrote:
1. Is David Kane claiming that every journal in finance names a "best article" each year? Excellent point! Not every journal does so!
2. I try to be only as serious as I need to be. With you I haven't needed to be that serious.
let me repeat what Tim said:
fi, the probability that someone who lived in the sampled area is in the sampled area and fo the probability that someone who lived outside the sampled area is outside the sampled area. They guess that both of these numbers are 15/16. This too is ridiculous. The great majority of the deaths were of males, so it's clear that the great majority were outside the home. So the relevant probabilities for f are for the times when folks are outside the home. And when they are outside the home, people from both the unsampled area and the sampled area will be on the main streets because that is where the shops, markets, cafes and restaurants are. Hence a reasonable estimate for fo is not 15/16 but 2/16. If use this number along with their other three numbers (including their ridiculous estimate for n) you get a bias of just 5%.
and here again, the easy way to test main street bias:
walk down a "main street" (busy shopping roads will do..) and ask every person you meet, whether they live in a street intersecting this one or not.
Upthread, Jody Aberdein asks me why Johnson et al went from assuming 2 working age males per 8 person household originally, to 2 per 7 in the published paper. I suspect they just picked up a scrap of information on Iraqi demographics from somewhere and decided to use it (7 is more realistic than 8 if memory serves). Maybe they find that extracting too many model parameters from the rectum causes haemorrhoids.
fi, the probability that someone who lived in the sampled area is in the sampled area and fo the probability that someone who lived outside the sampled area is outside the sampled area. They guess that both of these numbers are 15/16. This too is ridiculous.
this is the biggest problem with the assumptions, and the silence of the lancet attackers on it is deafening.
they really believe that:
1. the majority of people killed by violence in Iraq are NOT young male.
2. that stupid iraqis keep their families in the deadly houses along the mainstreet, while their males spend the days i a safer zone
3. on the other had people living in saver zones send their males into the dangerous zone for work..
according to this paper, your average market bombing would kill over 90% of people who live in an adjanced street!
David Kane: Did you notice how that spreadsheet you linked to has things listed alphabetically (sort of) within the 4 Keele ranks? The 45th row does not mean what you think it means.
Ye gods. David, what is it with you and spreadsheets? First you can't calculate a crude mortality rate even when Les Roberts helpfully inserts the formula into a spreadsheet for you. Now you can't figure out how the rows in a spreadsheet are sorted.
If by "listed alphabetically (sort of)" you mean, listed alphabetically within groups made up of journals with equal scores, you are correct. But it isn't only the four Keele ranks which are taken into account. For example, the reason why the Scandinavian Journal of Economics is listed above the American Journal of Agricultural Economics (despite their having the same Keele rank) is not because the guys at Keele suffer from intermittent dyslexia. The former has a higher KMS score than the latter. So while the Journal of Financial Economics could climb a few places by changing its name to Aardvark Studies, it needs to do a bit more than that to break into the top 30.
But as I said above, all this talk of reputation is beside the point. I will try to avoid letting you drag me into it again. Papers should be judged on their merits. For the reasons Tim and others have pointed to, the MSB paper would deserve harsh criticism no matter who got suckered into publishing it.
Kevin,
By all means, let us get back to judging the merits of the paper. Here is a summary of where we are. Tim claims that the paper is badly flawed and cites exactly three parameters. That is the extent of his critique. One of those parameters, q, he agrees with Johnson et al. (I have been guilty of referring to the authors as Spagat et al, but, of course, Johnson is the lead author.)
So, the entire criticism (from Tim at least) boils down to two parameter values: n and f (where f includes both fo and fi). I have explained above why Tim misunderstands what f is. Do you have a reply? What do you think f should be? I could imagine making a case for some smaller numbers, depending on how big you think the sampled areas are. But it seems obvious that the vast majority of Iraqis spent the vast majority of their time in the sampled areas (both their houses and the local neighborhood) during this period.
Finally, we have n. Now, I agree with any critic who claims that it is very hard to know what n is. But that is the fault of the Lancet authors. We do not even know if every house was included in the sample frame! Once you tell me what you think the actual procedure was (see our discussions in other threads), then I am ready to debate whether or not n is 0.5 or 1 or 5 or 10 or 20.
Again, thanks to Tim for creating this thread. I think we are making progress!
Well,
As has previously been pointed out you could go a long way to demonstrating why or not n was important by doing some kind of actual Monte Carlo analysis using the study protocol as described and some real maps, or even just grids. That would be preferable to 'here are some coloured in google maps for you to look at'. No?
Likewise you could prior to publication attempt some kind of empirical justification for your values of f.
Likewise you could attempt some empirical justification of the heightened danger of 'main streets' by actually looking at distribution of explosions/shootings.
You could also publish a sensitivity analysis that included values of n that include the possibility that your hypothesis is incorrect i.e. there is no excess danger distributed in the fashion you describe, or in the bias introduced by the sampling schema.
Certainly in the UK the Office of National Statistics can provde population data down to blocks of 1000 individuals and maps of these regions for street level correlation. Presumably this is beyond the ken of social science research?
Otherwise it seems to me that the haemarrhoid argument stands.
Jody suggests: "using the study protocol as described." The problem, of course, is that the Lancet authors have said, in various venues, that the protocal was mis-described in the published version of the paper. The Lancet has not published a correction. The authors have made various claims about the actual protocol used (both in public and in private correspondence), much of which is inconsistent with the published statement as well as being inconsistent with each other's statements. So, there is no way to do the Monte Carlo you recommend.
And here is where Tim could help!
Tim: Could you ask Les Roberts for a precise description of the study protocol? What, exactly, did the interviewers do?
Knowing that would allow us to flesh out Jody's (reasonable) suggestion in much more detail.
DK:
>One of those parameters, q, he agrees with Johnson et al.
On reflection, I have changed my mind about this. q = 5 is reasonable for main streets, but not for streets that merely intersect main streets.
DK: I don't agree with the abstract. Under any reasonable assumptions MSB does not make much difference.
DK: Yes I redefined fo, to something more reasonable. The only times if including in it are thois spent outside the home. But I'm thinking that their formula is so badly flawed (because it assumes that the distribution of deaths wrt to time of day is uniform) that it might be better to dump it and use an accurate one.
Hell, I typed up this long comment saying I disagree with Tim about q and now I see heâs changed his mind. Never mind, Iâm posting it anyway.
Itâs true that Tim gives the MSB paper a pass so far as q is concerned. I think thatâs because of the way he is framing his argument. He wants to highlight the completely unreasonable aspects of the paper, so he passes over the bits which are not too obviously wrong. So he can throw a bone to the authors just to show how impoverished they are. But if we are to vet the paper properly we canât do that.
That being so I donât think q=5 should be accepted without question. The unsampled region will include places which are simply too dangerous for interviewers to visit as well as places which happen to be too far from the nearest cross-street. So q<1 is entirely possible. A careful reader of the paper will see that of course, but the discussion (on pages 8 and 9 of the draft I found on Mike Spagatâs web-page) certainly doesnât do much to draw attention to it. The sensitivity analysis is still worse in that respect â q is restricted to positive integers in the tables. Figure 3 is misleading, being scaled to confine the region q<1 to a narrow vertical strip. (Itâs a classic example of how to fool a reader who is in a hurry.) The fact that dead interviewers report no results is considered when the authors defend cluster sampling as being a relatively safe method. But even there I see no reference to the fact that interviewer prudence may result in q<1.
Also, if the unsampled region is very large, as the MSB theory claims, then the vast majority of those who die in bed simply cannot be counted. But in any population, even in a war-torn country like Iraq, most people die at home in bed. Of course most such deaths are non-violent but itâs quite possible that if hospital resources are overstretched even victims of violence perpetrated in the survey space end up dying at home (most likely outside the survey space if n=10). Remember, itâs where they actually die that is relevant for calculating q and not where they were when the violence took place.
Hence I conclude that Tim is too indulgent on this score. But of course he is right to focus mainly on the other parameters â thatâs where the MSB case really falls apart.
David, since you asked Tim to open this thread on the MSB paper, wouldn't it be nice if we could stick to discussing the MSB paper and not your suggestions for pestering the Lancet authors? Why should Burnham et al help Johnson et al to determine the parameters of their wretched model? Can't they do some research of their own, or are they merely parasites?
As I've pointed out before, one can obtain precise information about the victims of other conflicts, e.g. Northern Ireland. Using that data it would be quite possible to study the biases inherent in a variety of different sampling methods.
Yeah you're right.
'The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed.'
I cannot see any way in which this description could be simulated. Absolutely none.
Certainly even if an analysis of bias this method generated were made, how possibly could it add weight to scientific arguments such as 'Analysis of Iraqi maps suggest n=10 is plausible' followed by a link to some coloured in maps inviting the reader to consider them. Pretty watertight I should say.
RE: Comment #9 - Best Paper of the Year
"Serious journals don't usually award "best article of the year" (except sometimes for student papers)."
The Lancet publishes the best paper of the year:
http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(09)60081…
That's hilarious.
... Deltoid should run a "best debunk of the year" award. Lambert gets my vote every time!
.. Deltoid should run a "best debunk of the year" award. Lambert gets my vote every time!
mine as well. for example how he debunked this article is perfect.
the Lancet deniers look pretty helpless in this topic. basically you are avoiding all points being made and focus on magazin awards...
Oscar wrote:
Yeah, and that's regrettable. It's a bad trend and a reaction, I suspect, to the burgeoning weight given to citation indexes as an indicator of journal quality -- "articles of the year" are intended to raise the number of citation links. As an aside, the Lancet (and JFE) are popularity contests: readers vote on the article while the Journal of Peace Research award was decided by a panel. I'm not sure which is worse -- I suppose it depends on whether you prefer the judgments of Simon, Paula, Kara, and Randy over the votes of the American public; or maybe JPR just doesn't have enough readers to do a popularity contest. BTW, the JPR panel don't appear to have had much training in biostat, epidemiology, or demography. If they had, they may have understood the problem the Johnson and Spagat overlooked.
Bleg: page 3 of the draft I'm looking at refers to "Little, 1982" but there's no Little, 1982 in the references (see page 23). Does the published version shed any light on this? (I refuse to shell out 20 dollars on the published article unless I receive credible assurances that it's a huge improvement on the draft.)
The relevance is that they imply they are following Little's "modelling approach" and it would be nice to know if somebody named Little really advocated the approach they are following, and if so, what justification s/he offered for it. They offer none themselves obviously, other than the usual "Gilbert would't give Mike his data! Waaah!" - the pathetic refrain running through the entire paper.
Tim? Robert? David? anyone?
Robert: BTW, the JPR panel don't appear to have had much training in biostat, epidemiology, or demography. If they had, they may have understood the problem the Johnson and Spagat overlooked.
Well I've no training in those areas either but let me guess. One big problem which jumps out at me is that to implement their approach to bias adjustment, you need to know the risk of death in the unsampled region. If you knew that, why in the name of all that's holy would you bother doing a mortality study in the first place? Just unsample the whole country and there's your answer!
I mean come on, don't tell me that's not a major conceptual problem with the MSB approach? I'm not for a moment suggesting it's the only one, but even ignoring the other turds in this crock of shit, that one has to offend any thinking person's nostrils.
But Ireland beat France so not even David Kane can piss me off tonight.
Just a side note on exactly how this paper won the article of the year award.
Read the whole thing.
The problem this raise for Tim is that he needs to explain, not only why Johnson et al are idiots, and why the editors of the journal are morons (both hard to do) but also how Cederman, Hovi and Mitchell could have screwed up so badly. What's the theory? That they are paid in members of the neo-con conspiracy?
Now, obviously, just because a paper is published in a peer reviewed journal, and just because it wins an award judged by three academic unconnected to the journal, does not mean that the paper is perfect, that there are not reasonable grounds for criticism and so on. But the burden of proof is clearly on Tim (and other critics) to demonstrate exactly why this paper is so "badly flawed."
By the unsampled region I mean the population outside the survey space. Sorry to be incoherent but this isn't just your average Saturday night. Don't know why I'm bothering with this crap to be honest.
David, this thread, which you requested, is about the paper. If you want to talk about the sociology of science and suchlike, that's a whole different topic. Can we have your response to the criticisms upthread, if you have one? Do you know where this "Little, 1982" paper may be found?
Robert claims (with no evidence): "BTW, the JPR panel don't appear to have had much training in biostat, epidemiology, or demography." Charming! How were you able to access their graduate school transcripts?
Kevin: I bet that Little 82 refers to: Little, R.J.A. (1982). Models for nonresponse in sample surveys. Journal of the American Statistical Association, 77, 237-250.
Tim writes: "DK: I don't agree with the abstract." Well, that is nice but, if we are going to make progress, then we are need to go into more detail. Do you disagree with the first sentence? The second sentence? And so on. An academic paper is made up of a collection of discrete claims. Which claims made in the abstract do you dispute>
Tim writes:
Well, by all means, feel free to come up with a better model and write a paper. But, for today, we are focusing on whether or not this paper is "badly flawed." If you continue to change the definitions of various parameters without telling us, it will be hard for us to follow your argument.
So, using their terminology, what do you think a fair value for f would be?
But the burden of proof is clearly on Tim (and other critics) to demonstrate exactly why this paper is so "badly flawed."
Tim and others gave the reasons. the numbers they use are a joke.
but neither you nor Robert Shone has even tried to defend them so far...
i like IPRI and the work they do. they just messed this one up. could we now get back to a discussion of substance?
...not even David Kane can piss me off tonight.
I underestimated you David. Sorry about that.
So, using their terminology, what do you think a fair value for f would be?
their "terminology" is utterly useless. obviously it doesn t matter, how much time children and women spend in the main street zone. they don t get killed in high numbers anyway.
david, why don t you give us an explanation for the high number of male Iraqis getting killed, when it is WOMEN, who spent all their time in the dangerous zone...
> A simpler explanation that addresses both the increase in violent deaths and the balancing decrease in non-violent deaths without resorting to a putative MSB is that there was a problem with attribution of cause of death.
Comparison of the "violent deaths" counts in Lancet2 and IFHS is complicated (if not rendered impossible) because of several factors that have to do with cause-of-death attribution and classification of "violent" vs. "non-violent".
One simple example of such a factor is the existence of a large category of "deaths of unknown reason" in IFHS, which are all classified as "non-violent".
http://probonostats.wordpress.com/2008/01/27/ifhs-violent-deaths
Of course, the cause-of-death issues are just a subset of the many problems, some of them severe, with the IFHS. See: http://probonostats.wordpress.com/2008/01/17/5-problems-with-the-scienc…
BTW, is there any indication of what was the sampling methodology of the IFHS? Without an enumeration of all Iraqi households, some sort of geographical sampling seems like the only way to go. Did the IFHS possess an enumeration of households?
I wrote:
>I don't agree with the abstract. Under any reasonable assumptions MSB does not make much difference.
David quotes the first sentence and then asks me what I disagree with. The answer is in the second sentence.
One simple example of such a factor is the existence of a large category of "deaths of unknown reason" in IFHS, which are all classified as "non-violent". http://probonostats.wordpress.com/2008/01/27/ifhs-violent-deaths
very good article.
i wonder why the "sceptics" have so far done little analysis of the interests of the iraqi ministry of health. they have ALWAYS tried to downplay the number of violent deaths, and have often been forced to admit that.
why do you think they didn t do the same in this study?
To see that their formula is wrong, it is sufficient to provide an example where it gives the wrong answer.
So let's consider a case where there are just two people, person A, who lives in the sampled area, and person B who lives in the unsampled area. The only risk of death comes from terrorist attacks on the local market. Both A and B spend one hour a day at the market, and the rest of the time at home.
By construction, A and B have the same risk of death, so no bias is introduced by just sampling A.
What does the MSB formula say? Well, in this case n = 1 (A and B are equal in population), fi = 1 (A never leaves sampled area), fo = 23/24 (B only leaves unsampled area for one hour per day), and q = infinity (all deaths occur in sampled area). Plug those numbers into formula and it tells you that R = 1.92, i.e just sampling A gives an estimate almost two times as high. If instead B is 12 people, n = 12 and R = 12.5 i.e the formula says the bias is a factor of 12.5, even though there is no bias.
It is easy to construct examples where the formula is wrong by an arbitrary amount. It is hard to construct plausible examples where the formula gives the right answer.
(For the purists out there: I didn't actually put q = infinity into the formula, but took the limit as q approached infinity.)
Tim Lambert writes:
Of course, the problem with Tim's "construction" is that A and B artificially, by definition have an equal chance of being killed, regardless of where they live.
It's a bit like saying that if you "construct" a case in which force doesn't equal mass times acceleration, then you can show that Newton's second law of motion is wrong.
I knew Tim misunderstood MSB in some ways, but I didn't realise the depths to which his ignorance on the matter plummets.
Robert Shone,
You donât have to work with Timâs construction. Try one of your own. Hereâs what I did: assume 9.5m people live inside the survey space where the probability of death is 0.0055 and they spend 45 percent of their time there. 10.5m live outside the survey space where the probability of death is 0.005 and they spend 60 percent of their time there. Their movements offset the risks of location to some extent but not completely, so the survey produces a slight bias which you can easily calculate. The MSB formula gives the wrong answer here too. It overstates the bias by quite a bit.
Kevin writes:
Nobody will be "working" with Tim's embarrassingly inept "construction". But I will be quoting it whenever he tries to pass himself off as some kind of expert on the subject.
Umm, Robert, F = ma is a physical law, so an example where it isn't violates the laws of physics. In my example, two people have the same risk of death. What physical law do you think this violates?
Not good enough, Tim. Your inability to see the problem in your "construction" shows more, perhaps, than simple ignorance over MSB.
Regarding 'not good enough':
On one hand:
We have some inductive reasoning, some good empirical evidence that give a range for the number of deaths attributable to the invasion of Iraq.
On the other hand:
We have some deductive reasoning with absolutely no empirical evidence, made by a group of people whose other arguments include accusation of scientific fraud, argument by incredulity, and cherry picking data to try to smear the empiricists above.
Which dear reader would you deem to be 'not good enough'?
Not good enough, Tim. Your inability to see the problem in your "construction" shows more, perhaps, than simple ignorance over MSB.
sorry Rober S., but your silence on the SUBSTANCE of this subject, tells me a lot about you.
here is what their formula says: (some simplification made by me):
a 8 person household INSIDE the mainstreet bias zone spends his time like this: 6 female/kids/elderly (i ll sum these up as female from now on) spend all their time in the zone, while 2 male spend about half (my simplification, doesn t really change the outcome) their time outside. this gives a 6 to 1 ratio between the two groups.
a 8 person household OUTSIDE the mainstreet bias zone has 6 female(etc) always OUTSIDE, and 2 male who spend half their time inside the mainstreet zone. (a moronic assumption, but we go along with the paper. slight simplification.)
this gives a ratio of female/male INSIDE the zone of 6:2 and OUTSIDE 6:2. (funny, their weird construct just allows men to swap places..)
this would give a ratio of adult men to rest of family being killed of 3:1.
but all available data shows, that male have a much higher risk of being killed! (about 2 times that of women of their own age, MORE if looking at violent deaths..)
[IFHS report table 27](http://www.emro.who.int/iraq/pdf/ifhs_report_en.pdf)
it is obvious, that where you live is NOT the most important factor, deciding whether you get killed or not!
sod writes:
Well, I've already written quite a bit about what you call the "substance" (ie parameter value examples and the assumptions behind them).
In fact you quoted, above, one of my previous long, detailed posts on it (and then you misrepresented both myself and the MSB authors by saying: "let me see: the number is huge, because they use the assumption that women stay at home all the time?" Not "at home", but "close to home", ie in the district in which they live - an important distinction.
Look, you can suggest different parameter values based on various assumptions. The assumptions you have about who spends how much time where, etc, may be plausible or not. You can always change them with better data. I think the MSB team kept it fairly simple - ie didn't make too many assumptions (although you may disagree with the ones they made). I think they made plausible assumptions (certainly much more plausible than Tim Lambert's), but you obviously disagree. That doesn't make the MSB research wrong - it make the assumptions behind example parameter values debatable.
The bottom line for me is what various researchers (eg Jon Pedersen, etc) have said. Which is that the MSB work represents important progress the field of estimating violent deaths in conflicts. Others frame is as being yet another hostile force against Lancet. So be it.
sod: You have interesting comments to make about IFHS. I suggest that you ask Tim to start a thread on that topic. For now, this thread is about Johnson et al 2008.
Tim writes: "Under any reasonable assumptions MSB does not make much difference." Well, it seems a big step from this to "badly flawed." I think it is helpful to break up Johnson et al into two parts: the model that they use and the parameter values they estimate for Iraq. If there is something wrong with the model (the math is wrong, the code behind their sensitivity analysis mistaken), then that would be a real problem. But, as far as I can see (I am still trying to process Tim's #55), no objections have been made to the math or the computer code.
The second part is the parameter values. Here, we are making some progress, but not much. (Tim's redefining of certain terms does not help matters.) So, let's revisit this. What specific parameter values do you find to be reasonable and what is your evidence for them?
Now, at this point, you might try to switch the burden of proof and say, "Wait! It is the job of Johnson et al to come up with, and defend, their choices." Let's go back to: "n, the size of the unsampled population over the size of the sampled population." I think (I have not checked this with any of the authors) that Johnson et al would say:
If the Lancet authors would tell us what the sampling plan was, then we might be able to pin down the value of n, but until they do, I think anything from 0 to 20 is reasonable.
In other words, it is not enough to say that n = 10 is wrong. You need to present evidence for what n is. Since you don't know what the sampling plan was, you have no evidence.
Now, if Johnson et al claimed that 10 was the only reasonable number, that numbers like 5 or 20 were impossible, then you would have a point. But they don't. They admit that n could be in a broad range and they provide a useful sensitivity analysis.
My comment No. 57 is hereby retracted. I shouldnât post on the morning after such a night before. I think Tim makes a very telling point in No. 55 and I will be interested in whether the MSB fans can do anything better than sniff.
David Kane: I bet that Little 82 refers to: Little, R.J.A. (1982). Models for nonresponse in sample surveys. Journal of the American Statistical Association, 77, 237-250.
Thanks David. Itâs here in case anyone wants to buy it. Nothing in it suggests to me that one can obtain useful estimates of bias by the Gourley method (colouring in street maps). So it looks as if Johnson et al merely referenced it just to give a scientific veneer to their paper. Alan Sugar had a good name for that particular marketing trick: the mugâs eyeful.
If you could actually demonstrate this physically, then yes, you would have shown that Newton's second law of motion is wrong. Actually, we know it's wrong, but it's a very precise approximation at relatively low velocity, therefore it's usefully wrong.
How does this disprove Tim's example showing that the formula in question is wrong?
Shone, Kane:
We're getting bogged down in details, and losing sight of the issue.
Bottom line - addressing the now multiple sources of estimates of Iraqi mortality, what central estimate and range do YOU accept for Iraqi excess deaths due to the war?
dhogaza writes:
Well, my point was not that you can't demonstrate that a given "law" (Newton's or any other) is wrong, but that you wouldn't do so credibly by means of a hypothetical construct in which you define in advance that it's wrong.
Kevin Donoghue
Do you think his "construction" demonstrates (as he claims) that the "[MSB] formula is wrong". Or is it some other telling point that he's making?
The research programme of the MSB team reminds me of the Underpants Gnomesâ business plan:
Phase 1: Get Burnhamâs data
Phase 2: ?
Phase 3: Publish estimates!
Can any of their defenders tell me what is supposed to happen in Phase 2? I donât want vague generalities. I want to know, for example, how it is proposed to estimate the risk of death in the area outside the survey space. Without that the MSB equation cannot generate a number. Assume Burnhamâs data includes anything within reason â maps, dates, family composition â even things he has never pretended to know. But you cannot assume things which are untrue by definition; for example, that he has information from households outside the survey space.
David Kane? Robert Shone? Prize-winning authors? (Tim hasnât banned you, so you are entitled to join the conversation.)
RS: You can demonstrate that they are inconsistent with their assumptions by a constructed example, which is what Tim has done
Kevin: Hospitals were killing zones if you were of the wrong sect, so there were excellent reasons why some casualties never were taken there, or left as soon as possible, if possible
TimL: Does this sort of stupidity remind you of the hockey stick issue, where unrealistic parameterization was used to try and falsify a basically robust result
Le: The last estimate I offered on this question (at least for violent deaths) was 100,000 (0 -- 300,000) as of July 2007. Glad you asked!
Kevin asks "Can any of their defenders tell me what is supposed to happen in Phase 2?" Well, I could offer some guesses, but that is not the purpose of this thread. Our purpose is to discuss the quality of Johnson et al 2008? You are curious about how Johnson et al 2008 would have been different had the Lancet authors shared the data with them. An interesting topic! But not for today. If you have specific parameter values that you think more appropriate (and evidence to back them up), then make your case.
Eli: I agree that the Lancet debate and the hockey stick controversy are similar, but perhaps for different reasons that you do! ;-) Also, do you have any substantive comments to make on the paper?
David, Robert S., why do you both simply ignore the problem with gender distribution among the killed?
this does completely contradict their version of "mainstreet bias"!!!
People can be skeptical, people can argue, people can disagree, people can make honest mistakes - that is not what is happening here.
Kane has shown clearly over the years in which he has been trying to promote himself using Iraq mortality studies as a vehicle that he is unable produce any substantive arguments.
Shone's inability to engage substantially the issues regarding the bias model shows that he as well is not writing with the objective of finding out the truth, but with other objectives - personal gain, political commitment, etc.
There is therefore really no way to engage these people in an honest debate. Responses to their comments can only result in eliciting more manipulative, dishonest comments.
By your logic, if I state that "n + m = 5", and Tim states "this is easily proven false by defining n = 1 and m = 1", then Tim hasn't falsified my equation because he's given a construct in which he's defined in advance that my equation is wrong.
Odd logic.
There's nothing physically implausible about Tim's hypothetical case. If the given equation fails under certain cases, then the constraints which must be met for it to be accurate must be stated along with the equation. And when applied to a particular problem, such as Lancet 2, one must show that the data being analyzed meets the stated constraints before one can state that the equation shows that something's rotten in Denmark.
No exceptions.
David Kane: You are curious about how Johnson et al 2008 would have been different had the Lancet authors shared the data with them.
No, that wasn't my question, though of course you keep going back to that in order to avoid addressing criticisms of the paper. Now, before you ask what criticisms I refer to, try reading the thread. Let's recall that you requested this thread. How many times now have we seen you return to your complaints about Burnham and how he won't feed these poor pitiful wretches with data?
If you have specific parameter values that you think more appropriate (and evidence to back them up), then make your case.
You want me to pluck parameter estimates out of my arse like Johnson et al? Sorry, David, I don't go in for that sort of thing. To me, statistical inference isn't about colouring maps downloaded fromm Google Earth. It's mostly about estimating model parameters, so first we need a model which is constructed in such a way that it can be estimated.
The MSB model alas, cannot be estimated, even in a world where Gilbert Burnham posts his data on the web for all to download. I take it you agree, which is why you keep ducking the issue.
Robert Shone: Do you think [Tim's] "construction" demonstrates (as he claims) that the "[MSB] formula is wrong".
AFAICT the algebra deriving the formula itself is correct so in that sense itâs right. I presume Tim accepts that, though if he says thereâs a howler in there Iâll certainly look again; heâs no slouch when it comes to maths. I take it Timâs claim is that the model is logically incoherent; not just useless for any practical purpose as Iâve been explaining to David. Iim's comment is not (yet) a proof of that, where âproofâ is defined as convincing to me. But donât read too much into that. Iâve often seen proofs containing the word âobviouslyâ where the author meant: this will be obvious when youâve thought about it long enough. And indeed it was, eventually. For now though Iâm only prepared to say that the MSB paper is worthless, not that it is also nonsense in the strictest sense of the word.
Eli,
Those stories about Iraqi hospitals were indeed on my mind when I wrote my earlier comment about the possibility that q<1. The MSB paper has many worse flaws, but the fact that the authors didnât see the need to consider that possibility, even if only to argue against it, reflects badly on them. Good papers are more judicious than that â the Lancet papers contain so many caveats about possible biases that they provide the critics with the best ammunition they have.
Kevin: Your tone is hardly helping this conversation. Anyway, you write:
No, I just view this as (almost) too obvious to discuss, but I am all about education, so here goes:
1) Seppo Laaksonen wanted to know the average number of main streets in each cluster. Burnham refused to tell him. (This isn't just about the L2 authors refusal to share data with Johnson et al.) If Laaksonen had that data, he might be able to come up with a rough sense of the actual coverage of the survey. If there are lots of "main streets" in the typical town, then n is probably fairly small. Almost every house will be near one of those main streets. If there are very few main streets, then n might be quite large since lots of houses will be nowhere near a main street (or any of its cross streets).
The more details that the Lancet authors release, not just about the actual data but about the procedures that they used, the more we are able to understand what they did. That knowledge allows us to come up with better (albeit still very rough) estimates for the Johnson et al (2008) model, or a different model.
By the way, I don't think the word "estimated" means what you think it means.
Sortition: If you don't feel like engaging in a conversation about the quality of Johnson et al (2008), then you should go away. The rest of us are busy here.
sod: With regard to #73, I confess to being confused. Gender plays no part in the model here, so it is hard to see how any gender breakdown in deaths can "completely contradict" the paper. Perhaps your point is that there use of gender in estimating things like f are inconsistent with higher male mortality? I don't really see that, but please make your case in more detail.
How could Laaksonen estimate n from the average number of main streets?
I think I'll stay away for a few days until yapping purse dog Kane finally wears out and people get tired of responding to him.
Someone plz ring me up when this happens.
Best,
D
sod: With regard to #73, I confess to being confused. Gender plays no part in the model here, so it is hard to see how any gender breakdown in deaths can "completely contradict" the paper. Perhaps your point is that there use of gender in estimating things like f are inconsistent with higher male mortality? I don't really see that, but please make your case in more detail.
Tim made the point in the original post. i wrote about it, in basically every post i wrote on this subject.
if female, kids and elderly 8who live in the mainstreet zone) spend all their time in the risky zone, while (the outnumbered) young male leave it for several hours per day, then we would expect the vast majority of death cases to be female, kids or elderly. but reality shows exactly the opposite!
Gender plays no part in the model here
it implicitly does. they assume, that the place where you live determines risk. they also assume that males leave their homezone a lot, while females don t.
when you chose a more reasonable basis assumption (one supported by FACTS) like "males have a much higher risk of dying from violence", then by using their other assumptions (males move around a lot) you automatically come to a conclusion that contradicts (their version of )the mainstreet bias theory!
In #43, Kevin offered:
Sorry to have taken so long to respond. I wanted to check out the actual published version of the Johnson paper to see whether the fundamental problem I mentioned had been addressed. It hasn't so, like Tim, [my earlier comments from when the paper was first discussed](http://scienceblogs.com/deltoid/2006/12/main_street_bias_paper.php) still stand. I see from that earlier discussion that no one really picked up on my point then (not that I blame anyone; my comments were necessarily brief and thus overly cryptic), so I'm pleased that Sortition has noticed it now and understands its ramifications.
The fundamental problem with the Johnson paper is that it is a "plausibility argument." There is no actual data analyzed so the authors have constructed a model with model parameters that they believe to be "plausible." Using these parameters, they conclude that Burnham's estimate of violent deaths was inflated. Tim, and others, have countered by pointing out that the arameters weren't plausible, and its defenders have been responding with "yes they are." And that's the back-and-forth that's been going on for a while now.
But I think the problem is that the Johnson plausibility argument is itself flawed, and the key is very much like the Holmesian "dog that didn't bark." Here's the key: the only major methodological difference between the Roberts study and the Burnham study was in the way that the starting point for each cluster was chosen. Johnson et al. have seized on this and use it to explain the difference in violent deaths by appealing to a Main Street Bias. But since the only major difference was the starting point, any plausibility argument must take into account all of the relevant differences between the Roberts and Burnham studies. The thing that everyone has been ignoring is the dog that didn't bark: it's the number of non-violent deaths.
If you go back and look you'll see that, compared to the Roberts study, the Burnham study showed many more violent deaths over the same period but a consistent estimate for the number of total deaths. Any plausibility argument must therefore explain both the increase in violent deaths, which MSB may or may not do depending on model parameters, and the decrease in non-violent deaths in exactly the right amount so the totals remain comparable, which MSB fails to address at all. MSB is a theory about only half of a problem. For any argument to be truly plausible it must fit all of the available facts, not just the ones it cherry picks.
So is there an explanation that does fit the available facts? Demographers, boring drones that they are, have spent way too much time examining "typical" patterns of error in survey data. For example, we often see that there are common patterns in how age is misreported so that there is "heaping" on certain numbers. For another example, we often see evidence of recall bias where events that happened long ago get misreported: it appears that people forget long-ago events far more than the make up non-existent events. There are lots of these examples of typical error patterns. So here's one more typical error: cause of death is often much less reliably reported than that a death occurred at all.
So the main problem with MSB isn't that the parameters are implausible (though they may be). It's that a much simpler explanation exists that covers both 1) the increase in violent deaths and the decrease in non-violent deaths and 2) is consistent with behavior we see in other mortality surveys: the attribution of cause of death as violent or non-violent between the two studies is off.
Note that this doesn't mean that it was the 2006 Burnham study that was off: it could just as easily have been that the 2004 Roberts study was off (or even that both were off but in different directions).
So, do I have any data? Nope. I'm offering a plausibility argument, just like MSB. However, unlike MSB, it's a plausibility argument that addresses both violent and non-violent deaths, not just one of them.
dhogaza writes:
That's not quite what I meant. Look, you can easily "demonstrate" that MSB is "wrong" without going through Tim's whole rigmarole (comment #55). You just have to state that by definition "people" (one or many) have the same risk of being killed regardless of whether they live in a dangerous area or a peaceful area.
Of course, you're not really "demonstrating" anything. But it might look that way to the gullible if you dress up the "no bias by definition" case in a plausible-sounding hypothetical situation.
Robert writes:
I think that is a fair summary. But don't forget that the reason that "no actual data [is] analyzed" in Johnson et al is that the Lancet authors refuse to share the data with them.
MSB is not attempting to address the "problem" of discrepancies between Roberts et al (2004) and Burnham et al (2006). I agree that this is an interesting topic and goodness knows that I have spent a lot of time on it myself. But this is not what MSB is about. You need to critique the paper they wrote, not the paper that you think they should have written.
Again, and sorry to be repetitive, but MSB has two parts: the actual model and the parameter estimates. If you think the model is wrong, prove it. Math is fun! If you think the parameter estimates are wrong, suggest some others. But the fact that the model (or the estimates) do not address the topic of the discrepancies between Roberts and Burnham is hardly relevant to Johnson et al. That is not their topic.
Perhaps. Needless to say, you had best not mention this criticism to Les Roberts. He thinks that the two studies are perfectly consistent with each other.
Robert Shone, in my example, there was no bias, but their formula said there was. This proves that the formula is wrong. I am sorry that this is too complicated for you.
Tim: On #55, is the market in the sampled area? I think it is, but just wanted to clarify.
Also, I am somewhat leery of an counter-example which requires taking limits (which doesn't invalidate your point, of course), so could you provide an example of where the formula gives the wrong answer without setting q = infinity? If it is "easy to construct examples where the formula is wrong by an arbitrary amount," then this should not be a problem for you.
Tim: On #55, is the market in the sampled area? I think it is, but just wanted to clarify.
yes. (this is pretty obvious from the values he chose for fi and fo..)
Also, I am somewhat leery of an counter-example which requires taking limits (which doesn't invalidate your point, of course), so could you provide an example of where the formula gives the wrong answer without setting q = infinity? If it is "easy to construct examples where the formula is wrong by an arbitrary amount," then this should not be a problem for you.
any q large enough will do. even q=5 will produce a bias of 1.62, q=100 already gives 1.9. i need to do some more thinking about the formula, but Tim s example seems to show, that there is a serious problem.
MSB is not attempting to address the "problem" of discrepancies between Roberts et al (2004) and Burnham et al (2006). I agree that this is an interesting topic and goodness knows that I have spent a lot of time on it myself. But this is not what MSB is about. You need to critique the paper they wrote, not the paper that you think they should have written.
David, you didn t understand what Robert said.
it is a similar problem with the male/female distribution of deaths: mainstreet bias would require a ratio of about 3 to 1 between female/kids/elderly and male population from the polled zones. but instead the study finds exactly the opposite, (young) male outnumber female (et al) by a HUGE margin!
reality contradicts their model.
Tim Lambert writes:
Well, that's because by definition, your hypothetical people have the same risk of being killed regardless of where they live. In other words, it's nonsense.
Why don't you bring your example into the messy real-world. A bomb hitting the market might start a fire or launch a piece of shrapnel which kills people sleeping in a nearby house - or whatever.
I think MSB gave the right answer. Your definition of "equal risk" was wrong.
ah i see, fi=1 (from Tim s example) doesn t make a difference between the "mainstreet zone" and the market.
the formula seems to be ok.
Having now taken the time to study Tim's example in #55, I think I see what the issue is. I don't know an easy way to put the math into a comment, but we are looking at formula 1 from page 5 of this pdf.
In Tim's example, "A and B have the same risk of death." Johnson et al write:
The tricky part (and perhaps the source of confusion) is whether Tim and the authors are using "risk of death" in the same sense. If they are, then Tim is wrong. If qi and qo are the same, then q (their ratio) is one and, as the formula makes clear, if q = 1 then R equals 1, and there is no bias, just as we would expect.
But perhaps the issue is more subtle. Tim is claiming that even though A and B have the same risk of death, then qi and qo are not the same because he is defining these terms in a different way than the authors are.
At this point, I am not sure who is right and who is wrong. But, for us to make progress, we need Tim to tell us what values for each of the terms in formula 1 he is plugging in to come up with his counter-example. He may very well be correct that there is a major flaw. But I, at least, can't follow the argument without a more explicit mapping between Tim's example and the formula in the paper.
sod: Or perhaps you could do this? You cite an example of q = 5, which means that the probabilities of death are 5 times higher for people in one area than another. But, in Tim's example, probabilities of death are equal.
Hmmm. I am obviously just commenting out loud here.
The distinction seems to be between: What are the odds of person A dying in a particular region (regardless of where he lives)? Versus: What are the odds of person A dying in the region in which he lives?
I seek clarification from anyone on this point.
A bomb hitting the market might start a fire or launch a piece of shrapnel which kills people sleeping in a nearby house - or whatever.
your shrapnel is a nice one. it will have to travel quite a bit, as the Lancet way of "mainstreet bias" actually has a very small chance of polling people on a "mainstreet".
so the shrapnel travels quite far and hits a house in a road intersecting with the "mainstreet". now it will kill male and female/elderly/kids at a 1 to 3 ratio. and when the lancet is polling them, this is the ratio of dead they would get. but they didn t!
Tim, if I understand correctly, you are getting your result by assuming rational behavior. Your two agents limit their risk by staying away from the marketplace. But the MSB assumptions rule out rational behavior. As the economists would say, q is taken to be a technological constant. If you are on Risky Street you cannot diminish your risk except by leaving.
That's a serious flaw in the model of course. It's "wrong" in the sense that it violates the principles which most social scientists (not just economists) apply to model building. But Johnson isn't a social scientist and Spagat is evidently an eccentric one. If they want to assume the agents in their model are as stupid as billiard balls, that's their prerogative I think.
Or am I missing something?
Kevin: I think you are missing something. Tim is not assuming rationality or anything else. He is just creating an example in which we know R is 1. He then plugs in the values of the various elements of the formula. He knows what these are by construction. Since R is not equal to 1 using these values, we have a contradiction. So, the formula is wrong.
Again, I am not sure that Tim is right because I am not sure if he and the authors are defining q in the same way. But Tim's approach is certainly sound and requires no assumptions of any kind.
sod writes:
Well, you're missing the point that Tim's "equal risk" exists only in some Platonic world which looks nothing like Iraq. If it's not shrapnel or fire it's something else negating his hypothetically assumed "equal risk".
Robert,
Thanks for the reply (#83). I think your theory is as good as we are going to get.
David Kane: [Tim] is just creating an example in which we know R is 1.
Yes, but he is doing it by allowing his agents to exercise a choice which the agents in the MSB model do not have. In the MSB model there is one way, and only one way, to limit your risk: leave the survey space. Tim's agents equalise their risks in a different way, which the MSB model rules out by design. Tim's agents spend the same amount of time in the riskiest part of the survey space - the marketplace. But in the MSB model the survey space does not have variations of risk within it.
I agree with you that Tim's approach is sound, or at any rate superior to the MSB approach. But he hasn't convinced me that there is any logical flaw in the MSB model. Sure, it's a crappy model but I think the logic is okay.
I am using exactly their definition of q. Where their model fails is the assumption that every location in the sampled area is equally dangerous. Since the sampled area includes main streets and streets intersecting main streets, their model assumes that main streets and streets intersecting main streets are equally dangerous, i.e. their model assumes that there is no main street bias.
Where their model fails is the assumption that every location in the sampled area is equally dangerous.
That's not a bug, that's a feature! That's how they force the result they want - the moment you step outside the survey space your life-expectancy improves. Of course as an approach to modelling it deserves all the ridicule you can heap on it.
It reminds me of a critique of capitalism I once read, where when you looked at the small print of the "model" you found fixed coefficients everywhere - only one recipe for producing any of the goods, consumers who all wanted a particular basket of commodities with no scope for substitution. When you got down to it, the guy was saying that markets don't work if we assume nobody can ever make any choices. Which is logical of course, but it doesn't stand up very well empirically.
Similarly, the MSB model is rejected by evidence that Iraqis are pretty adept at finding ways to reduce their risks. Relocation is not the only option, though judging by the number of refugees it's often the best available.
Note that this page of clarifications from the authors may be helpful to understanding the paper. Highlights:
...
Neither of these points gets directly to the issue of Tim's counter-example, but I thought that some readers might find them helpful. The link is via footnote 2 in the paper.
David Kane (or anyone else), did they address at all the much higher violent death rate for non-elderly adult males than for any other group? As Tim and others have pointed out that's a major problem for their MSB theory.
David Kane writes:
In the MSB formula R cannot be 1 unless n=0, q=1 or f=1/2. That makes sense when you think about it - the only way to avoid bias completely is an all-inclusive sample, spatially uniform violence or perfect diffusion of people among zones.
So, by that definition, Tim's definition of R=1 in his hypothetical construction is wrong. But we "know" that R=1 in Tim's example. Or do we? How do we know? Because it's defined as part of Tim's "construction". But it's a Platonic spook which exists in no real-world equivalent of Tim's artificial set-up.
i think it is pretty funny to watch you guys dance around the gender issue. if only Iraqis were as good at avoidance as you guys are....
Tim Lambert: Consider this the first draft of a reply to their paper.
Tim, I take that to mean that you are going to publish your criticisms, by way of a response in the Journal of Peace Research or something of that sort? Until now my feeling was that the MSB was so crappy it was best ignored, but the fact that a journal saw fit to publish it is a scandal. There ought to be a response.
When it comes to the sort of arguments which I think might carry weight with journal editors (as opposed to normal people) one of the worst features of the paper is the glib way the authors introduce the assumption fi=fo. In the draft I'm looking at they make no real effort to justify it. I take it the published paper is no better in that regard? But your "all deaths are in the marketplace" story highlights the fact that, in this model, there is one way and only one way that Iraqis can enhance their survival prospects. That is by spending as much time as possible in the relatively safe region outside the survey space. And how do the Iraqis avail of that possibility? According to the authors, they don't!
The fi=fo assumption, smuggled in with hardly a shred of justification, is tantamount to an assumption that Iraqis are stupid.
Now, what happens if we credit the Iraqis with a bit of intelligence? That's not such an easy question. If those who live in the back-streets have jobs on the main streets then they have little option but to accept the risks, as you note in your post. But quite a few seem to be unemployed so I wouldn't want to be dogmatic about this. It may even be the case that, on some reasonable assumptions, rational behaviour increases the bias. However the authors certainly ought to have addressed the issue.
There's another thing about the fi=fo assumption that I think should have set off alarm bells. In Appendix 2 where the "no-bias" conditions are explored, they set fi=fo just before they examine the cases where R=1 identically. By doing that they obscured the fact that fi+fo=1 is a perfectly respectable no-bias locus. It corresponds to the case where everybody, regardless of residence, spends the same amount of time in the survey space. It's far-fetched of course but it's certainly no less worthy of consideration than n=0 or q=1. In fact they do mention f=0.5, saying this is the case where everybody spends 12 hours per day in the survey space. But it's not a matter of the number of hours. Really, in a mathematical discussion, isn't it a bit odd to mention that there are 3 points in the domain of the R function such that R=1, while passing over the fact that one of those points is the sole survivor of the infinite set we just vaporised? Methinks they are trying to gloss over the fact that most urban dwellers, if employed, spend their working day on or near a main street.
Come to think of it n=0 and q=1 are infinite sets too. But as Jonah Goldberg would say, I believe my point is unaffected.
cruft: I don't know if they address that point. (I have not studied the paper and associated materials closely.) But, I don't see how it matters. It does not effect the math. It does not (at least as I can see) effect the parameter estimates. Now, of course, one can always make a model better by adding more details. Perhaps, in the next iteration, one might allow for the parameters to vary by gender or age. (See also the quote in #101.)
sod: Putting your comments in bold does not make them more persuasive.
Kevin writes:
This sort of hyperbole does not serve you well. Your claim now is that the authors, the editors and the prize committee are all idiots? That, for some reason, they can't see what is obvious to you? All these people with fancy Ph.D.'s and university appointments?
As always, I am a fan of challenging the tenured, but you need to be realistic about the number of people that have looked closely at it. They might all be wrong, but you should provide some evidence.
And, bad news, more journal editors are persuaded that this approach has merit. See:
J.-P. Onnela, N. F. Johnson, S. Gourley, G. Reinert, and M. Spagat, Sampling bias due to structural heterogeneity and limited internal diffusion pdf, submitted to Europhysics Letters 85, 28001 (2009).
Better alert the editors/referees of Europhysics Letters lest they engage in scandalous behavior as well!
By the way, your point about fi and fo is interesting. Isn't it best if we focus the discussion on these substantive points?
cruft: I don't know if they address that point. (I have not studied the paper and associated materials closely.) But, I don't see how it matters. It does not effect the math. It does not (at least as I can see) effect the parameter estimates. Now, of course, one can always make a model better by adding more details. Perhaps, in the next iteration, one might allow for the parameters to vary by gender or age. (See also the quote in #101.)
look David, you have been avoiding this point over 100 posts now. the model wouldn t be "better" or "improved" when it got gender right, but it would be NOT wrong.
as it is, this model suggests that the majority of violence victims found by Lancet should be women. many more elderly or kids. young men should be a tiny minority among those killed by violence! (and polled)
but instead young men are the overwhelming majority among those killed by violence found by the Lancet study! (and in reality, btw...)
J.-P. Onnela, N. F. Johnson, S. Gourley, G. Reinert, and M. Spagat, Sampling bias due to structural heterogeneity and limited internal diffusion pdf, submitted to Europhysics Letters 85, 28001 (2009).
nice, Spagat recycled the Baghdad bombing graph again. the one, on which he made the completely insane claim, that incidents that kill over 10 people "almost certainly cover over half of all deaths."
http://sod-iraq.blogspot.com/2008/04/spagat-and-kane.html
looking at sampling bias due to different distribution of violence makes sense. that is, why you do as many clusters as you can.
but the Spagat version of "mainstreet bias" is obviously false, and a seriously flawed attack on the Lancet study.
Somewhat amusingly, despite framing the paper for 'systems with structural heterogeneity' the one system they choose is of course the Iraq conflict. But so many other structurally heterogenous systems to choose from one would think.
In any case when it comes to parameterisation, we are referred to the learned reference 13, Johnson et al J Peace Res 2008, and thence to Spagat's coloured in google maps.
I have spent some time studying the MSB papers (i.e JPR and EPL) and the arguments here. I just wanted to add my own thoughts before anyone goes further with hypothetical analysis of cases etc. I have also studied this entire blog, and am quite shocked by the level of aggression and rudeness of some of the discussions. But, lets set that aside and turn to the math. As I explain below, there is nothing wrong with the JPR analysis:
Burnham and co-authors' analysis of the Iraq survey results (call this L2) implicitly assumes that within each survey set (i.e. cross street algorithm) there is no possible bias, i.e. R=1 in the Journal of Peace Research paper (JPR) language. In JPR language, this would indeed be the case if, for example, the samplable and non-samplable regions have either n=0, or q=1, or f=1/2 in the case that f_i=f_0. In my opinion, the point of the JPR article is to ask what happens if this assumption of implicit homogeneity does not hold. It considers arguably the simplest generalization of full homogeneity, by allowing the samplable and non-samplable regions to have different values for the model parameters. This could correspond, for example, to a case of different q values. The JPR authors' latest published paper, the EPL, generalizes this approach to allow for multiple subsystems (i.e. more than 2). Some of them may be in the samplable region, and some may not. They may even be male and female labels (i.e. gender). For this general case, the parameters now contain multi-valued indices to numerate these many possible subsystems, i.e. they are not just 'o' and 'i' hence the harder notation in the authors' EPL. This EPL formalism would allow, in principle, for a near exact calculation of R using exact details of markets etc. *provided* that the exact details of the samplable region are known. Without the details of what streets are sampled, however (or at least, what the samplable region is) any such calculation would be a waste of time. In other words, in the lack of any information about S_i details (e.g. whether markets are present or not) such a detailed calculation would also be unnecessary: Having just two subsystems is sufficient to show there is a large potential bias R. So this seems to be what the JPR proposed, and their EPL generalizes this approach for an arbitrary numbers of subsystems.
By adding in a single highly heterogeneous feature (i.e. a single market) Tim Lambert (TL) is implicitly asking for consideration of a *3* subsystem hypothetical example. There are now in principle *three* possibly distinct killing probabilities q, i.e. for the region S_i not including the market, for the market itself, and for the unsamplable region S_o. (TL assumes that the market is in the samplable region). To calculate the bias, a three-subsystem formula is required which can be obtained from the authors' EPL paper's formalism in exactly the same way as the JPR two-subsystem formula is derived. But why stop at a three-subsystem calculation? Indeed, stopping at third-order expansions is, as any mathematician knows, often unreliable. One should go further, to all higher orders: Indeed, if the JPR authors knew the actual streets sampled, then a very accurate multi-subsystem calculation could be carried out. But they do not, since L2 authors do not seem to have released it. Hence they have to stop their calculation at the first generalization level beyond L2's assumptions, i.e. the two-subsystem model of the JPR. To go further, i.e. to include specific markets, would be to add a specificity to the model which is not justified by the lack of detailed information provided by the L2 survey team. Moral of the story? L2 team should provide details of the surveyed streets! (or at least, the samplable region S_i). Failing that, the only way forward seems to be the one taken by JPR and EPL.
So what about the hypothetical TL market example? Can the two-subsystem JPR result still be used to calculate the specific bias for the TL setup, without resorting to the more general formalism of the EPL? Yes, but the person plugging in the numbers (TL?) needs to fully understand the meaning of the terms and the steps in the algebra. The key is in JPR Appendix B, where the final formula is derived. Start from the introduction: The samplable region, which is where TL wants the market situated, now has an implicit heterogeneity meaning that it cannot be represented by a single q. In other words, there is a probability of being killed q=0 for everywhere in the samplable region S_i *except* in the market which has some high value q=qm. (The following holds, by the way, irrespective of N_i and N_0, thereby addressing the concerns of a later blogger who had other suggested values). Reading through the first paragraph of JPR, for the specific example of TL, you can see that:
- The probability that during one day, for example, a randomly chosen person is resident in S_i and gets killed in S_i is q_i . f_i . N_i/(N_i+N_o) as given by the JPR. But remember from the JPR that q_i is the *averaged* killing rate in S_i, where the average is over the entire S_i subsystem (i.e. market and non-market). It would be a weighted average of qm for the market region of S_i, and 0 for the non-market region of S_i, weighted by the time spent in the market with respect to time spent in the rest of S_i. As an example, we can set f_i=1 as TL requests, just to say that the S_i resident never leaves S_i.
- The probability that during one day, for example, a randomly chosen person is resident in S_o and gets killed in S_i is q*_i . (1-f_o) . N_o/(N_i+N_o). Because of TL's desire to add a third subsystem, i.e. the market which adds heterogeneity to S_i, the killing rate probability q*_i corresponds to the killing rate in the market only. I emphasize that this 3-subsystem example should be treated with the EPL formalism -- but no worries, we can do that using the JPR equations by simply recognizing that q*_i is qm, since all the time the S_o resident spends outside S_o (i.e. all his time in S_i) is spent in the market.
- The probability that during one day, for example, a randomly chosen person is resident in S_o and gets killed in S_o is 0.
- The probability that during one day, for example, a randomly chosen person is resident in S_i and gets killed in S_o is 0.
The rest of the derivation then follows, but the expressions contain the new unknown parameter qm due to the additional heterogeneity introduced by TL. The eventual result? An expression which yields R=1 for the TL example. It is essentially just Eq. (4) of JPR, but with the appearance of an additional qm due to the additional heterogeneity of the market. No mystery, and no surprises. It all works.
Dare I offer some advice without getting abuse? I could say "Read through carefully the first paragraph of Appendix B, and allow for the additional heterogeneity that you have forced into the problem. Alternatively, just work through the general case in EPL."
But what I will also dare to say is: "Let's all ask the L2 authors to release information about the samplable areas. Ideally, houses surveyed -- if not, the name of streets surveyed -- if not, the name of streets in the samplable space S_i. Then we can all forget this endless hypothesizing, get out some more accurate parameter values, and see what the JPR authors predict for R. Maybe it is near 1? Maybe it isn't. Maybe there are aliens, maybe there aren't etc. etc. (you get the drift...).
To summarize: L2 assumes complete homogeneity within the pool of possible streets, which is a massive and unjustified assumption. In effect, they assume R=1. JPR generalizes this to allow for two-subsystem heterogeneity, showing that an R value larger than 1 can arise. EPL generalizes this to allow for n-subsystem heterogeneity. Both of these are of course also assumptions, but less so than assuming that everything is homogenous as in L2. TL wants even more heterogeneity, in his hypothetical case of a single market. However, the reality is that there is (at least) as much heterogeneity as there is number of people and streets in Baghdad. In principle this could be modeled using the EPL and JPR approach, but *only* if more information about the street sampling is known. Let's all ask them -- then maybe all this unpleasant discussion would disappear, and everyone would be freed up to do something much more productive for the world.
P.S. Gender could be incorporated by allowing the parameters in the JPR article to carry subscripts (f) and (m) for female and male. This is just the same as adding another subsystem label (see EPL). The equations would all then have double the number of unknown parameters, and be far more complex. Without additional details about the surveys, is there much point in doing this?
> I have also studied this entire blog, and am quite shocked by the level of aggression and rudeness of some of the discussions. But, lets set that aside
You know, it'll really helps the discussion if you don't preface your comment with a quick note on how you expect to get shouted down and bullied. Really, how do you expect any reasonable person to respond to this type of thing? Do you expect reasonable people to simply overlook your claims of persecution while you continue to scream them?
> Maybe there are aliens, maybe there aren't etc. etc.
If your criticism of Burnham et al. is based on the possibility of there being aliens in Iraq, then you have a huge problem.
Nick: Burnham and co-authors' analysis of the Iraq survey results (call this L2) implicitly assumes that within each survey set (i.e. cross street algorithm) there is no possible bias, i.e. R=1 in the Journal of Peace Research paper (JPR) language.
It might be an idea to read L2, Nick. Burnham et al don't "implicitly assume" that there is no bias. They discuss possible sources of bias at some length. In MSB language they consider both R<1 and R>1 since either is possible. The MSB authors brush one of these possibilities aside for no good reason. When it comes to presenting the estimates L2 is quite explicit about the estimators used and no reader could possibly be fooled into thinking that they have incorporated any bias adjustments.
...get out some more accurate parameter values, and see what the JPR authors predict for R.
My remarks about the business model of the Underpants Gnomes refer. One of the parameters you require, for example, is the risk of death outside the survey space. Burnham can't give you that. Try God.
I really don't think you've thought this thing through.
Nick writes:
Well put - this seems an obvious but key point wrt Tim Lambert's Platonic construction. Many thanks, Nick, for that detailed post which fills in a lot of gaps for me.
Kevin Donoghue writes:
Kevin is only half right here, at best. They may discuss "possible sources of bias", but they assume no bias in their study. In comments made by Burnham et al after MSB was suggested, they very explicitly claim no bias for their study.
As I explain below, there is nothing wrong with the JPR analysis:
you mean a model that predicts the majority of (polled) victims to be females is not wrong? when Lancet found the majority to be male?
But what I will also dare to say is: "Let's all ask the L2 authors to release information about the samplable areas. Ideally, houses surveyed -- if not, the name of streets surveyed -- if not, the name of streets in the samplable space S_i.
well, i ll limit my shouting to most basic part:
you will NEVER release the streets that you polled! this demand is insane!
P.S. Gender could be incorporated by allowing the parameters in the JPR article to carry subscripts (f) and (m) for female and male. This is just the same as adding another subsystem label (see EPL). The equations would all then have double the number of unknown parameters, and be far more complex. Without additional details about the surveys, is there much point in doing this?
is there a reason to look at a factor ("mainstreets"), when there is a different other factor (gender) that is massively more important?
the authors own assumption (males spend significant time outside their homezone) completely RUINS any attempt of a combined "mainstreet - gender" analysis. one of those two points (mainstreet) simply is irrelevant to the result!
Sod wrote: "...NEVER release the streets that you polled!"
Why?
What are they hiding? And why are you scared?
Remember, no identities required. At the bare minimum, Burnham et al. just need to tell the MSB team what streets are in the samplable region, and hence what streets are in Si (and by default So). All I can conclude from their refusal, is that (1) they don't know, or (2) they are hiding known weaknesses in the surveying. Or both, since (1) is just a specific case of (2).
Discussion of what Burnham has and whether he should or shouldnât give it to this or that individual is a diversion from the topic of this thread. However I will note that he has been quoted as saying: "Our goal was to reduce any type of risk to the community and the participants. While we have much of the raw data, we requested that anything designating the interviewers or the location of the neighborhoods visited not be sent to us."
Could anyone wishing to pursue this aspect of the Iraq mortality controversy please do so in an appropriate thread? Thank you.
Kevin Donoghue writes:
Actually it goes to the heart of this thread. We wouldn't be discussing some nonsensical Platonic abstraction (Tim's market) if we had real data on the topic to look at.
Kevin Donoghue writes:
"...is a diversion from the topic of this thread"
Did you read my earlier long post, explaining in detail why TL's claims of invalidity of JPR article, are themselves invalid? Thread topic over.
(By the way, which bit didn't you understand? Sounds like you understood none of it.)
I can see that this is not a discussion forum in the true sense, so I wish you all the best of luck in your future adventures and will now bow out from future posts here. Bye bye....
David Kane:
Well, they didn't address it in the "Clarifications" piece you linked to FWIW.
Huh? As the authors themselves point out, men from the szmpled zone likely spend more time on average in the unsampled zone than others living in the sampled zone. Similarly, males from the unsampled zone likely spend more time in the sampled zone. In other words, you would expect to see a greater sex/age differential in the unsampled zone than in the szmpled zone. But to get their numbers to work out, they would need a dramatically smaller differential there. It just doesn't add up.
What are they hiding? And why are you scared? Remember, no identities required.
you don t understand this. there are rules. you can t simply break them and publish data that will make people traceable.
Spagat needs to fix the problems in his paper, BEFORE any additional data is released. and don t expect miracles from such data.
what would some more information about the location change?
Nick: Did you read my earlier long post, explaining in detail why TL's claims of invalidity of JPR article, are themselves invalid?
Yes, it was long wasnât it? For the benefit of readers who might be in a hurry, this is the gist:
Which, if true, leaves the editor of the JPR looking a bit foolish. Not only did he publish the inferior product, but a jury said it was the best paper heâd got all year.
Hopefully Nick has only left to go and do something "much more productive for the world". Iraq could probably use further surveying work if he's up for that; his raw data might be offered to the awarded MSB team for some comfy-chaired admonishment from them. Vale Nick!
What is the actual equation that the paper derives for the bias? The actual paper seems to be behind a paywall, and the draft paper equations seem to assume f0=fi=f.
In any case, it is a perfectly valid thing to pick test cases where one thinks they know the answer and see what your "model" predicts. This is just what Tim did in #55. The fact that the equation is so far off in such a simple limiting case pretty much kills the model in my mind.
1) Thanks to Nick for taking the time to explain in #111. For those who don't have the time to read it, the key insight (which I had missed before) is that Johnson et al (2008) explicitly assume that qi and qo are constant throughout there respective areas. So, Tim's example is not a counter-example to the model since q in not constant across the sampled region. Unless Tim wants to continue this aspect of the conversation (presumably in a new thread), I would say we can conclude that, the math (at least) in Johnson et al is correct.
2) Gator: The draft pdf (linked to above) is, for the purposes of this discussion, equivalent to the published version. You claim that "The fact that the equation is so far off in such a simple limiting case pretty much kills the model in my mind." Well, please read Nick's response. Tim's example does not work because it does not meet the assumptions of the model. Any example that does meet those assumptions will produce the right answer.
David Kane and Nick. Tim describes a situation where the market is the dangerous place, and all deaths occur in the market. However, since the market is in the sampled area, one can just as well say that all deaths occur in the sampled area, and none in the unsampled. This is what Tim's example says. So the model should be valid in either case. The fact remains that for this simple limit of equal population in and out, and highly skewed risk factors, the model is wrong.
Anyway, "Nick's model"?? I don't see a Nick on the paper's author list. I just saw Nick making a new model where one has to identify and separately quantize "dangerous" areas of the sampled area. This is a whole new thing and apparently not captured in the paper under discussion. It certainly wasn't in the draft paper.
After a bit more thinking and reading, I think Nick has perhaps explained why the model in the paper does not work. (I admit I didn't read his explanation too closely because it was so longwinded...) The paper explicitly states that qi is the risk over the entire sampled area. If the sampled area includes safer areas (back from the main street) and unsafe (the main street) then the model will overestimate the risk of death for people living in the sampled area because they are assumed to spend more time there. In fact, they may spend the same amount of time in the actual risk areas (as in Tim's model) as the people from the unsampled areas. I.e. everyone goes to the market once a day, but other than that they stay away from the dangerous areas.
As Nick points out this model is too simple. But he is criticizing the Johnson et al model, not Tim's test of the model.
OK, my last comment in a bit...
One must also see that this simple risk model is likely to be wrong by realizing the violence is not randomly occurring in time or space. Presumably a "smart" terrorist will detonate their car bomb when the market is crowded, or when it is time to pray at the mosque. Assuming that the risk of death is proportional to the time in the sampled area is a gross oversimplification. This is in effect assuming the violence is random in time and space over the sampling area.
David, my example shows that the model used in the MSB paper is wrong. Their model assumes that there is no main street bias, so is only going to give the correct answer in the case of R=1.
Gator writes:
Yes, but it's not "valid" to do so by completely ignoring premises which have been spelt out for that model. The MSB authors were pretty clear in explaining where they were drawing the lines in terms of what Nick calls "two-subsystem heterogeneity". Lambert simply ignored that.
To put it another way (let's say if I were adopting Nick's precise-sounding jargon, which I can't take credit for): In a two subsystem framework, the market has to be located either in S_i or S_o. If the market is located within S_i, the samplable subsystem, then we have q_i > q_o, since the risk of death pertaining to the market has been absorbed in q_i. If A spends more time in S_i than B does, then A also clearly has a higher risk of death. Therefore, the risk of death is not the same for A and B as claimed by Tim, and sampling from S_i will introduce an upward bias. Note that there is no distinction between where within S_i the person spends his or her time, which is why why q_i is defined as the probability of death when present within S_i, not as the probability of death when present within some subset of S_i. How large this bias is depends on q = q_i / q_o, where one implicitly assumes that q_i > 0 and q_o > 0, i.e. there is always a non-zero probability of death in a war-zone. One could introduce a different three subsystem framework, in which there is a third subsystem, say, the market. In this case, however, the equations would need to be derived again, since they are based on a two subsystem model.
it is now 130 posts and not a single real reply to the gender problem. i can t really blame you for the avoidance of the subject, it is a critical error in the study.
so let us look at some other points:
Si, the samplable subsystem, then we have qi > qo, since the risk of death pertaining to the market has been absorbed in qi. If A spends more time in Si than B does, then A also clearly has a higher risk of death. Therefore, the risk of death is not the same for A and B as claimed by Tim, and sampling from Si
i don t think you understood Tim s point, and i think you assume the Spagat model to be more flexible than it really is.
a simple question should make the problem obvious. even to you!
how much time do you spend in the supermarket every week? how much more/less time would you spend there, if it was further away (closer)?
people go to the market to by stuff. they don t spend significant more time there, when it is closer to their home. so while you could model it, it wouldn t give a significant bias for the Lancet study.
on an other important point:
what use does Spagat et al see in their model?
I haven t seen the lancet guys claiming that their method of cluster choice is a model to follow. instead it was the most practical solution under EXTREMELY difficult circumstances.
with GPS systems getting tiny and extremely common and perfect aerial maps being readily available, i can t see many repetitions of the Lancet cluster choice in the future!
other problems with cluster choices (different violence in different regions) are well known and always taken care of in cluster sampling.
i simply can t shake the feeling, that Spagat is working on a theoretical paper (always popular, as few people can do math..) that is abusing the popularity of the Lancet paper.
Tim: I believe that Robert Shone's #132 answers your #130.
But, I could be wrong! If so, the best way forward is not to continue the discussion in this thread but to start a new thread in which you provide a precise description of your counter-example. Then, we could focus on precisely that debate.
So, if you still think that you have found a flaw which invalidates two published papers, please start a new thread devoted to just that flaw. I bet that Johnson et al might even respond. You can hardly expect them to respond to a point that didn't arise until comment #55 in this thread.
Robert Shone: In a two subsystem framework, the market has to be located either in Si or So.
It's not clear that we can say this. The draft I'm looking at is vague on that point. The survey space Si is a set of households. The marketplace is not a household. Is it outside the survey space by definition?
I'm not sure what to make of Tim's argument. For now I cling to my belief that, although the MSB model is unsatisfactory in just about every way a model can be unsatisfactory, it is logically valid. But maybe the way the thing is set up breaks rules I'm not acquainted with. Certainly the premises on which the model rests could have been more clearly stated.
I think the authors intended that all deaths take place in either Si or So. Nobody is allowed to die in the marketplace.
DK:
>I believe that Robert Shone's #132 answers your #130.
No, it doesn't. Their model assumes that the risk of death is the same everywhere in the sampled region. Which is main streets and cross streets. That is, it assumes that there is no main street bias in the death rate.
If you believe that main streets are more dangerous, then their model is wrong.
If you believe that main streets are not more dangerous, their model shows that the L2 sampling scheme is unbiased.
Choose one.
Tim Lambert writes:
Could you elaborate on the last sentence in that paragraph. It looks like a non sequitur to me (it also looks like nonsense). What do you denote by "main street bias"?
There is an old physics joke that ends with the punchline "First, assume a spherical cow." That is what this model has done. They have made a very simple model. Now they need to show it is applicable to the real world. A simple model like this should be either blindingly obvious or shown to match real world data.
The model is not blindingly obvious. It is too easy to come up with real-world situations that are not described by this model. Car bombs in the market, or the mosque. q is not well described as an average value over space and time. What about a Sunni that has to go into a Shite area? I.e., q is population dependent and not simply area dependent. Why would main streets be so much more dangerous? What about sectarian violence as various militias try to control neighborhoods and slums?
The authors present no evidence from the real world that might be used to convince us that this overly simplified model is nonetheless useful. They could look at data in the US. They could send a team into Iraq. They could do lots of things, but have done nothing to show this model is applicable to anywhere in the world. They have done nothing to show that the parameters they chose have any connection to the real world.
Could you elaborate on the last sentence in that paragraph. It looks like a non sequitur to me (it also looks like nonsense). What do you denote by "main street bias"?
Tim (an unimportant people like me) thinks that there is a higher risk of attacks in the REAL mainstreets.
but those have a lot of "traffic" from people who do not live nearby. and people living directly in the mainstreet actually have a tiny chance of being polled under the lancet methodology.
In a two subsystem framework, the market has to be located either in Si or So.
it could stretch from one into the other.
What about a Sunni that has to go into a Shite area?
it is obvious, that religious heterogeneity of an area would be a very important factor, when looking for violence.
but it has no place in their model, as it might not follow mainstreets...
sod writes:
That's a fairly banal observation, and doesn't invalidate MSB at all, no matter how you dress it up in Tim's oddly-worded assertions.
That's a fairly banal observation, and doesn't invalidate MSB at all, no matter how you dress it up in Tim's oddly-worded assertions.
if you claim so, it must be true....
you (again) don t get the problem. at all.
let me help you out:
if you chose a high n value (a huge part of the country wasn t possibly polled) and focus on "real big" mainstreets, then you get streets (or markets, if you prefer) that attract A LOT of visitors/traffic from outside its neighborhood. this is a problem for their model, as actually the casualty ratio for out/insiders will be very different from the one, that is based on where you live.
if you consider mainstreets to be "little mainstreets", with mainly local traffic/visitors, you will get a better out/insider ratio of casualties. but this is a problem for the study as well, as it means that more streets are considered mainstreets, bigger regions got polled and n is actually smaller than they assume it to be.
and of course, all of these problems are independent from the gender problem, that you folks still prefer to ignore...
sod writes:
It's not a problem for the model - rather it's a matter of debate for the parameters that you plug into the model. (Same for your "problem" with the n-value). See comment #3.
David Kane and Robert Shone.
Could you address the 'gender problem' that sod has repeatedly drawn attention to? I am curious to understand how you resolve this within the overall employment of the model in the paper.
Since people are having trouble following sod's gender argument, let's make it quantitative using the parameters given in the paper.
They have q=5 i.e sampled area is 5 times as violent as unsampled area. Working age males spend 1/4 of the time in the unsampled area, so their death rate is only 5x3/4 + 1x1/4 = 4 times the rate of the unsampled area. 2/7 of the population in the sampled area is working age males, so (4x2/7)/(4x2/7+5x5/7) = 24% of the violent deaths in the sampled area will be working-age males. But L2 found that 82% of the violent deaths were working-age males. That's a pretty big difference, don't you think?
Conclusion: the model and/or parameters bear no relation at all to the Lancet study.
We can also ask ourselves: what parameters do we need to plug into their model to match the observed distribution of violent deaths in L2?
Well, n has no effect and the 2/7 working-age males seems to come from the Lancet study, so the only parameter we can choose is q. Solve for the q value that gives you 82% of deaths amongst working-age males and you get 1/42 (i.e unsampled area is 42 times as violent as sampled area). Plug that into their model along with n=10 and f=(13/14) and you get R = 0.11, implying that there 5.4 million violent deaths in Iraq. Or maybe, just maybe, something is wrong with their model.
Bernard J: I address the gender issue here. You may find other entries in that thread to be of interest.
Tim: If the model were designed to predict the gender ratios of deaths than you might have a point. But the model does not do that. Nor does it predict the daily temperature in Baghdad. That doesn't mean that the model is wrong, it just means that you are misusing it, as Robert explains in #142.
Now you are, of course, free to argue that the parameter value for q = 5 is wrong and that, given higher mortality for men, a value like q = 10 would be more accurate. Perhaps you are right! If you are, then R (the bias) would be even higher. Is that the case you want to make?
Again, the purpose of the paper is not to argue that q must be 5 or n must be 10 or any specific set of parameter values. The purpose is to provide a model which can be used --- Have you finally given up the ghost in arguing that the derivation of the formula is wrong? --- in a situation with non-universal sampling and to calculate the bias for different ranges of the various parameters.
Err David, with their model the higher mortality rate for men implies that q is lower than 5, in fact it would have to be 1/42.
And no, I'm not misusing their model. It purports to say something about deaths in Iraq and where they occur. It doesn't say anything about temperature.
I have never argued that the derivation of the formula was wrong. The formula is wrong because their model is wrong. The model is wrong because it assumes that there is no main street bias.
David Kane: If the model were designed to predict the gender ratios of deaths than you might have a point. But the model does not do that. Nor does it predict the daily temperature in Baghdad.
It is my understanding that total deaths = male deaths + female deaths. David, can you tell us what assumptions we need to make about the distribution of female deaths to rescue the MSB theory from sod's critique?
Tim Lambert: Solve for the q value that gives you 82% of deaths amongst working-age males and you get 1/42 (i.e unsampled area is 42 times as violent as sampled area).
Ah, but thatâs just for men! Maybe for women, children and the elderly the sampled area is 7 times as violent as as the unsampled area! Wouldnât that give us a both-sexes all-age-groups weighted average q of 5? See, thereâs always a way. (Although Iâm not going to swear to it that even that fix works.)
Coming soon from Johnson et al: Age and Gender Bias in Epidemiological Studies of Conflict Mortality.
"All models are wrong, some models are useful." This model is clearly not useful. The authors show no link of model predictions to reality, and it is too easy to think of realistic situations where the model is clearly wrong.
Tim Lambert writes:
You forget that young males are more likely to be victims of violence for other reasons (this seems to hold in peaceful societies as well as in war-zones - for example, in the UK, males aged 16-24 are much more likely to be victims of violence than any other group). And given the inclusion of combatants in L2, this factor will probably be even more pronounced.
Unless you introduce a lot more data and many more assumptions, the issue of gender is just another red herring, along with Tim's hermetically-sealed Platonic market. Please see Nick's post #111 again for comments on introducing further "subsystems".
To address sod's many posts which express the following on gender:
It simply does not follow, for the reason I've just given in #150
Many of the violent deaths in Iraq took place in mosque bombings. The sexes are separated in mosques.
The 3Ms mosques, marketplaces and mainstreets all have something to do with this, however, the entire mainstreet argument appears wrong to me. While bombings will be concentrated on mainstreets, death squads would be concentrated on back streets and in private houses. Abductions are a lot easier on narrow lanes than wide streets with lots of traffic. The whole Spagat thing appears to be a flight from reality.
I would like to present my take on the mathematics of the gender issue (male vs. female deaths). I'll start with an example.
Assume that we are dealing with a population of 50 males (nm=50) and 50 females (nf=50), which together constitute a population of size 100 (n = nm+nf = 100). Let us assume that males are exposed to some disease with a probability qm=0.03 and females with a probability qf=0.01. How many males are expected to develop the disease? This is given by nmxqm = 50x0.03 = 1.5. The corresponding number for females is nfxqf = 50x0.01 = 0.5. In the total population of 100, altogether 1.5+0.5 = 2 people are expected to fall ill.
What if, instead of having the gender specific probabilities, we are given the probability q that a person, regardless of gender, falls ill? This probability is given by weighted population average of the corresponding male and female probabilities, resulting in q = 1/(nm+nf)x(nmxqm+nfxqf) = 1/100x(1.5+0.5) = 0.02. So what can we do with this number? Since it is a population specific probability, we can use it to calculate the expected number of people in this population who will develop the disease, which is given by nxq =100x0.02 = 2. This of agrees with our above result, as it of course should. We cannot, however, use q to calculate gender specific expectation values. We can calculate qxnm = qxnf = 0.02x50 = 1 but this is meaningless at the level of the male population, or female population, by itself. This calculation will certainly not yield the expected numbers of males or females falling ill (1.5 and 0.5, respectively), since for that we need the gender specific probabilities as above. It can only be used to calculate the expected number of people falling ill disregarding their gender, or equivalently averaging over the gender.
Let's put this in the context of the JPR paper. There qi is defined as the probability of a violence related death to anyone present in the samplable region. It does not distinguish between males and females and represents a population average (or sub-population, in this case, as it is limited to individuals, males and females, in the samplable region). The same argument holds for qo pertaining to the non-samplable region. Here, then, qi plays an analogous role to q in my example. We cannot therefore use qi in the present case to calculate the gender specific expectation values any more than we can do so in the above example. To do that, we would need to know the gender specific probabilities (qm and qf in the example) and the numbers of males or females in the population (nm and nf in the example). In this case we have neither of these. If it is men who are sent out to take care of the potentially dangerous tasks, their probability of death could be significantly higher than its counterpart for women, and one would expect a higher number of male casualties than female casualties. Also, note that even if we know the gender composition of a typical family, it does not allow us to infer the underlying number of men and women in the population, because we don't know the gender compositions of the other families. Since the JPR model does not distinguish between the genders but uses a population-wide probability in its formulation, it cannot be used to calculate the expected number of male or female casualties.
What if, instead of having the gender specific probabilities, we are given the probability q that a person, regardless of gender, falls ill?
John, you didn t get to the core of this problem.
we aren t "given" any "probabilities that a person regardless of gender falls ill".
the model is very clear:
1. people who live in a mainstreet zone, had a higher probability of being polled.
2. your risk of death increases significantly, ba the time that you spend in the mainstreet zone.
3. females (from the polled region) spend significantly more time in the mainstreet zone, than their males do.
conclusion: we would expect to find more females killed by violence in lancet poll, than males.
if you think that male and female have a different risk, that is fine (as it is obviously true). but it is in contradiction to their claim, that where you live (and spend your time) is THE important factor for your risk to get killed.
if you think that males do the more dangerous "jobs", that is fine as well. but again it is contradicting their claim, that the location of your home determines your risk of death. (if males from the polled regions and those from outside do the risky job at the market, their death risk will be similar/same)
You forget that young males are more likely to be victims of violence for other reasons (this seems to hold in peaceful societies as well as in war-zones - for example, in the UK, males aged 16-24 are much more likely to be victims of violence than any other group). And given the inclusion of combatants in L2, this factor will probably be even more pronounced.
true. but i haven t heard a lot about a "mainstreet bias" among british victims of violence.
obviously more violence will happen on mainstreets. but how much time a young male spends there, has little or nothing to do, with how close to the mainstreet he lives. (at least not in a "one street further" sense, like in the Spagat version of mainstreet bias)
and please could you guys stop pretending that the gender issue isn t par of the paper. Spagat makes EXPLICIT assumptions about gender. he is using it, to calculate how much time people spend in the danger zone!
Certainly Johnson et al. are quite specific about the fact that they are assuming two working-age males in a seven-person household. These males spend six hours per 24-hour day outside their own zone. The other five members of the household spend all their time in their own zone. Also, the survey space (Si) has one-tenth the population of the unsurveyable space (So).
Sorry folks, but complaining about sodâs taste in fonts wonât get you around this problem. (Although FWIW, I would like my reading made a little easier too.) Talk of adding subsystems won't get you around it either. We are faced with certain facts of arithmetic here:
Total population = working-age males + others, and
Total deaths = working-age male deaths + other deaths.
If anyone can generate aggregate numbers consistent with these facts, and the parameter values suggested in the JPR paper, please share them. Iâve tried and I canât seem to avoid the kind of absurdities Tim comes up with.
In #148 above I wrote, facetiously: âMaybe for women, children and the elderly the sampled area is 7 times as violent as as the unsampled area! Wouldnât that give us a both-sexes all-age-groups weighted average q of 5? See, thereâs always a way. (Although Iâm not going to swear to it that even that fix works.)â
Just as well I wasnât going to swear to it. That fix doesnât work either. (Because the aggregate q isnât a weighted average of the subset qs.)
So, what does the defence have to say?
Note also that in their model working-age males have a higher violent death-rate because of working-age males from the unsampled area visiting the sampled area -- with their parameters they would make up 40% of the deaths despite being only 29% of the population. But their model predicts that the Lancet survey would be so strongly biased that it would find that just 24% of the deaths were working-age males.
Tim Lambert writes:
No. In your misreading of their model, males have a higher risk. They make no such claim, and so such prediction follows from their gender assumptions for example parameter values for f. (See my comment #150).
sod writes:
Yes, but it doesn't follow from this that your predictions (and Tim's) regarding male/female violent death ratio can be imputed to the MSB model. (See my comment #150)
To summarise, on the gender question:
1. The MSB authors suggest parameter values for f based on a few gender-related assumptions.
2. Tim Lambert and sod impute to the MSB model, based solely on 1., predictions regarding male/female violent-death ratio.
3. This is invalid for reasons which should be obvious (see my comment #150).
Robert Shone accuses Tim: In your misreading of their model, males have a higher risk.
From the draft paper: "Given the nature of the violence, travel is limited; women, children and the elderly tend to stay close to home."
The lit. crit. approach to these discussions is always entertaining, but statistics is not a literary sort of field. It is subject to what Alwyn Young called the tyranny of numbers. Is any defender of the paper prepared to face the arithmetic?
Well, Tim does misread the MSB model, since it doesn't claim that "working-age males have a higher violent death-rate because of working-age males from the unsampled area visiting the sampled area..." (which is the comment of Tim's that I was referring to, as Kevin Donoghue well knows).
But you misread Tim - he didn't say that the paper claims that. He said that "in their model working-age males have a higher violent death-rate because of working-age males from the unsampled area visiting the sampled area" - it's not an explicit claim made in the paper but it is a logical implication of the assumptions made. There are more males in the unsampled area (So) because it's ten times as populous. The working-age males visit the more dangerous survey space (Si) while everyone else stays at home. QED, pretty much, unless you want a proof written in symbolic logic.
Kevin Donoghue writes:
It's not a "logical implication" that the MSB authors have "in their model" (to use Tim's words). It's Tim's own problematic logic, which he imputes to the MSB paper. (Why problematic? Pretty obvious - see #150).
Well, Tim does misread the MSB model, since it doesn't claim that "working-age males have a higher violent death-rate because of working-age males from the unsampled area visiting the sampled area..." (which is the comment of Tim's that I was referring to, as Kevin Donoghue well knows).
Robert S., you did not understand this. again. basically everything that you write is either a pretty wild claim or simply false. this one is the latter.
the values of n and fi and fo, plus what they write about males and females in their paper, makes it a simple task, to calculate the ratio of males inside the mainstreet zone.
Assuming that there are two working-age males per average household of seven (Burnham et al., 2006), with each spending six hours per 24-hour day outside their own zone,
a man living inside the zone spends three times as much time in the mainstreet zone,as one living outside of it. but there are many more man, living outside this zone! (n=10)
if Lancet had polled males on the (main)street, they would have found the majority of them not living nearby!
*******
the Spagat paper makes two claims, that contradict each other, when you factor in a simple fact:
1. Claim: where you live is very important for your risk of being a victim of violence.
2. Claim: males travel around a lot. they spend 1/4th of a day outside their home zone.
3. FACT: the majority of victims of violence in iraq are male.
if the killing (of males) in the mainstreet area was happening indiscriminating on the street, it would kill more men who live outside the zone, than those living inside. (see my calculations above) this would lead to a much smaller bias, than the paper claims.
if the killing (of males) in the mainstreet zone is targeting them at their homes (not well supported by evidence, btw), the question is, why their females don t suffer a similar toll of violence.
sod writes:
Sod, with respect, I think I do understand. Tim and yourself are attempting to impute to the MSB paper claims (or logical implications) about male/female violent-death ratio, based on that model's gender-related assumptions for suggesting values for f.
What I'm saying is that other important factors besides space-time location affect the male/female violent-death ratio (for example: is a male combatant more likely to be killed; are men more likely to be targeted by gunfire, etc? - see my earlier comment #150).
Obviously space-time location in itself is an important factor in the risk of death, and it raises the whole possibility of bias (which is what the MSB paper tackles). However, you can't impute specific predictions of male/female violent-death ratio to the MSB model based solely on the gender-related assumptions for suggesting f.
Sod wrote: "if the killing (of males) in the mainstreet area was happening indiscriminating on the street, it would kill more men who live outside the zone, than those living inside. (see my calculations above)"
What?? Have you seriously read and thought about this statement. It is ridiculous.
The L2 analysis makes an implicit assumption that there is *no* main street bias. The JPR paper takes the first step beyond that, which is to look at what happens if attacks are not completely random in terms of types of urban layout (which is a reasonable question to ask). But somehow out of this, you have managed to tie yourself up into knots to such a degree that your criticism of JPR is based on details which are way beyond the scope of either JPR or the L2 implicit assumptions. If you worry so much about gender and location issues being wrong in JPR (and hence important), why don't you attack the L2 in the same way? When they backed out totals, no corrections were made there for such street-location/neighborhood/gender mobility biases? Why not? Can one assume that they don't exist? Are they obviously exactly zero? No, they are not -- and that is the point of the JPR paper, to show that such biases are not obviously zero.
Would you demand your money back from a model train that you purchased, just because it didn't have real seats in it when you unpacked it at home? A model is a model, and its purpose is to show the effect of what might lie hidden in reality. 'Reality' here is the L2 study, and the model in JPR is aimed at showing what might be missing from this. End of story.
sod,
You don't have a hope in hell of persuading Robert Shone to look at the numbers. That's not his scene. It's all about words. In this case, is it okay to say that something happens in the model when what you mean is that the assumptions clearly imply that it happens? If you're fond of reading Heidegger and Derrida you can maybe have an interesting conversation with him. Then again, maybe not. I'm not so I wouldn't know.
Would you demand your money back from a model train that you purchased, just because it didn't have real seats in it when you unpacked it at home?
No, but if the wheels didn't fit the tracks I would. There is a concept in law: fitness for purpose. As anyone who has read this thread will surely understand by now, Johnson et al. is not of merchantable quality.
A model is a model, and its purpose is to show the effect of what might lie hidden in reality. 'Reality' here is the L2 study, and the model in JPR is aimed at showing what might be missing from this. End of story.
Ron, you missed the main part of the story. both Lancet and reality agree, on males being the majority of victims. but the Spagat model makes such an outcome extremely unlikely!
all my comments above are based on the assumptions of the Spagat model.
your train seats example was the most stupid one, among several pretty moronic "examples" that were given by those who support the Spagat paper.
i am really curious:
please give an explanation: what is killing those males who live in the mainstreet zone, but leaves visiting males and their households untouched?
shouldn t this explanation be in the paper? why isn t this MORE important fact part of the name of the "bias"?
mainstreet males get killed when they carry the trash outside-bias" might be a start...
What I'm saying is that other important factors besides space-time location affect the male/female violent-death ratio (for example: is a male combatant more likely to be killed; are men more likely to be targeted by gunfire, etc? - see my earlier comment #150).
well, i agree with Kevin: numbers possibly simply aren t your thing.
so i ll try three simple questions:
1. (as above) if there is a factor that is more important than "mainstreet bias", then why talk about "mainstreet bias"?
2. do you really believe, that the majority of combatants in Iraq stay in the road that they live in?!?
(hint: attackers AND defenders may take casualties occasionally...)
3. how does the gunfire pick out local men among the majority of non-local men on a mainstreet?
Kevin Donoghue writes:
Which numbers are we talking about? Tim Lambert's? Here they are again:
What is wrong with Lambert's numbers? They're derived based on the false assumption that the only factor affecting male/female violent-death ratio is space-time location. Not surprisingly, the MSB paper doesn't make that assumption. Lambert is falsely imputing stupid assumptions to the MSB paper.
Sod wrote "if there is a factor that is more important than "mainstreet bias", then why talk about "mainstreet bias"?"
The JPR paper looks at a potential bias due to a type of street bias. It does not look at potential bias due to other possibilities -- that was not their job. But it *was* the job of the L2 authors!!! This is an obvious possible bias -- now you are raising others. So do us all a favour, get on to the L2 authors and ask them how they managed to disregard these all these plausible potential biases. The L2 authors are either so very smart that they know how to estimate them and disregard them (... and yet, not so smart as to add this to the publications??) Or maybe they were smart enough to know that there are many potential biases that they haven't accounted for, but hoping that noone would notice?
The L2 authors, having made a claim of a result based on surveys, have the scientific duty to show that potential biases that many people would think of as reasonable, are not in fact present.
Do you get this? Let me repeat..... *The L2 authors, having made a claim of a result based on surveys, have the scientific duty to show that potential biases that many people would think of as reasonable, are not in fact present.*
You bunch of L2 defenders are sad, sad cases ..... God knows what you do for a career....
...get on to the L2 authors and ask them how they managed to disregard these all these plausible potential biases.
To which they will reply: we discussed potential biases at great length; please read the paper.
Which numbers are we talking about? Tim Lambert's?
No. The numbers in Johnson et al.; q=5, n=10, f=5/7 + 2/7 x 18/24=13/14 etc.
Assumptions have consequences.
Robert Shone is really arguing that it is not fair to compare the Spagat model to reality. It is a model and the model does what it does. All bow to the model. Treat the model on its own terms.
I'm wondering though, in this framework, how RS figures the paper has anything to do with the actual situation in Iraq? Can he connect anything in the model with any facts?
Kevin Donoghue: "To which they will reply: we discussed potential biases at great length...."
Wrong. Their discussion of potential biases is not complete.
'At great length' is neither a correct summary of their paper, nor does it mean such a discussion is complete.
Gator write: "I'm wondering though, in this framework, how RS figures the paper has anything to do with the actual situation in Iraq? Can he connect anything in the model with any facts?"
Can you connect the L2 inference to facts? i.e. L2 sampled households in particular types of street environment, and made the huge unjustified leap of multiplying that up to the level of the entire population.
If L2 had said "We sampled households in 'X' type of street environments, and found that Z percent of them had known casualties" then that is one thing. But to then say "and therefore multiplying up to the level of the population, and ignoring the restriction Y, we will also get Z percentage" -- that is just folly.....
After reading some of the posts it is clear that the "gender problem" arises from an erroneous interpretation of the model and in that respect I agree with the point raised in 150 and 167 by Robert Shone, which are closely related my comment in 153. You really need to make assumptions about male-specific and female-specific probabilities before you can make any statements about the number of male and female casualties. One may assume that males and females have identical probabilities to die, but then the entire gender issue is contingent on this assumption. Exploring this assumption may be interesting, but it is not an assumption that is made in the MSB papers. Again, I'm talking about the "probability of a male to be killed" and the "probability of a female to be killed". The mixing of people is conceptually separate from death probabilities and is mathematically governed by an independent parameter. To estimate mixing, one may or may not used gender based arguments, but in either case the male and female death probabilities need be specified separately *if* one wishes to make statements about male and female casualties.
I would ask those of you who believe the gender problem to be genuine to provide *mathematical expressions* for the male death probability and for the female death probability, as well as for the expected numbers of male and female casualties. I'm not asking for an example, but an expression using the notation of the paper. If we are discussing the model as is, you will not need to introduce any additional notation nor make any additional assumptions, just the expression will be sufficient.
Gator writes: "...All bow to the model"
So Gator, what is your model for estimating potential biases? Merely to say "there aren't any biases, beyond the ones that L2 dismissed in words" is not a scientific argument.
If you are a scientist, you will know that systematic errors have to be estimated in order to be discounted. If you can think of a potential error, then estimate its potential effect. That is how science works.
"Bow to L2" is what you really wanted to say -- right?
My above post should have said:
If L2 had said "We sampled households in X type of street environment, and found that Z percent of them had known casualties" then that is one thing. But to then say "and therefore multiplying up to the level of the population, and ignoring the restriction X, we will also get Z percentage" -- that is just folly.....
Ron,
This thread is about the merits of Johnson et al. If you want to make the case that itâs a good paper, castigating Burnham et al. doesnât do it. Even if Burnham issues a press release tonight saying âSorry, I made the whole thing upâ, that wonât make Johnson et al. a good paper. For the reasons why it isnât, read Timâs original post and the comments in the thread.
Ron,
If you look at the messages in this thread, you'll notice that there is actually very little discussion of L2. I don't believe anyone here has claimed that L2 is totally free of bias, or that its estimates are beyond reproach.
The sampling design of L2 was intended to eliminate mainstreet bias. Johnson et al are not arguing that the L2 authors were ignorant of the phenomenon; they are arguing that the measures taken were insufficient, and they present a model which purports to prove this point.
Many of the people here find the assumptions underlying that model to be dubious, and further argue that the distribution of the casualties inferred by the model are entirely unrealistic. If the model is wrong, however, that doesn't mean that L2 is correct; after all, there is more than one way to be wrong.
You might want to look back at post #4 if you want an example of a potential problem in L2, which does not involve mainstreet bias. And since you were wondering what some of the people here do for a career, I'll help you out a little: the guy who wrote post #4 is a professor of demography.
Regards,
Bruce
Bruce wrote: "The sampling design of L2 was intended to eliminate mainstreet bias."
With all due respect, this is not the case. It was intended to eliminate heterogeneity at a larger scale (e.g. towns, directorates etc.) but not the fine-grain heterogeneity at the level of streets.
Bruce write: " Johnson et al are not arguing that the L2 authors were ignorant of the phenomenon..."
They *are* indirectly arguing this. Since the words main-street-bias were coined by them apparently, and no mention is made in any of the Burnham et al. team's previous publications about such possible heterogeneity at the street-level, I think that the Johnson et al. article is precisely flagging up something which L2 did not address. Whether it is a large or small bias can then be debated -- but the fact that prior to JPR it was not mentioned at all, makes the JPR paper interesting in itself. The fact that the JPR authors then produce a quantitative expression for the mean-field correction (i.e. heterogeneity) at the lowest order in order to capture this possible missing L2 street-level heterogeneity (i.e. two distinct subsystems rather than one) makes the JPR article very worthy. The JPR article then *suggests* some parameter values, inviting the L2 authors to provide more information to modify these estimates. All reasonable in my view.
Bruce wrote: "..the guy who wrote post #4 is a professor of demography."
This is very, very worrying. It is just as worrying as when several famous epidemiologists initially rushed to state there is nothing wrong with the L2 approach. How can he, or others, vouch for something without even estimating the possible biases?
I happily agree that there may be several other sorts of biases which are equally possible, and some may prove to be more important than the JPR bias. But that remains to be seen when L2 release more information. Before that it is impossible to prove one way or the other).
To the professor of demography: What happened to the healthy skepticism of academics?
Is that the kind of critical thinking that is taught in demography?
In short: Johnson et al. are looking at one potential source of bias in generic sampling within systems with heterogeneity. They then suggest an application in Iraq. The JPR stands by itself in terms of an abstract model. No problem. To get truly Iraq-specific model would require Iraq-specific knowledge about what to model -- which comes back to requiring Iraq-specific information about street heterogeneity during sampling.
Finally, it is clear that Tim Lambert's criticisms are merely intended to deflect critical attention away from L2. Let's talk about L2 -- oh I remember, it is to all practical purposes, bias free. No need to ever question its assumptions....
I just randomly walked out onto my street and asked the first 5 people I met if they know the color of my car. 4 did. Multiplying this 80% record up to the population level, approximately 50 million people know the color of my car. (Or maybe, I ought to mention that 4 of these 5 people happen to live a few houses away...)
Still sure that MSB is way off the mark? Why don't you try this simple exercise yourselves?
I know that there will be posts objecting to this experiment. I am sure that these criticisms will point out how my 'simple case to test L2' implicitly mis-applies the L2 sampling procedure -- yet this is *exactly* what Tim Lambert did with his supposed 'simple case to test JPR' earlier on this page.
What is wrong with Lambert's numbers? They're derived based on the false assumption that the only factor affecting male/female violent-death ratio is space-time location. Not surprisingly, the MSB paper doesn't make that assumption. Lambert is falsely imputing stupid assumptions to the MSB paper.
i doubt that you understand their model. the calculation of R is dependent of the fi and fo values. those are simply defined by their assumptions on male and female behaviour. fi is very high, because females are assumed to stay at home. fo is very high, because only males are assumed to leave their area.
here is a simply thing to do: (weird, that i didn t have this idea before)
we know that the vast majority of victims of violence are male. so why not simply restrict the model to male persons? n stays at 10 (stupid, but ok for the moment), so does q. fi is lowered to 3/4 8male leave the mainstreet area only for 6 hours, fo is reduced to 3/4 as well (see Spagat paper) lo and behold, we nearly managed to reduce R to 1.83!!!
The JPR paper looks at a potential bias due to a type of street bias. It does not look at potential bias due to other possibilities -- that was not their job. But it was the job of the L2 authors!!! This is an obvious possible bias -- now you are raising others.
ahm, no. the "male bias" isn t a bias. it is a RESULT of the study. and a fact, confirmed by all other data sources...
The L2 authors, having made a claim of a result based on surveys, have the scientific duty to show that potential biases that many people would think of as reasonable, are not in fact present.
no. actually those attacking the paper are supposed to find evidence for an error in the paper. Spagat hasn t done that.
the Lancet paper makes a pretty strong point: they used this method, because they thought other methods to be to risky for the doctors doing the polling.
i think that those of us who didn t walk the streets of iraq after the Samarra bombing should think hard, before they make accusations.
Ron: Wow, what a strawman.
The fact remains that Spagat et al have done nothing to connect their model to anything in the real world. Until they do it is useless to comment about what it says about the real world when it is so easy to come up with real world situations that contradict the implications of the model.
Hi Ron --
The term "mainstreet bias" may have originated with Johnson et al, but the concept certainly isn't new; it's just a variation of geographic bias. And whether or not it is a large bias or a small bias is directly relevant here. There is no way to reflect every aspect of heterogeneity in a survey... but we don't need to reflect every aspect of differentiation. We only need to reflect those which will affect our results.
If you find the concept of the paper to be very worthy, that's fine. The concept could be sound, but that does not mean that is is relevant to Iraq. A large part of the dispute here centers on the parameter values that the authors "suggest." Garbage in, garbage out: even if the model is sound, invalid parameters will yield invalid results.
Regarding your comment that the professor of demography's comments are "very, very worrying," I'm not sure why you are worried, or what you think Robert was "vouching" for. I'm also not sure whether you meant that Robert was being insufficiently skeptical, or whether I was being insufficiently skeptical. In either case, it seems that what bothers you isn't a lack of skepticism: it's the fact that the skepticism is directed toward a model that you've chosen to accept at face value.
As to Tim's intentions, it seems to me that he is addressing the criticisms of L1 and L2. That is rather different than deflecting them.
Regards,
Bruce
You might want to look back at post #4 if you want an example of a potential problem in L2, which does not involve mainstreet bias. And since you were wondering what some of the people here do for a career, I'll help you out a little: the guy who wrote post #4 is a professor of demography.
what (the other) Robert wrote in comment #4 is perfect, but has a major flaw for this discussion:
all the people arguing for the Spagat paper, are 100% convinced, that both Lancet papers are complete fabrications. that the total deathtoll comparison between the two contradicts the Spagat paper has ZERO value in a discusSion with them.
just keep the argument in mind, in case you ever happen to talk to some rationale people..
I just randomly walked out onto my street and asked the first 5 people I met if they know the color of my car. 4 did. Multiplying this 80% record up to the population level, approximately 50 million people know the color of my car. (Or maybe, I ought to mention that 4 of these 5 people happen to live a few houses away...) Still sure that MSB is way off the mark? Why don't you try this simple exercise yourselves?
oh, i just did. i numbered the two "mainstreets" of my small home town. (actually i live in the mainstreet bias zone..) thewn chose one of them randomly. numbered the roads departing from it, choosing one at random again. on that road i chose 5 (random houses).
i ll ask them for the colour of my car tomorrow...
I know that there will be posts objecting to this experiment. I am sure that these criticisms will point out how my 'simple case to test L2' implicitly mis-applies the L2 sampling procedure -- yet this is exactly what Tim Lambert did with his supposed 'simple case to test JPR' earlier on this page.
no. you still don t understand what he did.
sod: lo and behold, we nearly managed to reduce R to 1.83!
But if I'm understanding you correctly the bias for non-workmen is then 3.67 because f=1 for them. It's the bizarre assumption that fi=fo that produces most of the craziness, I think. They could have told a more plausible story with fo>fi, the idea being that people avoid the dangerous survey space as much as possible. But then they would have had to think about the underpinnings of the model, which they preferred to leave vague. Is a mosque in Si, or in So, and how do you tell? The paper doesn't say.
Kevin Donoghue said: "....It's the bizarre assumption that fi=fo that produces most of the craziness, I think."
The JPR paper does not need to make this assumption. See Equation (4) of JPR -- it is general. As in all theory papers, it is then interesting to look at certain special cases of the parameters to get a feel for what is going on. But Eq. (4) and the whole JPR theory does *not* have this fi=fo limit applied.
Sod wrote: "...no. you still don t understand what he did."
Err.... I don't understand what L2 actually did (not what they say they did, what the team actually did).
As for TL? Yes, I fully understand what he said/did. His criticisms are misguided.
PS. I would place good money on your hometown layout being quite a bit different from Baghdad....
Sod said: "...small home town..."
Wait a minute, you are a genius. So small home town maps to Baghdad in terms of layout? Even though your hometown has only two mainstreets... and is small.
Errr.... look down and check the hole you just shot in your foot..
Sod wrote: "...all the people arguing for the Spagat paper, are 100% convinced, that both Lancet papers are complete fabrications. that the total deathtoll comparison between the two contradicts the Spagat paper has ZERO value in a discusSion with them...."
Incorrect about the percentage. About 90+....
As in all theory papers, it is then interesting to look at certain special cases of the parameters to get a feel for what is going on. But Eq. (4) and the whole JPR theory does not have this fi=fo limit applied.
the problem is, you change the parameters so that they make sense, and the paper stops contradicting the Lancet results.
Err.... I don't understand what L2 actually did (not what they say they did, what the team actually did).
feel free to replicate their results. nobody is holding you back!
Wait a minute, you are a genius. So small home town maps to Baghdad in terms of layout? Even though your hometown has only two mainstreets... and is small.
so mainstreet bias does only apply to towns the size of Baghdad? that point is interesting. where does Spagat say this?
and you might notice that my method is much more similar to the lancet one, than what you proposed..
As for TL? Yes, I fully understand what he said/did. His criticisms are misguided.
no, you didn t understand what Tim did. if you were interested in car colours, he would advice you to look closely at gas stations.
sod,
I think the following probabilities of death will get you results which, though implausible, are not completely insane:
Workmen in Si: 0.075
Workmen in So: 0.015
Others in Si: 0.005
Others in So: 0.001
Even these values donât quite work, but I think thatâs because the model is constructed in such a way that it canât be sensibly disaggregated into sub-models. Obviously that, if true, is a drawback of the model; but itâs hardly the worst.
John,
Thanks for pointing me in (I hope) the right direction. Iâm sure sod and Tim donât need to be reminded of the basics but I do, now and then. Not sure if my comment to sod answers your question but itâs the best I can do for now.
Ron,
Equation (4) is just a scaled-up ratio of two expected values. I think youâll find even demography professors can come up with things like that. But it took three physicists, an economist and a statistician to rig the parameters so as to get the results Tim quite rightly objects to. Equation (1) wouldnât bother anyone very much if it were not for the specific numbers Johnson et al. are pushing. However I wouldnât like it myself, since even in the general case it assumes Iraqis are stupid (fi=fo and independent of q). I donât usually like models which assume such extreme irrationality.
Sod wrote: "feel free to replicate their results. nobody is holding you back!"
First step: Get L2 to tell us exactly what their survey team did.
Sod wrote: "...mainstreet bias does only apply to towns the size of Baghdad?"
JPR applies as a general theory (i.e. Eq. (4)) to any system with heterogeneity of two subsystems. EPL to two or more. The L2 street bias will have varied from town to town in its value. Is that so hard to understand for you?
Sod wrote: "...no, you didn t understand what Tim did"
Yes I did.
KD wrote: "..Equation (4) is just a scaled-up ratio of two expected values. I think youâll find even demography professors can come up with things like that."
Good, then have any of them proved that Eq. (4) is mathematically incorrect? Not just that "one can do a better approximation" -- I mean they have to show that it is mathematically incorrect??
KD wrote: "....Equation (4) is just a scaled-up ratio of two expected values. I think youâll find even demography professors can come up with things like that. But it took three physicists, an economist and a statistician to rig the parameters so as to get the results Tim quite rightly objects to."
Where in the literature does Equation (4) appear? It doesn't. So the three physicists, an economist and a statistician did a little more than you are willing to give them credit for...
Equation (4) does *not* assume fi=fo. Is that clear?
Want it to be q-dependent? Derive an appropriate form yourself.... What non-linear dependence will you chose? Hmmm, I bet you wish you knew more about the specific L2 neighborhoods surveyed, then you could pin down the q-dependence. Aha, now you agree with me -- L2 need to release info about this in order to make a proper evaluation about whether MSB is a major bias.
Sod wrote: "...no, you didn t understand what Tim did. if you were interested in car colours, he would advice you to look closely at gas stations."
You cannot ask a gas station questions. And since all this is about asking people questions (i.e. a survey) then you have problem. You are actually admitting that where you ask people questions can affect the result. QED.
Go back to my earlier color-of-car survey example. To me, it sums up the essence of MSB. And it is reasonable, simple -- and beyond L2's considerations of potential bias...
It encapsulates a major, major flaw in L2. There may be many others, but MSB lives as a possibility...... It will do until L2 clarify exactly what was and wasn't surveyed. Of course, they don't want to release this data (irrespective of security issues) since it may invalidate their study....
Here is **a challenge** to the L2 defenders and/or MSB attackers, which is potentially very useful for quantifying MSB:
Walk out onto your street and ask the first 5 people you meet if they know the color of your car (or house etc.). Multiply this fraction who know, up to the population level of your country. Is the result reasonable, or larger than reasonable? I doubt if it is smaller than reasonable. Hence R>1.
**!! CHALLENGE TO TL and others... !!**
Walk out onto your street and ask the first 5 people you meet if they know the color of your car (or house etc.). Multiply this fraction who know, up to the population level of your country. Is the result reasonable for the number of people in your country who know the color of your car, or larger than reasonable? I doubt if it is smaller than reasonable. Hence R>1. Hence TL should, as an apparent scientist, agree to change this blog title (or open a new one) titled:
** Lancet publishes badly flawed paper (i.e. L2) **
ron,
It would only be applicable if you asked those people the color of their own car. Burnham, et al. polled households for deaths in their own households not their neighbors.
Luminous beauty wrote: "ron,
It would only be applicable if you asked those people the color of their own car...."
No.
ron,
You can't tell shit from shinola.
The stupid, it burns!
Ron, a few people have taken the time to respond to you on the good-faith assumption that you are genuinely interested in this discussion. Your messages have become so absurd that I have to conclude that you are either a troll, or you are very, very badly confused, not only about L2, but about the MSB paper as well. (Longtime Deltoid readers: does this remind you of that time when that guy from Chicago Boyz was here, arguing that the high crude mortality in some clusters could be offset by negative crude mortality in others???)
Setting that aside, Ron, I'm happy to take your challenge. But remember... we are testing "main street bias," and I don't live on a main street. So to see whether or not asking on a main street will bias the results, let's do this: I'll first pick a random neighborhood of my city, and ask 5 people if they know the color of my car. Then I'll go to one of the main streets, and ask if they know the color of my car.
I am willing to bet that in the random neighborhood, exactly zero people will know the color of my car. And I am also willing to bet that on the largest street in my city, exactly zero people are going to know the color of my car.
I am also willing to bet that in a complete nationwide census, the number of people who know the color of my car will be somewhere around... oh, I dunno, maybe .000003%.
In which case, I think we could say that my random survey was fairly accurate. We could also say that main street bias didn't significantly affect the results.
LB wrote: "It would only be applicable if you asked those people the color of their own car. Burnham, et al. polled households for deaths in their own households not their neighbors."
OK, I just did it: For various reasons I have seen 3 people outside of my household in the last few hours. I asked them to name the color of their car. Two of the people couldn't even remember if they had a car, while one named the color of his car. Why? The first two live in a warden-controlled gated community for people needing 24/7 care because of dementia. So do 2/3=66.7% of the population not know the color of their car? I don't think so.
My point is: Bias is the norm, not the exception. Every finite study is susceptible to implicit biases. These biases have to be analyzed and quantified. MSB does this for one type of bias, albeit in a simplistic way, and produces a general formula (Eq. (4) for 2-subsystems, and EPL for a general number). You've done a good job, JPR authors.
LB and BS (.. telling initials me-thinks...) wrote various things which are non-scientific.
Simply point me to the exact statements that I made that are incorrect and *prove* they are incorrect, and then we have a basis for discussion. Otherwise, just carry on being rude. You are getting a D-grade so far in the debate...
BTW, a question to LB: What is the precise methodology underlying the phrase "The stupid, it burns!". It seems to have syntax errors, with a bias....
Bruce Sharp wrote:
" I'll first pick a random neighborhood of my city..."
THAT IS THE POINT!!! L2 didn't pick a RANDOM neighborhood, in the sense that random should mean unbiased.
You got it Bruce, good on you!!! At last!!!
You have now officially understood the problem with L2, and simultaneously the value of JPR. Case closed. I am leaving....
**Kevin Donoghue**: You wrote in your reply to sod and myself regarding probabilities of death: "Workmen in Si: 0.075 Workmen in So: 0.015 Others in Si: 0.005 Others in So: 0.001" I'm glad you did because I hope this clarifies the gender discussion. It is clear that the model as presented in EPL could be augmented to incorporate gender, but that would double the number of model parameters, and would probably lead to even more controversy. My point really is that the way the model has been put forward in the JPR article simply does not deal with gender. Therefore, any attempt to compare "predicted" male and female casualties to empirical data is simply misleading. To get to the gender issue in the first place, you need to make your own assumptions about gender specific death probabilities, as you did above. That is fine per se, but what is misleading is to say that the MSB papers make assumptions about male and female death probabilities when, in fact, they don't. Therefore, there really is no contradiction with any kind of evidence regarding male and female deaths, simply because the model does not make any statements about male and female deaths. It only operates at the level of two distinguishable sub-populations of individuals (samplable vs. non-samplable).
**Sod**: After reading the EPL paper carefully, it is clear that the death probabilities qi and qo are, in fact, population averages, just like in the disease example that I presented. This is given explicitly in the EPL paper, at the very bottom of page 2 and very top of page 3 (granted, they are a bit difficult to see in the text). If one allows for Ni subsystems in the samplable space, i.e. as many subsystems as there are people, then qi is simply the average of death probabilities for everyone in the samplable system, men, women, children, etc. Please see also my comment to Kevin above.
About some of the criticism directed against the parameter values. Perhaps the draft of the JPR paper is different from the published version, but in the actual paper there is Table I, in which the values of all of the parameters, q, n, fi, and fo are varied. There are altogether 225 different parameter combinations listed there. In the EPL paper, there are nine different surface plots that explore the parameter space. Alternatively, one can of course also just plug in any values in eq. 1 in the JPR paper and see what comes out. The parameter values in the text are considered "plausible" in the JPR paper, and in the EPL they are based on "plausibility arguments". What is plausible to one person may not be that to another, but it doesn't sound like the authors are suggesting that these parameter values are carved in stone. They provide a simple, transparent equation, carry out sensitivity analysis, and write in the abstract of the JPR paper: "The authors provide a sensitivity analysis to help readers to tune their own judgements on the extent of this bias by varying the parameter values."
John
Sod: After reading the EPL paper carefully, it is clear that the death probabilities qi and qo are, in fact, population averages, just like in the disease example that I presented. This is given explicitly in the EPL paper, at the very bottom of page 2 and very top of page 3 (granted, they are a bit difficult to see in the text). If one allows for Ni subsystems in the samplable space, i.e. as many subsystems as there are people, then qi is simply the average of death probabilities for everyone in the samplable system, men, women, children, etc. Please see also my comment to Kevin above.
you have it wrong John. they are explicitly using gender to calculate fi and fo.
an extreme example will show this easily: if their working males would spend ALL their time outside their homezone. (just a slightly different approach to their 6 h version..)
the outcome would be clear: Lancet 2 shouldn t have found a SINGLE working age violent death of a male in their sample.
Sod: After reading the EPL paper carefully, it is clear that the death probabilities qi and qo are, in fact, population averages, just like in the disease example that I presented. This is given explicitly in the EPL paper, at the very bottom of page 2 and very top of page 3 (granted, they are a bit difficult to see in the text). If one allows for Ni subsystems in the samplable space, i.e. as many subsystems as there are people, then qi is simply the average of death probabilities for everyone in the samplable system, men, women, children, etc. Please see also my comment to Kevin above.
i think you don t understand me. i don t oppose the idea of sampling bias. their formula is ok.
just everything about using the lancet paper as an example is wrong. all their chosen values are wrong. their modelling of the "mainstreet" is wrong. (lancet actually has a low probability to poll any of their chosen "mainstreets") taking Lancet as an example is wrong, as it was a method that was chosen for security reasons and is unlikely to be repeated. all the attacks on Lancet are wrong. (data release. they simply don t have a place in such a theoretical work)
and their results are wrong, as shown for example by the gender issue, or by comment #4, and and and..
John: what is misleading is to say that the MSB papers make assumptions about male and female death probabilities when, in fact, they don't.
I don't see how you get to that conclusion. The probability that a female resident in the survey space dies outside it is zero, because Johnson et al. explicitly assume that she never leaves. Similarly a female non-resident can't die in the survey space. Other restrictions follow from the fact that working-age males spend a specified amount of their time outside their own zones. So, for example, if you know the probability that a working-age male from Si dies (anywhere), you also know the conditional probability that a working age male dies in So specifically. And so on. When you take these things into account, there isn't much elbow-room in the model.
It's because of these restrictions that I had trouble finding the four probabilities which (I think) meet the objection sod has been making throughout this thread. The assumptions n=10, q=5, f(for working-age males)=0.75, f(for all others)=1 impose horribly tight restrictions on the probabilities.
Johnson et al. want their model to justify their claim that Burnham et al. estimate three violent deaths for every one that actually takes place. If the assumptions make any sense they must be consistent with the composition of the death-toll as well as the total - there has to be some disaggregated version of the model which generates R=3 as desired. After some doubts, engendered by sod's protests, I've come to the tentative conclusion that the MSB approach can handle that problem, but only if we take all aspects of the MSB squad's worldview on board.
In short, although I think they are quite wrong about Iraq and I don't think much of their model, it's not impossible to reconcile the gender-composition of deaths in Burnham et al. with their assumptions. But if you think it's easy, I urge you to put some numbers in a spreadsheet and see if you can come up with an alternative to my four probabilities.
Sorry John, this was a bit cryptic:
In fact I'm not sure that's strictly true. But what I have in mind is that (1) your f gives you the weights to apply to the two (unknown) probabilities and (2) your q gives you their ratio. It may be true that males and females could have distinct q values, which complicates things, but you're stuck with the given q for the total. (I can only say it may be true because I haven't been able to make an example, with distinct q values, work in practice.)
If I was a bit more diligent I'd set up the whole system of equations in a systematic way in order to see just what's possible and what isn't. But I'm a bit bored with it at this stage. Nobody seems to want to defend the model, complete with the actual parameter values the authors propose. And if we can all choose the parameter values we like, what's the point?
Ron in #209: Having thus vanquished King Arthur by bleeding all over him, the Black Knight declared victory and wobbled away on his legless, armless torso, to bask in the warm glow of triumph.
Looking at the draft version of the paper, I'm confused by one of the assumptions that the authors make in support of their fractions of time spent in or outside the survey space. Perhaps I am misinterpreting something? The draft says:
"Assuming that there are two working-age males per average household of seven (Burnham et al., 2006), with each spending six hours per 24-hour day outside their own zone, yields fi = fo = 5 / 7 + 2 / 7 Ã 18 / 24 = 13 /14 ."
Does that mean they are assuming that the working males are always working inside the survey space? That would imply that every factory, market, office, warehouse, garage and so on is in the survey space. For every work location to be on a "main street," you'd have to be willing to use a pretty liberal definition of "main street."
I'm also having a problem with the derivation of q. The authors' argument that the main streets are going to be prime targets for IEDs, car bombs, etc., makes sense, but on the other hand, I would think that other types of violence (abductions, for example) would be more likely to take place in more isolated areas. So how do you quantify the relative levels of violence? Obviously, several people have already commented on the way the parameters are pulled more or less out of thin air, but the support for q=5 in the draft seems bizarre. Is this the same in the published version? In the draft, they write:
"Given the extent and frequency of such attacks, a value of q = 5 is plausible. Indeed, many cities worldwide have homicide rates which vary by factors of ten or more between adjacent neighbourhoods (Gourley et al., 2006)." The link to Gourley 2006 in the draft is no longer valid, so I can only guess what that document says. What I find striking, however, is the idea that a drastic difference in homicide rates in other cites would have any relevance at all in this context. A tenfold difference in violence could, I suppose, be used as justification to say that we know that values as high as q=10 are plausible... but then, depending on whether violence was higher inside or outside the survey space, we could also say that it means that values as low as q=1/10 are also plausiible. In other words, it doesn't tell us anything at all.
I don't see any problems with the logic of the model, but I'm with Kevin: without accurate parameters, I don't see how it could ever be useful.
I've commented briefly on this debate at my blog:
http://dissident93.wordpress.com/2009/02/15/pope-of-debunkers/
Hi Robert --
Thanks for posting the link. However, I don't understand your first objection, concerning the parameters. You say that nobody knows the correct parameters "partly because the Lancet authors won't release necessary information." What could they possibly release that would allow you to determine an accurate value for q? Maybe I'm entirely confused (always a possibility), but I'm not sure how you are supposed to determine the possibility of death outside the survey space without... you know... conducting some sort of survey.
Regards,
Bruce
Oh, so Shone reckons I'm the Pope now? What are the odds that David Kane will describe his nonsense as a tour de force?
I just stumbled upon the [now classic CMR thread](http://scienceblogs.com/deltoid/2007/08/robert_chung_on_david_kane.php).
Highly likely I should say.
_Bruce S. ...What could [Lancet study authors] possibly release that would allow you to determine an accurate value for q?_
If the Lancet authors released the information of what streets were in the samplable region (i.e. not necessarily the sampled streets, just the candidates during the selection process) then by superimposing maps such as the BBC's one of Baghdad bombings and attacks, which are at street level resolution, an estimate of the q-values (in particular, qi/qo) could be made. Better still, if they released the actual streets surveyed (i.e. not necessarily the exact houses) this would improve the estimates of q-values.
_So yes, they could release very important information to pin down the MSB parameters. Without this information, there is no point in asking questions about market-specific scenarios. Unfortunately, the Lancet authors are preventing any advancement of this topic._
Ozzy, thanks, that helps a bit. I hadn't seen a map of attacks. That would certainly be an improvement over the s.w.a.g. (scientific wild-ass guess) algorithm. But wouldn't the reporting of violent incidents suffer from just as much main street bias the violence itself? I'd think that a map of violent incidents from a media outlet would represent essentially the same incidents as those cataloged by Iraqi Body Count. I'm not necessarily sold on the Lancet figures, but I think virtually everyone here would agree that the IBC figures represent only a subset of the actual violence. Wouldn't it be reasonable to assume that at least part of that discrepancy between the IBC figures and other estimates comes from incidents that occur in relatively isolated areas?
Regards,
Bruce
_Bruce S....yields fi = fo = 5 / 7 + 2 / 7 Ã 18 / 24 = 13 /14 ."
Does that mean they are assuming that the working males are always working inside the survey space? That would imply that every factory, market, office, warehouse, garage and so on is in the survey space. ...._
Some points:
1. Their theory does not assume fi=fo. See Eq. (4) of their published JPR article.
2. The closer fi and fo are to 1, the higher the bias R since the more localization of people in their own q-specific zones (i.e. qi or qo). So if you would have preferred that males from So be assigned less time in Si, then this makes R even larger and hence MSB even bigger.
3. There is no assumption about 'main streets' per se being the root of the bias. The words 'main street bias' does not mean that 'there is a bias for people living/working on main streets'. Instead it means 'a bias which has to do with the use of main streets as an initial identifier in the Lancet sampling selection process'. Since that is a bit of a mouthful, they shortened it to 'main-street-bias'. This is what I remember at the time, and this is what one of them personally told me.
Following up on this last point, I notice a very common misconception by people attacking the JPR on this thread, in which they interpret the JPR team's 'main street bias' as having something to do with living on a main street or working/walking on a main street. It is not. Main street bias just means 'a bias which has to do with the heterogeneity of streets in terms of where car bombs, street markets and their attacks are likely to occur'. It was simply the fact that the Lancet team started with main streets as the reference point for selecting cross streets, that led to the name. If the Lancet team had used 'unpaved streets' as the initial reference point for choosing cross-streets, then this would have *also* led to a bias (of possibly very different value, but of similar cause: i.e. street heterogeneity in terms of casualties-on-the-street).
The key point is that attacks etc. are not evenly distributed across a city in the same way as throwing darts blindly at a map of Baghdad (for example). Attacks are localized in areas around cross street to main streets (see the BBC maps, among others). Main streets in Baghdad, for example, tend to have relatively few actual residents -- they are essentially dual carriageways. People tend to walk around on cross streets -- and that is where markets are, and where cars can easily park. Hence market bombs, car bombs, sniper attacks, convoys etc. are concentrated there. It does not mean that no attacks, kidnapping etc. occur elsewhere, or that other back-alleys do not contain people living in fear. It just means that casualties are concentrated around cross-streets to main streets. Hence MSB, hence R>1.
Because of this possibility, the Lancet study's street selection should have been designed such that there was no street heterogeneity -- or if there was, then it should have been corrected or quantified by the Lancet authors. They didn't do this, or even mention this in their original paper -- hence the whole JPR story....
Ozzy, on point 1, my apologies: as I noted, I have only seen the draft, and not the published version. On point 2, yes, I realized that less time in Si makes MSB larger. I didn't raise the question because I want to send the results in one direction or another; I raised the question simply because it seems like a bizarre assumption, and a poor model for the real world. On point 3, thanks... that's a helpful clarification.
If the Lancet authors released the information of what streets were in the samplable region (i.e. not necessarily the sampled streets, just the candidates during the selection process) then by superimposing maps such as the BBC's one of Baghdad bombings and attacks, which are at street level resolution, an estimate of the q-values (in particular, qi/qo) could be made. Better still, if they released the actual streets surveyed (i.e. not necessarily the exact houses) this would improve the estimates of q-values.
all of this is false. the method was chosen, because road names and accurate maps were NOT available.
the BBC map is useless. the red dots are the size of a cityquater. incidents that kill 10+ people are far from forming a majority of kills.
and even then, these dots only show where the killing happened, NOT where the people involved lived!
Their theory does not assume fi=fo. See Eq. (4) of their published JPR article.
Eq. (4) is in an appendix in the draft and, I presume, in the published paper also. Wherever it is presented, Eq. (4) is completely uncontroversial. It is no more a model of survey bias than "Trade surplus = Value of Exports - Value of Imports" is a model of international trade.
To obtain a model you have to go beyond definitions and tautologies. You have to make some behavioural assumptions. That's what Johnson et al. do; Eq (4) is merely their point of departure. So yes, their theory does assume fo=fi.
But even then, although their R(q,f,n) function is unsatisfactory in that regard, nobody would bother arguing with them if they weren't pushing for the parameter values Tim rightly objects to, and relying on question-begging devices such as their BBC map. (If media sources are provide a reasonably complete picture of the violence then it's obvious that Burnham et al. are way off; the whole argument turns on whether there is a lot of violence that we don't see reported.)
_Sod...all of this is false. the method was chosen, because road names and accurate maps were NOT available.
the BBC map is useless. the red dots are the size of a cityquater. incidents that kill 10+ people are far from forming a majority of kills. and even then, these dots only show where the killing happened, NOT where the people involved lived!...._
Actually, Sod is not correct on various points. First, some identifier of main streets (and cross streets) needed to be made in order for the cross-street-to-a-main-street to be randomly selected. This information is what is needed for estimate of q to be made.
The BBC map is quite useful. See the authors EPL article, which demonstrates that size of dots is adequate for distinguishing, at least in an approximate way, information about relative frequency of attacks near cross-streets as compared to other street topologies.
Most importantly, Sod, the q parameter does not depend on where people live. It is simply the rate of attacks in that subspace. So qi is the probability of attacks in subspace Si,and qo is the probability of attacks in subspace So. This is easily generated using the BBC map, as long as some rough idea of where the main-streets and cross-streets are -- at least, which are the ones that could have been picked (and hence define Si, and by extension So).
_Kevin Donoghue....Eq. (4) is completely uncontroversial. It is no more a model of survey bias than "Trade surplus = Value of Exports - Value of Imports" is a model of international trade. To obtain a model you have to go beyond definitions and tautologies. You have to make some behavioural assumptions. That's what Johnson et al. do; Eq (4) is merely their point of departure. So yes, their theory does assume fo=fi...._
It is very nice to see that Mr. Donoghue agrees that the JPR's main result, an analytic formula for bias R based on street heterogeneity, is completely uncontroversial. Let's just pause and take that news in!
Now, the remaining points are again, with all due respect, wrong. To compare Eq. (4) and the analysis that produced it, to ""Trade surplus = Value of Exports - Value of Imports" is a model of international trade", is bizarre and misguided.
Next point, their theory does not assume f0=fi. The theory of superconductivity does not assume a given implementation of the parameter values (e.g. phonon frequencies), nor does the theory of gravitational attraction assume certain distance separation of objects. One obtains a theory, with parameters, and then (as the JPR authors nicely put it in their introduction) they put in candidate values leaving readers to choose their own.
Unless you cherry-pick a vanishingly small set of possible parameter values (i.e vanishingly small compared to the space of possible parameter values) you will not get R=1. And as soon as you reach that straightforward conclusion that R is not 1, you are led to ask the Lancet authors to address the issue by clarifying the R value. Whether R is actually near 3, smaller, or larger, remains to be seen.
I apologise to my colleagues for multiple postings, but I missed this comment:
_Kevin Donoghue....If media sources are provide a reasonably complete picture of the violence then it's obvious that Burnham et al. are way off; the whole argument turns on whether there is a lot of violence that we don't see reported....._
Rather incorrect. All that is needed is the relative danger of samplable Si region compared to the non-samplable So region. This will give the q. The fact that the media may have under-reported each of those red circles by factors of 5, 10, 100 etc. is irrelevant for the estimation of q since q is simply a ratio.
My above comment "...The fact that the media may have under-reported each of those red circles by factors of 5, 10, 100 etc. is irrelevant for the estimation of q since q is simply a ratio..." was rather loose. It is correct in substance, but those factors would of course remove practically all attacks from ever taking place. What I meant was: any under-reporting by some reasonable factor (e.g. 3) would not matter for the purposes of estimating q.
..'Lancet authors to address the issue by clarifying the R value'..
Sorry. The Lancet Authors should go provide empirical evidence for n, q and f (which would most likely now have changed considerably) because someone can colour in some google maps and has some odd ideas about womens place in the home?
Dear Jody. You wrote: ".The Lancet Authors should go provide empirical evidence for n, q and f (which would most likely now have changed considerably) because someone can colour in some google maps and has some odd ideas about womens place in the home?."
All they need to do, is declare what the samplable space was at the time of the survey (i.e. Si, and hence So by default). If they are willing to provide more than this, yes it would help pin down the q values more -- but not necessary. Just Si and So will do. We can all do the rest together....
How would that help us with N0, Q0 and f0 and with explaining the gender distribution in the presented results? What would we then do with the exact street addresses of each survey point to find out these values?
"...declare what the samplable space was at the time of the survey ..."
Just to be crystal clear: This requires simply a list (or even simpler, a Google map with lines) identifying the main streets and cross streets used in the Si pool, from which the actual sample selection was then drawn. Then everything not included, is So by definition.
No names of households needed, nor the specific names of streets used as starting points. Just the initial *pool* which made up Si.
Obviously, giving the specific streets would be a real bonus to getting accurate q values, and then better estimates of the f's and n's based on other information. But I guess this type of release will never happen. So to get going, all one needs is Si.
Jody write: "How would that help us with N0, Q0 and f0 ?"
Hmm, now I see you don't really understand what these parameters are -- and hence what the model is. In that case, I don't know what to say other than repeat: Yes, it will help with q immediately. Then from access to information about housing density and distribution of commercial activity at the time from other sources, we can get at better estimates of Ni, No, fi, and fo. Then R. Job done. Even gender labels can be added, by generalizing f etc. to carry the gender label. ANd the EPL result generalizes the R calculation. Etc...
But without Si and So, no further progress beyond JPR can be made. The whole conversation becomes pointless.
So how exactly would you work out S0, Q0 and F0, F1 then? Just for it to be clear in my mind.
I have to run off, but quickly:
"how exactly would you work out S0, Q0 and F0, F1 then? Just for it to be clear in my mind..."
So is the rest of the streets that are not in Si. So knowing Si, and knowing all the streets that exist, So=All-Si.
Knowing So, and taking e,g, the BBC map, one can simply count the red circles in Si and So, and the ration is qi/qo. Then the f's can be deduced roughly by knowing something about the employment and markets inside and outside of So. This is available, very roughly, from lists of businesses and markets. Apologies for rush: Got to go, back later....
So for when you're back..
Again just to be clear it is not that we just need Si from Burnham et al, but that we also need to estimate qo using some mortality data that by definition cannot have come from Burnham et al.
For this we choose the BBC's map of media reporting of deaths, saying that under reporting is not an issue for a ratio.
So we are to adjust a result because we believe the sampling method was not random, and we are to adjust this result by a factor generated from media reported deaths.
Anyone wish to collaborate on a paper? I propose a factor, let's call it R, due to Media Street Bias. Intuitively I think this should be proportional in some way to the amount of whisky available locally, the other events in the news that day , and whether or not the military drive you to one area or another to report on what they think you should. We could probably get it into a journal if we make the formula look complex enough.
Ozzy, in #228, I think you are missing the point that both Kevin and I tried to make: if you truly believe that mainstreet bias affects the level of violence, why would it not also affect the reporting of violence? Are we supposed to believe that although terrorists don't stray from the main roads, journalists do? I don't know how to make the point any more clearly than I did the first time: there is clearly a discrepancy between the toll reflected in media reports (i.e., the IBC count) and the actual toll. That discrepancy is deaths unreported in the media. You can't use the media reports to provide a baseline figure for the deaths that the media doesn't report... and that is figure you need to determine realistic ratios.
ozzy: It is very nice to see that Mr. Donoghue agrees that the JPR's main result, an analytic formula for bias R based on street heterogeneity, is completely uncontroversial.
Eq. (4) is not based "based on street heterogeneity" whatever that may mean. It's based on the assumption that we have two sets Si and So and we make inferences based on a sample taken from one of these only.
ozzy: their theory does not assume f0=fi.
My theory:
I could be wrong of course. Tim, has Tim Blair or somebody of that ilk linked to this post? I'm seeing a pattern in the comments.
Ozzie writes:
Nonesense. If someone takes your guesses and model and gets that niobium is superconducting at 400 K you get laughed at. At best you then adjust the model to account for bounds on the parameter space, at worst, if the guesses are realistic your model gets tossed on the trash heap.
What? Is this the noodly appendage theory of gravitational attraction?
**Jody**: I would ask you to take a look at my earlier postings on the gender issue. There is little point in repeating the argument here, but the main point is that the model in its present form makes no statements about male and female deaths. Therefore, there is no discrepancy with "reality".
**Bruce**: I would be interested in hearing your view on the following. It seems to me, and I believe everyone here would more or less agree, that the media only reports some fraction of the deaths, i.e. underestimates the actual death toll. In particular, they probably report only the most devastating cases with most casualties. So for example, the BBC map only shows cases with more than 10 deaths. But I would have assumed that whether these incidents get reported is independent of, and hence uncorrelated with, their location. In other words, whether incident X (with 10 or more deaths) taking place in Y gets reported is independend of Y. Do you see a problem with this assumption?
John
John: ... I would have assumed that whether these incidents get reported is independent of, and hence uncorrelated with, their location.
That seems very wrong to me. Numerous reporters have made it clear that they had very little freedom of movement in Iraq at the height of the conflict (it's probably better now). Whether something got reported had a lot to do with whether the scene could be reached safely. Whatever about L2, there is no doubt at all that camera crews had a main street bias.
I am back, and was very disappointed to see the level of thought in the responses.
In the last few exchanges I had before I left, I thought some element of reasonable progress was being made. Now I come back, and see that all the people against MSB papers have resorted to simplistic, trite comments without substance or focus. You really don't like a proper debate do you?
I think it is a waste of my time trying to engage you with responses. I suspect my comments are uncomfortably reasonable for you. I had seen some others drift away from this site, and now I see why -- with childish comments like "Media Street Bias", and others, you guys' thoughts are worthless.
Hi John --
I think that assumption is reasonable enough. However, at best it only address part of the problem. Incidents with high death tolls are a subset of the total deaths, and again, the problem is not the known incidents. The problem is the unknown incidents. Abduction and murder, for example, would represent a different subset of deaths. Do we have any reason to believe that this subset occurs with the same relative frequency inside and outside the survey space? As noted above, I think you could make a reasonable argument that these incidents would be more likely in isolated areas. Does a map of high-fatality incidents tell us anything at all about these deaths?
Jody: I would ask you to take a look at my earlier postings on the gender issue. There is little point in repeating the argument here, but the main point is that the model in its present form makes no statements about male and female deaths. Therefore, there is no discrepancy with "reality".
if this is so simple, why don t you simply explain, how and why all those men in the "mainstreet zone" get killed, while their women, spending MORE TIME there are left unharmed.
there might be possibilities, but they are not very likely.
Just to be crystal clear: This requires simply a list (or even simpler, a Google map with lines) identifying the main streets and cross streets used in the Si pool, from which the actual sample selection was then drawn. Then everything not included, is So by definition.
i love it, when people are giving homework to others. i love it even more, when you know in advance, that whatever result you will get, they wont accept it.
if i was part of the lancet team, the last thing on my mind would be coloring some maps for you...
so here is my counter offer:
why don t you provide fully colored maps of all towns in Iraq, that would give a n=10 number?
what you and others seem to miss is this: even the smallest village has (a) mainstreet(s). there isn t one "mainstreet concept" that will fit all places of Iraq.
you would also need population density data. in a country, that can t account for its total population...
again my advice: waste some serious research time on this (6 months and some money?) and you will learn about futility...
with childish comments like "Media Street Bias", and others, you guys' thoughts are worthless.
media street bias of course is real. sunni areas? sadr city?
in a place with streets without names, journalists will move events to the closets landmark.
sorry, but just because you don t understand it, its neither childish nor worthless.
it is simply bizarre: people have doubts about tiny details in the Lancet paper, but accept the IBC numbers as a gospel truth.
'Media Street Bias' ... worthless
So an explicit, albeit contested sampling schema is debated for some 246 comments in this blog, and yet we are to qualify this with a correction factor based upon a BBC map of media reported deaths. Whilst 'Media Street Bias' might be considered a childish comment, perhaps we could have some explanation of how the media's sampling protocol is more robust
?
Tim: I see you more of a Martin Luther myself, so Shone and I will just have to agree to disagree on this.
By the way, if you still think you criticism of Johnson et al (2008) stands, then you ought to write it up clearly (shouldn't take more than a couple of pages), post it and start a new thread. You can hardly expect the authors to bother with this endless thread, especially since the criticisms you make in later entries (like #55) have nothing to do with the criticism you make in the initial post.
Sod's "...tiny details in the Lancet paper..."
Tiny details?? TINY details?? The non-random sampling (i.e. bias toward cross streets with higher numbers of car bombs, market bombs, sniper and convoy attacks) is a tiny detail???
Idiot.
David, of course I think my criticisms of the MSB paper stand. I'm not sure what I wrote that might make you think otherwise. I will collect things together and make a new post, but I want to draw some maps so it might take a little while.
Tim wrote: "...I want to draw some maps..."
Without knowing the main streets?? Get this everyone: Tim is going to draw conclusions (with the undoubted predetermined goal of criticising JPR) based on MAPS in which he will ASSUME he knows the L2 list of possible MAIN STREETS!!
This is priceless....
Tim: don't you realise by know, that to go beyond JPR you need to know about Si space? In other words, unless you know something about what MAIN STREETS were included, and hence what cross streets were samplable, your time is completely wasted on this.
I repeat: you need to know what is a MAIN STREET in Baghdad etc. How many are there, therefore what type of streets are cross streets. etc. etc. Then you can start....
Anything you do is, a priori, invalidated without knowing this -- it is immediately worthless. So do us all a favour, and ask the L2 authors about their samplable space Si???
Tim wrote "...of course I think my criticisms of the MSB paper stand..."
Note the use of 'of course': In other words, *irrespective* of the fact that the ensuing discussion debunked Tim's JPR criticism, he will (and always will) blindly criticize JPR. Basically, it ticks him off that JPR makes a valid point, and that they might indeed be right....
I agree with Tell, Clearly it would be totally impossible to do any kind of analysis without this detail. Certainly picking one's own main street or several and comparing with a grid based random selection or some such wouldn't in any way set any kind of bound for this problem.
For what it is worth, there appears to be an infestation of Roshes here, or perhaps that is Roshii, in any case a Lott.
Eli and several others here were involved years ago in a very long thread on USENET about McIntyre and McKitrick's complaints over the Mann, Bradley and Hughes papers. One of the energetic M&M defenders was a certain Nigel Persaud, who, it later turned out, was McIntyre and who in disguise showed up a few other places including Deltoid. Now far be it from Eli to moan about pseudonyms, but in that case, and the bunny strongly suspects this, there are Mary Roshii amongst us. On re-reading the old thread, it is obvious that a lot more progress could have been made if Nigel had stood up, simply because he was defending his own work and if he had not had to pretend that he didn't know about a bunch of stuff we could have all contributed more. FWIW
The bunnies can all now play pin the tail on the Rosh
You don't have to have the same list of main streets, you just have to have a list of main streets. You divide them up into subsets, run the calculation on the subsets and compare the ensemble result. Surely one does not have to tell a condensed matter physicist that?
Since the âgender argumentâ has cropped up a lot in this thread, it may be worthwhile to try to wrap it up here. The crux of the matter is that the assumptions made by Johnson et al. do have significant implications for the distribution of deaths. In what follows let âmenâ be shorthand for working-age males (of whom there are 2 per household of 7) and âothersâ be shorthand for women, children and old men.
To avoid too many messy fractions, let the population, N=154 (each one standing for a little under 0.2m Iraqis). On the assumptions made by Johnson et al. this population is divided up (by residence) as follows: 4 men in Si; 40 men in So; 10 others in Si; 100 others in So. Since men spend a quarter of their time outside their own zones, while others spend all their time in their own zones, the average numbers in the zones are: 13 men in Si; 31 men in So; 10 others in Si; 100 others in So.
With the population divided into men and others, we have four probabilities to consider instead of the two (qo and qi) in the aggregate model. To avoid messing with subscripts, let the probabilities of death be as follows: for men in Si: w; for men in So: x; for others in Si: y; for others in So: z.
It turns out that we are quite restricted in our choice of these probabilities. What looks like a four-dimensional space actually has only one dimension. Firstly, the probability of death of randomly selected unisex individuals in Si has to be q times the corresponding probability in So. Hence:
(13/23)w + (10/23)y = q[(31/131)x + (100/131)z]
A second equation ties the (w,x,y,z) vector down further. The expected number of deaths among residents of the survey space Si is 3w + x + 10y. When scaled up by a factor of 11 to provide the Burnham (L2) estimate of deaths for the whole population this becomes 33w + 11x + 110y. This corresponds to the numerator in Eq (4) of the Johnson et al. paper. The denominator is the expected (actual) total deaths: 13w + 31x + 10y +100z. The ratio of these two numbers is the bias, R, hence:
33w + 11x + 110y = R(13w + 31x + 10y + 100z)
A third equation is given by the fact that the L2 estimate of deaths is about 2.3% of the Iraqi population so that:
33w + 11x + 110y = 154(0.023) = 3.542.
With three equations to tie down the four unknown probabilities, we have some flexibility in choosing (w,x,y,z) but we may not have a lot, bearing in mind that being probabilities they must be in the interval [0,1]. Do the values proposed by Johnson et al. for their parameters, and hence for R, leave us with any plausible-looking probabilities? The handiest way to summarise the options is to insert their proposed values (q=5 & R= 2.9512) into the above equations and express w, x, y and z as functions of t, where 0 < t < 1. For the sake of readability Iâve multiplied all coeeficients by 1,000 so these are the expressions for expected deaths per thousand:
Men while in the survey space Si: w(t) = 25.968 â 2.062t.
Men while in So, outside the survey space: x(t) = 20.617 â 20.616t.
Others in the survey space Si: y(t) = 22.348 + 2.68t.
Others in So, outside the survey space: z(t) = 6.391t.
WARNING: I am very prone to errors in calculations of this kind, as Iâve already demonstrated a couple of times in this thread. Iâve checked this a couple of times but basically itâs worth what you paid for it.
Kevin Donoghue writes:
>My theory: ozzy = ron
Correct!
But don't stop there.
Go figure. I thought Ozzy was somebody else.
I was thinking: Ron = Lancet Debunker = Tell
By the way.. Kevin, you might want to repost #256 in the new thread, since this one has dropped below the fold.
Regards,
Bruce
>I was thinking: Ron = Lancet Debunker = Tell
Also correct.
Keep going.