Deltoid

David Kane seems to be asking how the variance of a ratio can be small, when both the variance of the numerator and the denominator are big. Say we have r = X(post)/X(pre). Can r be estimated precisely, even if X(pre) and X(post) are estimated with considerable error?

Kane assumes that the answer is no, but this assumption is wrong. When X(pre) and X(post) are highly correlated, the ratio of their means can be estimated precisely, even if the individual means can't be.

Posted by: Ragout | July 24, 2007 10:34 PM

Kane,

It seems that you are mistaken regarding some basic points of statistics. Before we clarify those points it would be hard to get into the specifics of your argument.

The original analysis was done in a frequentist framework. When carrying out such an analysis, there is no distribution of an unknown parameter (such as CMRpost) to talk about, since the parameter is not assumed to be a random variable.

Therefore, statements such as P(CMRpost < 3.7) = this-or-that, which are liberally strewn about you paper, are completely meaningless.

Posted by: Sortition | July 24, 2007 11:44 PM

David,

Your simulations prove nothing because they assume the data are normally distributed. Given the inclusion of Falluja, the data are wildly nonnormal (nor are they unimodal).

The annual mortality rate per 1000 in post-war Fallujah was 100-200 above the Iraq-wide mean of 12.6. If the data were normal, mortality rates of -100 and more should be equally likely. But of course, negative mortality rates are not possible, and hugely negative mortality rates are a serious problem with your method.

To say this another way, if the data were really distributed normally with mean 12.3 and SD (12.3-1.4)/1.96, then the odds of observing a Fallujah in a sample of 33 clusters should be infinitesimal. Since we do observe Fallujah, and we think that this is a typical sample (as you argue), it follows that the assumption of normality is wildly wrong.

Posted by: Ragout | July 25, 2007 1:06 AM

1) Sortition, I don't know how to say this any more clearly: The exact same proof applies if you are a Frequentist. Since the authors themselves never describe their results this way (see Burnham's quote), I don't see much point in using the mind game of repeated samples. If there is a particular part of the proof that you don't understand, be specific. I am eager for feedback.

2) Ragout. It is not I who assume that the confidence intervals for pre- and post-war mortality are normally distributed. It is the authors. Now, they may be right to make this assumption. They may be wrong. But it is their assumption or, rather, the assumption of the statistical software that they use.

3) But even if you wanted to use the data from Falluja to reject a normal distribution (a reasonable thing to do), any fatter tailed distribution (say, t), makes the problem worse not better. The intuition should be obvious. There fatter the tails of the distribution for post-war mortality, the more likely it is that post-war mortality is below 3.7.

4) None of this is to say that there isn't a way to change the model to get the answers that the authors want to get. If you assume that the distribution of post-war mortality is right-skewed (i.e., that only big increases in death are possible, not big decreases), then you ought to be able to somehow get the 1.6 -- 4.2 confidence interval for the relative risk. But that is not the model that the authors used. (See their paper for the exact description.) Therefore, they must publish a correction, or withdraw the paper.

5) Note that I would not spend too much time on that exercise. If you try to create a new model, you would need the raw data to estimate it. The authors now report that the data is no longer "available." I am trying to get clarification on just what they mean by that, but my sense is that the data is gone. (The individual level data, not the aggregate data that Tim kindly posted.)

6) By the way, that thread from Tim's posting of the data brings back lots of fond memories, eh? I miss Seixon! And hope that dsquared chimes in on this topic. But note that BrendanH was the first (?) to suggest that the relative risk confidence interval must be wrong if Falluja is included. See his full results here. I am not sure how to translate between what he has done and what I did, but it seems like we agree that, if you include Falluja, you can't reject the null hypothesis that mortality in Iraq is unchanged.

Posted by: David Kane | July 25, 2007 7:58 AM

Erratum to the above after checking my own posts on IV estimation - the ratio of two normal distributions is Cauchy distributed, and there aren't any non-trivial cases where the ratio is exactly normal.

Posted by: dsquared | July 25, 2007 8:25 AM

Just to restate what others have said (basically, so if it comes to vote counting, I've expressed a vote): if we start with a population of Iraqi districts post-invasion, we can estimate death rates (although almost noby seems interested in doing so). We can, if we choose, say they are all in the same population (located inside Iraq), and note there is a huge variation in death rates. Or, we can note that data correlates with political fact, and that a single district, involved in high-intensity war has a higher death rate. If we restrict our population to districts that are not involved in high-intensity war, we get a more homogeneous population, covering most of the country at that point, and have sampled one site which is politically and statistically distinct. I come across the same work in my field. I think it's much more useful to have estimates of a specified population (say, weights of adults) than to have data that's not so easily categorized (weights of humans, including fetuses, 3 week-olds, microcars filled with midgets, etc.). Kane's argument is made. Personally, I think it's a lousy one, with a poorly specified universe.

Posted by: stewart | July 25, 2007 8:27 AM

Thanks to dsquared for his comments. High quality discussion like this is precisely why I asked Tim to post the paper at Deltoid. To substance:

1) I am not assuming anything about the distribution of the ratio of two normal variables. Although the paper has details, the basic trick is that, if pre- and post-invasion CMR is normally distributed, then the difference between the two is normally distributed. I show that the distribution of this difference overlaps significantly with zero. If that is true, then the lower bound of the relative risk confidence interval is too high.

2) Yet dsquared's comment raises another possible approach. Assuming zero correlation between the estimates, we can easily simulate the RR risk using the given CMRs. For R users, this would be:

quantile(rnorm(mean = 12.3, sd = (12.3 - 1.4)/1.96, n = 10000) / rnorm(mean = 5, sd = (5 - 3.7)/1.96, n = 10000), c(0.025, 0.5, 0.975)) 2.5% 50% 97.5% 0.3032679 2.4624131 5.0236085

Note that the mean is what the paper gives us. The upper bound is high, but not by much. The inconsistency is, as in my paper, with the lower bound.

Now, obviously, this approach is not how the original paper estimates the RR. But it is further circumstantial evidence that, if the CMRs are correct (and I am pretty sure that they are), the RR risk must be wrong.

3) As to Jeff Harvey's charming question, the more that I study this topics, the more convinced I am too trust Jon Pedersen's judgment: 100,000 excess violent deaths.

Posted by: David Kane | July 25, 2007 9:41 AM

David, thanks for your answer. In other words, mass carnage and slaughter. A vast crime against humanity, for which the occupiers are obliged to pay reparations. Moreover, I suppose this also means that you support the idea of convening war crimes trials for the various leaders and their cronies involved in this debacle e.g. Bush, Cheney, Rumsfeld, Blair, Straw, Hoon, Berlisconi, Aznar etc.

Posted by: Jeff Harvey | July 25, 2007 10:20 AM

David, you don't explain why you think the CMR's are correct. Yeah, outliers increase variance (I'm so well read I actually knew that). Robert (who I hope shows up here) did his own analysis of the data and adding Fallujah increased the CI, but it also shifted the whole thing to the right, as intuition would suggest and the probability distribution he got for the excess deaths has several peaks, which I assume has something to do with how often the Fallujah cluster pops up in resamplings. Speaking as the ignorant layperson here, if I saw the result the L1 team got with the CMR including Fallujah, I'd suspect the software package was employing an inappropriate mathematical model. Like dsquared, it's not clear to me that finding a bombed-out place like Fallujah suggests the existence of other places where mass resurrections occurred.

SG, I tried that argument myself, in modified form. What if Bush had dropped a nuke on Fallujah, killing 200 in that cluster? Think of the variance and just imagine how much this would increase the probability that the invasion of Iraq lowered death rates. It didn't seem to strike David as a reductio ad absurdum like I hoped.

Anyway, in the interest of furthering my statistical education, can someone concoct a nice simple example we could do by hand which would show how adding a very high mortality cluster to a sample of low mortality clusters would widen the probability distribution so much that there is now a greater chance of lower death rates than before? I'd like to see this if it can be done. I hate computer statistical packages which give wildly nonintuitive results if you can't follow the details. Please type slowly, with lots of explanation, so I can follow. Merely telling me that outliers increase variance is unsatisfying. Or ignore me--I'm just trying to get free tutoring lessons.

I'll start. Suppose you have two clusters with mortality rates of 0 and 1. The average mortality rate is 0.5 and if you resample you get 1 run of 0,0, two runs of 0,1 and 1,0 and one run of 1,1. One out of the four runs gives an average mortality less than 0.5. Now suppose you add a third cluster with a mortality rate of 100. If I understand bootstrapping and how you'd use it to get average mortality rates, you resample with replacement 3 times because you have 3 clusters, average the mortality rate for the three clusters, then do it again over and over. In this case there's only 27 possible resamplings. And only 4 of the 27 give you an average mortality rate below 0.5. Only 1/27 gives you an average mortality rate of 0. The distribution of average mortality rates has shifted to the right. It's a lot wider, but it behaves the way I intuitively expect. I don't find an increased chance of getting an average mortality rate less than 0.5 just because I added an enormous outlier.

So can someone concoct a simple example that behaves in a surprising way, so that I could understand how adding a Fallujah-like outlier would make a rational person think "Huh, I guess the invasion might have lowered death rates after all."

Posted by: Donald Johnson | July 25, 2007 11:18 AM

Ragout's point is the correct one by the way - once you have assumed that a massively bimodal dataset has a unimodal distribution, you are never going to get a meaningful answer. The Figure 2 diagram actually has 1.34% of its probability mass represented by states in which Iraqis were being resurrected from the dead, and the idea that you can correct for this problem by simply truncating the distribution at zero is clearly wrong; the very large outlier on the positive side isn't a reason to believe in a lot of non-outlier cases on the negative side.

Hang on ...

David, what are you going on about here talking about the authors assuming normal distributions? A quick read back of Roberts et al (2004) reveals, at the bottom of p3, that

"As a check, we also used bootstrapping to obtain a non-parametric confidence interval under the assumption that the clusters were exchangeable. The confidence intervals reported are those obtained by bootstrapping. The numbers of excess deaths (attributable rates) were estimated by the same method, using linear rather than log-linear regression."

So basically your analysis is completely tangential. Roberts et al got their confidence intervals by bootstrapping from the empirical distribution of their data. The empirical distribution of the data wasn't normal and it wasn't nearly normal.

David, what's your source for your remark to Ragout above that:

It is not I who assume that the confidence intervals for pre- and post-war mortality are normally distributed. It is the authors. Now, they may be right to make this assumption. They may be wrong. But it is their assumption or, rather, the assumption of the statistical software that they use.

It seems to be flatly contradicted by the passage from p3 quoted above and I really can't find any other places in the text where they talk about a normal distribution of the noise (in fact, the terms "Normal" and "Gaussian" don't appear in the paper at all). Is this from private correspondence?

At present it seems to me that when you say:

"If you assume that the distribution of post-war mortality is right-skewed (i.e., that only big increases in death are possible, not big decreases), then you ought to be able to somehow get the 1.6 -- 4.2 confidence interval for the relative risk. But that is not the model that the authors used. (See their paper for the exact description.) Therefore, they must publish a correction, or withdraw the paper."

then you're wrong - they used a model which only had big increases, because the "model" was a resampling of the actual observations, which of course contained Fallujah but did not contain "anti-Fallujah".

Posted by: dsquared | July 25, 2007 11:32 AM

Thanks to dsquared for these comments. Initial thoughts:

1) It seems clear to me that the authors use two totally different methods for calculating confidence intervals. The passage that dsquared quotes from applies only to calculations for the relative risk. The paper is not completely clear on this point, but the full paragraph implies this.

2) It seems certain to me that the confidence intervals for the estimates of crude mortality rates assume a normal distribution. I quote the full paragraph in the paper. Besides the software clues, the key fact is that the authors provide an estimate of the design effect. This is a natural product of the usual normal distribution approach. How would you get a design effect if you were calculating the confidence intervals for CMR using a bootstrap? I don't think you can, but counter-examples are welcome.

3) Goodness knows that this discussion would be a lot more productive if the authors were to release their code. Elizabeth Johnson, the graduate student who actually did the calculations, does not respond to my e-mails or phone calls.

4) The other way that I can be certain (?) that dsquared is wrong in his inference that the authors used a bootstrap or other empirical method to calculate the confidence intervals for post-invasion CMR is that, if they had, the confidence interval would not have been symmetric. But it is. So, almost certainly, we are dealing with a normal distribution.

5) All of which does no prevent SG (or anyone else) from using a skewed distribution for modeling post-war mortality. Go ahead. But that is not what the authors did.

Further comments sought.

Posted by: David Kane | July 25, 2007 11:50 AM

Kane,

You made a fundamental mistake - you are using meaningless terms, symbols and arguments (namely, all those that refer to distributions of CMRpre, CMRpost and RR).

You do not attempt to dispute that, yet you still expect your "proofs" to be accepted as valid. This is absurd. Either come up with valid proofs or abandon your position.

Posted by: Sortition | July 25, 2007 12:09 PM

Let me summarize again. Roberts et al estimate pre- and post- CMR using the mean, and calculate confidence intervals under the assumption that the means are approximately normally distributed. David notes that the confidence interval of the difference in these estimators includes zero, assuming normality. Then, Roberts et al estimate the relative risk, using some other method, and get a confidence interval that does not include one.

David claims that these findings are inconsistent, but this is true only in a trivial sense. Roberts et al calculated the CIs for the means under one assumption, and the CIs for the RRs under another assumption. In particular, in calculating the RRs, they modeled the log of the data, while when calculating the mean CMRs, they used the levels of the data. In addition, they took into account the correlation of the data when calculating the RRs. Finally, they reported bootstrapped SEs for the RRs, rather than assuming normality. So it is not at all surprising that CIs for the RRs and the CIs for the difference in means would give different answers.

By the way, one problem here might be that the description of RR estimator in the Roberts et al paper is totally opaque. They mention overdispersion, so I assume they used a negative binomial regression. It does seem clear, though, that they modeled log deaths, took into account the correlation over time in deaths, and reported bootstrapped SEs for the RR.

Posted by: Ragout | July 25, 2007 12:19 PM

Thanks for this very helpful feedback.

1) Ragout is exactly correct when he writes:

Let me summarize again. Roberts et al estimate pre- and post- CMR using the mean, and calculate confidence intervals under the assumption that the means are approximately normally distributed. David notes that the confidence interval of the difference in these estimators includes zero, assuming normality. Then, Roberts et al estimate the relative risk, using some other method, and get a confidence interval that does not include one.

Correct! Part of what we are arguing about above (like the normal assumption) would refute this summary, but I believe that ragout has it right. This is certainly what I mean to argue. Ragout goes on:

By the way, one problem here might be that the description of RR estimator in the Roberts et al paper is totally opaque.

Correct! Although I find ragout's guess about what they did reasonable, neither he nor I (nor any of you) know the exact procedure used for estimating the RR, although it seems reasonable to assume that a bootstrap was used.

All of which raises an interesting point. You don't need to use a bootstrap to calculate an RR in this (very standard) cluster set-up. So, why did they? What would the result have looked like if they just used the usual approach, as BrendanH did? My guess is that they would get the same (not statistically significant) answers that Brendan got. Since they didn't like those answers, they went searching for a model that would produce the answers that they wanted. A bootstrap did. Once they found that, they did not bother to check that the RR results were inconsistent with the CMR results.

Of course, this is all pure speculation, but there is no reason to use a bootstrap for the RR and not for the CMR.

ragout claims that my claims are true only in a "trivial sense." Perhaps! No author can fairly judge the importance of his own work. My preferred analogy is to scales and weighing. Imagine that the Lancet authors had reported that using scale A, each of two bags of apples weighs 2 pounds. Using scale B, those same two bags of apples weigh five pounds together. I then assert that, since 2 + 2 != 5, the conclusion is wrong. ragout says, "Well, they told you that they used two different scales. Your claim is trivial." Perhaps. But a scientific paper needs some minimal amount of internal consistency. You can't assert that 2 + 2 = 5 and then blame your scales. The scales are your responsibility. Why do these scales give mathematically inconsistent answers?

Posted by: David Kane | July 25, 2007 2:06 PM

David, there are two problems with a normality assumption. One is the mass at negative mortality as you say.

The other -- and bigger -- problem I see with normality is the Falluja outlier which is wildly improbable under a normal assumption. And this outlier doesn't come out of left field. You expect to see positive spikes in violent mortality in a war zone. (See also SG's comments about WWII above.)

So it's good to hear you did work with truncated normals to address the thin left tail, but I would expect the thick right tail to be the far bigger issue.

Posted by: David Kane's friend | July 25, 2007 2:55 PM

My friend notes:

So it's good to hear you did work with truncated normals to address the thin left tail, but I would expect the thick right tail to be the far bigger issue.

Not for my purposes. It does not matter to me what the mean post-war CMR is, nor how fat the tail to the right. All that I need to show concerns the lower confidence interval. I am only calculating the probability that CMRpost < CMRpre. For that calculation, all that (really) matters is the bottom percentiles for CMR_post (its 2.5th percentile and its mass below CMR = 3.7). The right hand tail (whether short or long or somewhere in between) does not matter for my proof. It is irrelevant.

Now, of course, the bigger a tail you place on the right hand side, the less mass there is to go elsewhere, including the left. I do require that the 1.4 lower CI is correct, as presented in the paper. Again, no one has shown any reason (meaning a specific alternate model) to doubt that. Roberts et al made a mistake, but not in estimating CMR.

Posted by: David Kane | July 25, 2007 3:17 PM

My entire paper assumes Falluja is included and only addresses the claims made by Roberts et al which also include Falluja. Now, the reason that the CI for post-war (versus pre-war) is so much larger is because the estimate for Falluja post-war is so different (bigger) than those for other clusters. Falluja pre-war, on the other hand, looks a lot like other clusters. You write:

The point is -- because of the Falluja data point -- the far right tail is very important in determining the lower CI.

I agree that this is possible in theory, but I have to see a concrete demonstration. Pick any right skewed non-negative distribution that you like which a) Has a mean around 12 (which will be needed to match the mean of the data) and b) has lots of mass well between 5-12 (because lots of clusters are not nearly that violent) and c) has a tail which stretches out to easily include Falluja and d) does not assume the conclusion by having zero mass below 5.

I do not think that there is an distribution which does this very well, or any better (in aggregate) than the normal distribution that the authors use. Suggestions welcome.

Once we have such a distribution, we can estimate pre and post CMRs along with their confidence intervals. We can then check to see if these are consistent with the reported RR. Perhaps it will all make sense! I doubt it. And, anyway, my point is just to demand a correction/retraction of L1 as published. It is wrong. The authors are welcome (if they can find their data) to create a new model and then publish that one instead. But they first need to admit their mistake. Or someone needs to demonstrate mine.

Posted by: David Kane | July 25, 2007 3:57 PM

Dang it, Robert, I was about to post a link to your old excess mortality graph and claim I'd done it.

Posted by: Donald Johnson | July 25, 2007 4:21 PM

Robert makes a fair point which I hope to address soon. SG suggests an interesting model which I encourage him to work out himself.

For now, I want to step back and try to claim why this arcane dispute matters. Recall that the most quoted result from the article was the 98,000 excess death estimates (with confidence interval 8,000 to 194,000). Now we all realize that this estimate excluded Falluja. Now, here is a question for Lancet defenders, especially serious folks like Tim, dsquared, Robert, BrendanH and others:

Take the exact same computer code that produces the 98,000 (CI 8,000 - 194,000) estimate. Now, don't exclude Falluja. What does the code produce?

Now, without the exact code, it is tough to know the answer to this. I argue that the authors do a lot to try to imply that the mean estimate would rise to 300,000 or so and that the confidence interval would get wider but still safely exclude zero, something like 150,000 to 600,000.

But this is, I am fairly certain, false. Using the exact same code, I bet that the lower bound would be well below -100,000. See the paper for details. Moreover, it is almost certain that the authors knew this and that they purposely organized and wrote the paper in such a way as to hide this fact, to purposely mislead readers (including smart readers like Tim and dsquared) into thinking that including Falluja would move the lower bound of the confidence interval for excess deaths up. After all, excluding Falluja is "conservative!"

Now, before diving into the weeds on this topic, I would first love to establish what smart Lancet defenders believe. So, what is the answer to my question above?

Posted by: David Kane | July 25, 2007 9:28 PM

Very cool discussion. I had classmates over there, although most of my year group is out of the field by now. Dave, you seem smart...for a Marine. ;)

Posted by: TCO | July 25, 2007 9:51 PM

SG writes:

What you are doing is claiming that a very very unusual event observed in a sample of 32 observations is a true random draw from the same distribution as the other 31 observations, when we have STRONG evidence to suspect that it is a random draw from a different distribution.

No. This is what the authors do. See their paper. They measure the post-invasion CMR using a normal distribution with the Falluja cluster treated identically to all the others. The clusters are exchangeable. If you think that this is nuts, you should bring it up with Roberts et al, or with the Lancet's peer reviewers.

Posted by: David Kane | July 25, 2007 10:10 PM

"Tossing in a big outlier can drive the lower CI limit down. Imagine 5 points; 3, 3, 3, 3, and 3. Now toss in one more point, 4. The upper confidence limit will now be higher, but the lower confidence limit will be lower."

Oh, goody, someone at my level maybe. From what I glean in this thread, it seems to depend on what sort of distribution you fit to the data. If you assume a normal distribution previously you had no variance and your bell curve was a delta function spike at 3. Add the 4 outlier and there's a nonzero variance and so there's a chance of a result less than 3.

But if you do a simple bootstrap (as I understand it anyway), that won't happen. Finding a 4 doesn't give you any reason to think there's a 2 in the vicinity. You resample from 3,3,3,3,3, and 4 and there's no way any collection of clusters you get will have an average value of less than 3. Because, you know, they are all 3 or bigger.

This whole business seems vaguely arbitrary to me, probably from not understanding it enough. But the bootstrap approach (as I understand it) seems to stick closer to the data you actually have.

And David Kane, yeah, when you brought this up a few weeks ago and I looked at the CMR with Fallujah I thought the result was nuts. Maybe someone should have objected. BTW, I thought it was 32 non-Fallujah, with Fallujah being 33. And that brings up something I once thought of. The prewar and postwar time periods are of different lengths, which complicates things, but there was 1 violent death prewar, which means one violent cluster. There were 15 clusters with violent deaths postwar, or 14 violent clusters and 1 superviolent cluster if one wants to treat Fallujah differently but I'm lazy and don't. I don't know how to correct for the time duration problem, but this alone kinda suggests an increase in violent deaths due to the invasion. Maybe you could do simple binomial statistics on clusters, classifying them as violent or nonviolent. This is about my speed. Lumping Fallujah in I get a variance of Npq = 33(15/33)(18/33) = 8.2. Sqrt 8.2 is 2.8 and I'll double that for my CI half width and you have 5.6, so my informal CI for violent cluster number out of 33 would be 9.4 to 20.6. I'm not sure what to do with the prewar number since the time duration is different, but say it counts as 2 violent clusters instead of 1, which seems generous to me. The variance would be about the same, so if that's 2 violent prewar clusters with a variance of 2, then that's 2 plus or minus 2.8 and with a maximum value of 4.8 there's no overlap with the postwar CI. I did a bad thing there--spreading my CI for the prewar violent cluster number into the negative range, when really the distribution skews right and could be approximated by the Poisson distribution, but hell, it's not going to do too much harm in this case. I think the upper end of the CI goes up slightly if you do it right. Someone who took this seriously could do it more carefully, but it seems clear the war caused an increase in the number of violent clusters. What a surprise.

Posted by: Donald Johnson | July 25, 2007 11:30 PM

For example, David:

We estimate that there were 98000 extra deaths (95% CI 8000-194 000) during the post-war period in the 97% of Iraq represented by all the clusters except Falluja.

So they have restricted their report of extra deaths to the low-risk towns outside of Fallujah, and have explicitly given a figure for the death rate in 97% of Iraq. Does this sound like it might possibly represent a covariate analysis to you? With insufficient data for the remaining 3%?

Posted by: SG | July 25, 2007 11:39 PM