Alastair Mackay on the Lancet study

Tim Lambert,
You are too kind by half in your introduction.
A small correction: readers who follow the link will note that I specifically do not claim statistical expertise--a more parsimonious explanation of why I don't discuss the statistical methodology. Actually, that's relevant to the question raised by the post, or that was meant to be: how do non-experts go about understanding and evaluating controversies where expert qualifications are important?

how do non-experts go about understanding and evaluating controversies where expert qualifications are important?

They shouldn't try. Go get a PhD in stats, then we'll talk.

AMac,

All credit to you for showing that the case against Roberts et al. can be argued in a reasonable manner. You fail to convince me when you say that the editor should have refused publication. Sure there are places (well at least one) where it might be worded better, but it is up to the required standard and that should be enough. Fortunately I don't need to elaborate because the comments of Robert McDougall at WoC say it for me, better than I could.

Go get a PhD in stats, then we'll talk.

Needless elitism. I don't have a PhD in statistics, or in anything else. I've got a business school degree with a reasonable econometrics component, and that's it. I happen to have been lucky enough that my career has pushed me down a path where I have had to deal with real, live, wild data a lot of the time. That's how you learn these things, not at graduate school.

Kevin Donoghue, dsquared, Robert McDougall as well,
Thanks for taking the time to discuss these matters cogently.
Expert Statistician,
Thanks for your comment too. As your quote suggests, I'm saying that this problem is generalizable beyond statistics; a degree in X may not be the answer. A topic for another day.

I'd be happy to leave the methodology alone, if those citing the study would confine themselves to the statistically significant results, e.g. "The study found as few as 8,000 or as many as 194,000 excess deaths as a result of the U.S. invasion of Iraq." Then at least they'd be making honest and meaningful statements.

TallDave. You are incorrigible!

AMac, I didn't say you were an expert on statistics -- I said you know something about statistics. The folks who criticize the methodology know less about statistics than you do. All of the expert statisticians consulted about the study have pronounced the methodology sound.

the fact still remains that The Lancet article claimed a high confidence interval only because its margin for error was so large. Come on - it said it was 98% confident that there were between 10,000 and 200,000 deaths. That doesn't mean you split the difference and say 100,000 people died.

Steve68: Another potential student for a statistics course.

"it said it was 98% confident that there were between 10,000 and 200,000 deaths. That doesn't mean you split the difference and say 100,000 people died."

Steve, you don't "split the difference". The mid-point of a normal curve has the highest probability. 98,000 is the midpoint of the CI in this study, therefore it has the highest probability of being the true population value.

Shirin: Do you ever get the feeling that your hitting your head against a brick wall? It seems to me that certain bloggers are never going to accept rational mathematical and statistical arguments alter their mind set. To these people one shouldonly take as "true" the lower bound of the CI, not the mid-point nor the upper bound (unless of course that showed how many people were saved by the war).

Shirin, the width of the confidence interval also guarantees that the likelihood of the "true population" being 98,000 is itself very small. As discussed in earlier threads, 88k-108k carries with it a mere 16% likelihood. Furthermore we have no idea what assumptions have been made to normalize the data.

Chorlto, yes, I do. And it is not only rational mathematical and statistical arguments they will not accept, it is any fact or reality that conflicts with what they want to believe.

Pacal, Steve appears to believe that choosing the midpoint is a simple matter of "splitting the difference". Regardless of the depth or shallowness of the curve, this is a complete fallacy.

My comment addressed your statement that "it has the highest probability of being the true population value." True per the norm-distributive assumptions of CLT, but also subject to the variability connoted by the CI (which is a critical part of the information imparted by any such figure.) The same factors that make the range ridiculously broad also make 98k rather unlikely to be even approximately correct.

Pascal, that argument has been addressed repeatedly here and elsewhere, and by any number of people far more qualified than I am. Hopefully with time there will be additional studies. In the mean time, perhaps it is because I am more aware than most of the realities in Iraq both before and after the U.S. invasion and occupation, but I do not find the results of this study terribly unrealistic.

I know how I "evaluate" the article. I look at the cover. I see that it is published in Lancet, the " world's leading independent general medical journal. The journal's coverage is international in focus and extends to all aspects of human health....The Lancet is stringently edited and peer-reviewed to ensure the scientific merit and clinical relevance of its diverse content."

I notice it is not published by Luck & Loon, hand printers to the Gaia and pagan community, but by Elsevier, a respected member of the capitalist business world. A major multinational actually.

And then I think: "By golly, I think I might trust the contents of this publication. If my previous beliefs conflict with what it says, perhaps I should change my mind."

If my life is interrupted by some ghastly brainfart and I am rushed to hospital so fantastically trained surgeons can labour for hours repairing my fragile bonce, it is extremely likely that the people who do this will be avid readers of Lancet.

The procedure to save me might even be learnt from enquiries and contact based on an article in the magazine.

I have this vision of lying in my bed while some denialist in this debate is brought in bleeding from the ears. I see them shouting and struggling, saying "No, no, don't let those doctors near me. They read Lancet and are clearly deceived."

But of course the denialists are clear minded sceptics, who would never dream of trusting their lives to that huge socialist cartel, the medical profession.

David Tiley, thanks for a great chuckle!

Sorry if this info has been posted already, but does anyone have a graphical representation of the Lancet study death toll vs. confidence level? Does it appear in the study itself?

Pop Trot,
In an Excel spreadsheet, if you put whatever number you like (between 8,000 and 196,000) in the cell A1, and set
B1=NORMDIST(A1;98000;45000;TRUE),
it will give you the value for the cumulative normal distribution. That isn't exactly right for the Lancet study but it is good enough to be going on with. Using that you can generate the elongated s-shaped curve of the cumulative distribution.

You guys can argue about methodology and statistics until you are blue in the face. The simple fact is that it is all made irrelevant by the original sample group which is far too small to give us conclusive, accurate or reliable data. The study's pre-war infant mortality rate (29 per 1000 livebirths) is derived from 8 actual infant deaths. The overall findings are derived from just 46 pre-war deaths in 14.6 months and 89 (or 142 if we count Falluja) post "invasion" deaths in 17.8 months.

All of the estimates and probabilities that stem from this data are like houses built on quicksand, they are based on a flawed foundation. What we need are larger studies with more sample groups and more data points. A lot more. And the study should be conducted by an independent body and the methodology agreed upon beforehand.

---Ray D.

By the way, if you guys all know so much, then explain exactly how the study came up with the 98,000 figure using the data from the 32 clusters excluding Falluja.

This should be entertaining...I wouldn't be surprised if none of you actually know exactly how they did it or what you are even really talking about.

I'll even give you a hint: It was based on 32 regression lines with a boot-strapped confidence interval.

---Ray D.

Ray D, everyone except the US and UK governments agrees on the need for a larger-scale study.

By the way, if you guys all know so much, then explain exactly how the study came up with the 98,000 figure using the data from the 32 clusters excluding Falluja.

Is this a genuine request for information, or are you just acting the ass? I suspect the second.

For what it's worth, they fitted a generalised linear model to each cluster's pre- and post-invasion death rate. They created a likelihood function for the data they observed relative to an assumed multiplicative (or additive in logs) risk factor, and picked the risk factor which maximised the (log-) likelihood function. That's one model, by the way, taking the 32 ex-Fallujah clusters as data points, not "32 regression lines" (in fact there was no "regression" in the sense of least squares as the model was estimated by conditional ML; the word "regression" in the paper refers to the assumed underlying model, not the fit).
Then they bootstrapped the confidence intervals by taking each cluster's pre- and post-invasion death rates and shuffling them around (attaching one cluster's post-invasion death rate to another cluster's pre-invasion death rate to create a pretend cluster) and re-estimating the model. Repeat this enough times and you get a bootstrapped distribution of "risk factors" for the various pretend datasets. Take the 5th and 95th percentiles of that distribution and call them your confidence interval.

I daresay I couldn't replicate this study as simple as that; I'd have to bone up on conditional maximum likelihood because my experience is in time series econometrics, not panels and cohort studies. But it is entirely clear from the paper how the estimation was carried out, and I suspect that your question is more an attempt to intimidate than to gain information. Do please be very sure that this won't work, and do please be very sure that if you attempt to come back picking nits in my explanation with the intention of trying to pretend I am bluffing my statistical knowledge, you won't get away with it.
Now here's one for you! On the basis of what argument from sampling theory do you claim that 46 prewar deaths and 89 postwar deaths are "too few" to give "conclusive accurate or reliable data"? Show your workings. Dickhead.

I'll even give you a hint: It was based on 32 regression lines with a boot-strapped confidence interval.

LOL. Beautiful. I eagerly await Tim Lambert's (or "liberal"'s or "Shirin's") disguisition on bootstrapping a confidence interval. Maybe dquared, MBA can pitch in (no using excel, though dsquared, that's rookie stuff!)

(btw, sorry for that epithet, Tim; I thought it was warranted under the circumstances)

David Tiley,

That's an "argument by authority," generally not convincing in a logical sense.

While Lancet is certainly prestigious, a study is not rendered infallible by virtue of having the name "Lancet" stamped on it. I'm sure we could find a long list of Lancet studies that turned out to be wrong, as we could with any major journal that's been around for awhile.

"there is no very good way to make general statements about percentile confidence levels of this sort of distribution"

-dsquared

How useful is all that bootstrapping again dsquared?

dsquared, eager to hear your estimate of the study's confidence interval net of Fallujah. Your expertise in bootstrapping and time series analysis of financial data should help you out there quite a bit. Tim Lambert gave it a whack some time ago and flunked the course. Want to have a go?

There's a point at which people who don't want to contribute to a discussion should be ignored, and a point at which they should be excluded. I leave it to the reader to determine where Pascal is.

Pascal, you are way out of your depth. Give it a rest. This is your only warning.

Pascal,

When you say net you mean gross, don't you?

dsquared,

Thanks for that, I'm sure you're right. When I last ran regressions, many decades ago, computers were in big air-conditioned rooms guarded by the college IT specialists. If techniques like bootstrapping had been available they would have suspected us of playing games (how else would we be burning up so much CPU time?) and kicked us out. So we just ran regressions and reported that the residuals looked kinda funny, further research obviously required.

Tim, the ridiculous flourishing of credentials among you guys is really disgraceful and embarrassing (especially irrelevant credentials like econometrics, in which I have background as well, but so what?) dquared has himself admitted the dubiousness of the study's confidence interval, settling instead for the tepid conclusion that "it doesn't include zero." Yet he (and you) confidently return to the mean value in the study as if it is a safe baseline, which it isn't.

If the term bootstrapping upsets you Pascal, please explain how you would go about analytically determining the CI. Thanks!

Pascal, you're taking quotes out of context, which makes me think that you don't know what the context is. My remark about the impossibility of constructing a reliable confidence interval referred to the data including the Fallujah cluster.
As you presumably know if you've done an econometrics course recently (or if you read my post and understood it, or for that matter closely read the study), the bootstrapped confidence intervals are constructed by a process that assumes the clusters are interchangeable. The Fallujah cluster isn't interchangeable with any of the others. Hence, you can bootstrap estimates with the non-Fallujah clusters, but not with the full dataset.
Kevin: it really is amazing what you can do these days. I used to be on the mailing list for a program called BUGS, before I realised I was not understanding a single word they were talking about, and there is basically an absolute revolution coming in applied work because of computer power getting so cheap; basically, Bayesian analysis requires loads of numerical integration which used to be ludicrously difficult and approximate, but now that you can just run 10,000 iterations of a Monte Carlo process any time you feel like it, Bayesian econometrics is the new hot topic.

Rob, I'm not upset by the term or the procedure of bootstrapping. I am aware that deriving a confidence interval for any data set as disparate as the one under discussion yields an uncomfortably wide confidence interval no matter what method is used. As neither I nor 99.99% of the public at large has access to an EpiInfo workstation, I asked Les Roberts what the confidence interval might look like were the discarded Fallujah cluster included. I thought this a fair question since a confidence interval had been expressed for "risk of death" including this figure, as well as several other measures.

Instead of answering with a hard figure or range, this is what he wrote me: "[N]ot giving a number or range that could have been discredited and undermined the validity of the study seemed then and now like better service to the Iraqis." Richard Garfield wrote "because it would be very wide it was not calculated." Do these sound like good answers to you?

The question of confidence interval width is germane not only to the "lowest statistically significant number" that everyone has seen fit to ignore altogether. It also addresses the likelihood of the central figure itself. The wider the confidence interval, the less likely is any individual point estimate. The CI attached to 88-108k is only 16%. So claiming this is "quite likely" or "very likely" or anything like that is statistically and colloquially FALSE.

It also addresses the likelihood of the central figure itself. The wider the confidence interval, the less likely is any individual point estimate. The CI attached to 88-108k is only 16%

Less of a debating point than you apparently think it is, and an absolutely textbook example of "Kaplan's fallacy" - you are trying to pretend that uncertainty about the point estimate is the same thing as reason to believe that the true number is lower than the point estimate. Whatever the confidence interval, the probability that the true number is more than 98k is 50%; if I believe your confidence interval figures (no idea why I would), I can say with 58% confidence that the true figure is greater than 88k.

I said nothing whatsoever about lower or higher (although I notice that you seem exceedingly reluctant to discuss the unaugmented lower bound of the 95% CI.)

If Kaplan's fallacy is "the confidence interval is very wide and therefore the conclusions of the study are quite imprecise," it isn't a fallacy, it's the plain truth. No one is claiming that 100k is exactly as likely as 25k or 5k -- that's your straw man. 88-108kk is more likely than 15-35k, but not a whole lot more likely. Yet the claim is made again and again that 100k "is most likely correct" and you don't seem to object at all to this casual abuse of statistical language. Most other people have bothered to inquire after this interval and its meaning, and rejected the study as too imprecise to be valuable.

88-108k is more likely than 15-35k, but not a whole lot more likely

Yes it is. Using NORMDIST in Excel, I get 17.5% probability for 88-100k and 4.8% for 15-35k - the bootstrapped probabilities might be a bit different, but not a factor of four. Who on earth did you think was going to believe that a 20k interval constructed around the mean was going to have the same probability as one out in the left tail?
Oh yeh, faker, 100k is exactly as likely as 25k or 5k because any point value on a continuous distribution has probability zero. Hahahaha, see how silly you look now.

dsquared:

Oh yeh, faker, 100k is exactly as likely as 25k or 5k because any point value on a continuous distribution has probability zero. Hahahaha, see how silly you look now.

Nobody could be that silly, right? Wrong

@ dsquared:

My, my you are getting a little testy aren't you. Maybe the problem is you have no answer for my arguments here or at Chicago Boyz. I know it hurts to get your backside kicked in an argument, but calling me names only reveals how intellectually bankrupt you are.

By the way, this quote stems from Les Roberts in an email:

"the 98,000 was based on 32 regression lines with a
boot-strapped confidence interval"

And, oh yeah, here is why the sample group is too small:

Even if you had 10 more deaths in either the pre-war or "post-invasion" data, it would change your final estimate by approximately plus or minus 25,000 to 30,000. So even if just a few families were disproportionately hard hit by bombings, etc. it would throw the results of dramatically.

Clear enough honeybunch?

---Ray D.

Chicago Boyz debate here

BTW my NORMDIST gives me 16.5 for 88-108 vs. 4.8% for 15-35. I'm pretty sure you're doing something wrong if NORMDIST is giving you 17.5% for an interval only 12k deaths wide. Was that a typo?

I'm still eager to hear your results for the study's confidence interval inclusive of the Fallujah outlier. Tim "2,000-14,000" Lambert earlier provided us with a cuff that involved the risk of death CI, but unfortunately he turned out not to know what he was talking about. If you help me figure out what the CI would be including the Fallujah sample, I would really be grateful, and I'm not being sarcastic.

In order to keep the discussion civil, I have deleted comments from Ray and Pascal They can resubmit them without the name calling.

Ray, you keep making arguments that show you don't understand basic statistics. Yes, if you had 10 more deaths in the sample it would change the estimate. How likely is that to happen? That's what the CI is telling you.

Your claim that I don't know what I'm talking about is false, as is your claim that I somehow got the risk of death CI wrong. I recommend that you read the study.

Tim Lambert wrote:
"All of the expert statisticians consulted about the study have pronounced the methodology sound. "
This is not true. Emeritus Professor John Brignell of the University of Southampton specialises in the analysis of signals from sensors- a maths and statistics intensive area. By any rational criterion, he is an expert statistician, and Tim knows that he is dismissive of this study.
Perhaps "everyone who agrees with this study thinks the methodology is sound".
yours
per

The substance of my comment was that of course, 16.5% is not "the same probability" as 4.8%, but they are not so far apart as to recommend the certainty of the one versus the other. I never claimed they were the same probability, as Tim and dsquared well understands, each being exceedingly brilliant and perceptive.

Brignell is not an expert statistician. He is a crank who believes that the entire field of epidemiology is a scam. That he dismisses this study doesn't tell us anything useful since he dismisses all such studies.
See here.
Note the extensive comments from per/not creatively snipping/David Bell in that thread.

Pascal, a 67% CI goes from 50,000 to 150,000. It is more likely than not that the number is 100,000 (to one significant figure).

@ Tim,

So are you going to delete dsquared's comment or is your comment policy one-sided to favor those who share your opinion?

---Ray D.

Tim, you write:

"Ray, you keep making arguments that show you don't understand basic statistics. Yes, if you had 10 more deaths in the sample it would change the estimate. How likely is that to happen?"

Well, how likely is it out of 988 households that two or three might be the victim of a particularly bad bombing, terror attack, shooting, disease, etc. in Iraq? Because that is all it would have taken to throw the results significantly off. It doesn't seem unlikely to me at all. But since you are the big self-proclaimed expert on probability and statistics, why don't you tell us exactly how likely/unlikely it is.

---Ray D.

No, I'm not deleting dsquared's comment. He gave a really good explanation of bootstrapping. He should not have included the insult, but if you ever come up with a comment that contains as much information, you can throw in an insult for dsquared. If you don't think that is fair, go away.
And if you don't know the answer to "how likely is that to happen?", you didn't understand what he wrote.

Alright Tim, I will leave you alone here in the echo chamber since it seems that is what you really want.

---Ray D.

I think Ray that he probably wants intelligent discussion, not the parading of ideological prejudice or the parroting of "talking points for dummies". This is no comment on you or your postings, but a guess at what Tim - and probably the majority of his readers - might be looking for from posters here generally.

Jemima, the posts here consist of variations on the same stale point, that the "methodology" of the study was sound (which no one is contesting, so much as its low power, and the way the results were misinterpreted and misrepresented by just about everyone including it seems the authors of the study themselves.) The rest is posturing.

I wonder if there is a confidence interval too broad for Tim and dsquared to accept as indeterminate, or even useless? Were the sample size halved, they could have claimed an even higher upper bound for their CI, a far wider and even more damning 66% interval. They could go right on ignoring the lower bound as they have done here. They could go on sneering at the public consensus regarding what constitutes useful knowledge, tut-tutting over the rest of the world's innumeracy and ignorance of accepted epidemiological practice. It would certainly be a cheaper and easier study to perform, and it would have about the same effect upon public opinion and policy as this one has.

Well speaking of numeracy Pascal, in one of your own contributions above containing more of your own work and a little less of the ad hominem, ubercritic stuff, there seems to be a bit of self delusion when you proudly produce from Excel some rough calculations for confidence intervals but try to laugh off the odds ratio of approximately 4 to 1 that they represent. Who other than yourself do you expect to be persuading that odds of 4 to 1 are not highly signififcant in the context of your own demonstration, I'm wondering?

Who other than yourself do you expect to be persuading that odds of 4 to 1 are not highly signififcant in the context of your own demonstration

If the confidence intervals represented 1% and 4%, would you understand why "four to one" sounds simply ridiculous ? 16.5% is highly unlikely, so is 4.8%, and yes it's really that simple, jemima, no PhD or MBA required. 88-108k is still a rather wide range around the midpoint figure in which all of you seem to have such immense confidence, and the odds against it being accurate are over 6 to 1! A longshot to any bookmaker on earth. "Highly likely" it isn't. The way to fix this problem is simple: a larger sample. Your only truly analogous prior survey (Kosovo) was an order of magnitude more precise.

Yes, Tim, I'm really really silly for writing that because Gaussians are continuous distributions, every point has equal probability of being attained (that only intervals have positive probabilities and can be compared). Yes, and Brignell is a hack too. Maybe if you engaged in the arguments rather than calling people silly or hacks, it would do you some good.

First, tell me which set of numbers is closer to 3, those between 0 and 1.04 or those between 2.975 and 3.025? I am curious, because you know, for a Gaussian with mean 3 and stdev 1, those between 0 and 1.04 have 2.3% probability whereas those between 2.975 and 3.025 have 2% probability. Also, could you post, right here on your blog, what the probability is of 8000 deaths and the probability of 98,000 deaths. Your email reply to my "silly" post didn't really explain it.

Finally, has there ever been a study done by someone reaching conclusions that confirm a view held by those on the Right that you liked or a study confirming a view held by those on the Left that you didn't like?

Oh and by the way, I appreciate the link.

Put your faith in the numbers, Pascal, just the numbers! What you're saying and believing to be "simple", no effort needed, is really "simply" meaningless. The 4 to 1 odds have meaning independent of the confidence interval likelihoods (whether 1% vs 4%, or 4% vs 16%, or 16% vs 64%). And it's OK if not all of us can grok the theory, it's why professional statisticians get paid to do it for us (they don't get paid enough to put up with the noise and namecalling from the armchair ideologue experts though, and that's "for sure").

I think, anyway, that you're quite right Yevgeny - you were really, really silly for writing that!

The 4 to 1 odds have meaning independent of the confidence interval likelihoods

Yes, and more far relevant than "how many times more likely" is one versus the other is the absolute magnitude of each, hence my example. If you don't understand that 16% vs 64% is a more meaningful distinction than 4 vs 16 or 1 vs. 4, I really don't know what to say. Maybe you're right to let the pros handle all that "grokking."

Interesting that TallDave (I'm a short David, btw) talks about my comical piece as an '"argument by authority," generally not convincing in a logical sense.'

I am sure the philosophers would have fun with that from both points of view. It did provoke in me the simple observation that the far Right is consistently involved in undermining the authority in our institutions of scientific knowledge, even though the point of that authority is that it is painstakingly built up on the basis of the best truth-creating tools we have.

It's not the facts of a particular case, its the underlying aura of trustworthiness they are aiming at. If I was a genuine conservative, I may think that this mob is far more dangerous than any bunch of organised Lefties. They are attacking the idea that we are a society based on science.

They are attacking the idea that we are a society based on science.

We are a society based on science? News to me. Care to elaborate?

you were really, really silly for writing that!

Jemima, Yevgeny seems to be a doctoral candidate in mathematics at a prestigious university, amply endowed with "truth creating tools" of all kinds, not some dilettante MBA or computer scientist. Since you're clearly awed by credentialed scientists and the aura of trustworthiness surrounding them like a halo, mightn't you cut him some slack?

Mr. Vilensky seems to be making the same error that several of the Lancet critics make in translating reality to math and back again. You cannot have a partial death, or -- even more absurd -- a point death.

In other words, when someone talks about 100,000 dead in a Guassian distribution, that person is obviously talking about the interval between 99,999 (exclusive) and 100,000 (inclusive). That interval has a probability on the positive side of zero.

I'll settle for your comments on the substance, thanks Disputo, although somebody will probably prove himself silly by assuming that because you didn't say everything that could be said just then, that was only because you couldn't have said a fair bit more about statistical measure theory, or whatever :)

Pascal you're out of your depth (as I think Tim suggested to you earlier), and so is Yevgeny - there are lurkers prowling here who eat doctoral candidates in maths as hors d'oeuvre before getting down to the real business of chowing out on unwitting, widemouthed posters. Don't lightly turn yourselves into snacks, would be my friendly advice to you and Yevgeny. If you insist on playing on though you'll at least be getting somewhere if you can build on your recent progress with avoiding the ad hominems (which were a dead giveaway before).

Best, j

88-108k is still a rather wide range around the midpoint figure in which all of you seem to have such immense confidence, and the odds against it being accurate are over 6 to 1!

But the odds of it being at least 88k, as dsquared pointed out, are 58%. We have a study that shows a significant likelihood of more than 88k people dying - and you would suggest it be disregarded because we can't be certain whether it's 88k or 130k. I think a large number of people, including Tim and dsquared, figure that knowing with confidence "At least x number of people died" is not much less valuable than knowing with the same confidence that "Between x and y people died". I think they'd like a new study done with more samples, as well - they've called for it repeatedly on this website.

Yes, and Brignell is a hack too. Maybe if you engaged in the arguments rather than calling people silly or hacks, it would do you some good.

Maybe if you read the several posts and comments (which Tim actually linked to) in which he provides detailed arguments as to the problems with Brignell and responds to a range of arguments offered in Brignell's defence, it might do you some good.

Ok, Pascal, you say that 16.5% is "highly unlikely". Well I've got a 16.5% confidence interval for you: 50,000 or less excess deaths. So, according to you, it is highly unlikely that there were less than 50,000 excess deaths (and that's not counting Falluja or similar places).

Yevgeny, Brignell is a crank, not a hack.
I explained to you in several emails why your post was silly, as did dsquared and now Disputo in comments here. I suggest you correct your post.
As for your question about Left/Right issues, is everything Left/Right politics to you? Is there some official Left position on all scientific questions that I'm supposed to know about? Can I guess that you raised your silly objection because you are a right-winger and supported the Iraq war?

the posts here consist of variations on the same stale point, that the "methodology" of the study was sound (which no one is contesting[...])

I think you'll find, Pascal, that it's taken a good deal of work by Tim, dsquared and other defenders of the Lancet study to bring the debate to the point where the methodology isn't being dismissed out of hand by those who haven't so much as read the paper or troubled themselves to understand it. For this, they deserve considerable credit.But there can be no doubt that "posturing" - in its many forms - pollutes and ultimately derails these online debates. It's hard to avoid being sidetracked by condescending and other petty responses, especially when they're directed at the person, and people aren't usually dedicated enough to make the requisite effort. A pity, IMHO, because there's much that can be learned here. Now to the part of my post which might actually be listened to by someone (so to speak): a link to an Australian radio interview from a few days ago with Les Roberts: http://www9.sbs.com.au/radio/index.php?page=wv&newsID=108407BTW if you're reading this, Tim, could you please take a couple of minutes to review my several posts in the "Lancet Denial" thread regarding one of your contributors who steadfastly refuses to either support or retract defamatory allegations directed at a small publication comprised of, AFAICT, honest and dedicated journalists. Slanders, even on blogs, should be either substantiated or withdrawn.

Following on from JOT's comments, perhaps Tim we could find a way to get popular support that will enable us to withdraw power from the band of extremist liars whom currently occupy Whitehall and the White House... after all, their policies based on lies and deceit are causing death and misery on a grand scale. By contrast one individual, Shirin, who makes a lot of excellent posts on this thread, is being blacklisted by JOT on the basis of a single comment. Strange world.....

It seems to me that Roberts and friends have established a strong case that the number of deaths in Iraq during and after the war significantly exceeds the number of deaths immediately before in an equivalent time period. Because of limitations on the ground it was difficult to carry out an optimal survey.

Moreover, some places in Iraq including one of the clusters have taken abnormally high damage. As dsquared has explained, the statistical analysis used makes it impossible to meaningfully include that cluster in the bootstrapping randomized shuffling of pre and post war deaths which gives the study statistical power. A little thought shows that the objections raised vis-a-vis how the results would be changed by shuffling a death from one cluster to another, or one death more or less are falsified by this method of analysis. Moreover, it is clear that those advancing such arguments, either have not read the paper, or are clueless with regard to this method of analysis (including me until I read dsqs comment), or are simply trying to pull the wool over everyone's eyes. After all the arguments over the central limit theory and the confidence interval are about what you would get in a good ninth grade class.

The serious issues raised have to do with how one could improve the confidence intervals and perhaps what the likelihood is that a better survey would yield a difference central limit. It is clear that those criticizing the Lancet study here and in all places I have seen on the net or elsewhere (if there are such, please raise your hand and sign the petition), WITHOUT EXCEPTION, also are not calling on the US, British and Iraqi authorities to carry out an improved survey, indeed, they are resisting that course of action. Those who defend the statisitcal methodology, again without exception, are. This proves that the attacks we have seen here on the Lancet study are dishonest at best.

On a somewhat different note, I might tell Disputo, that after last week, I am half dead. OTOH our friend Yevgeny not only thinks people are continuous (well I suppose you could count loss of an arm as 1/15th dead, but the survey does not go into that), but that excess deaths are unbounded on the positive and negative sides.

Pascal wrote:

I wonder if there is a confidence interval too broad for Tim and dsquared to accept as indeterminate, or even useless?
Yes there is; if the 95% confidence interval had included a relative risk ratio of 1.0 I would have said that the study could not reject the hypothesis that the invasion had made things better rather than worse. In return, would you accept that the study did reject that hypothesis at the 95% confidence level?
Yevgeny: I really do think you're quibbling. Any interval constructed around the mean will have greater probability than an interval of the same size constructed around a point other than the mean, for a normal distribution, or for the bootstrapped empirical distribution actually used. "Numbers in the middle of the confidence interval have higher probability" is unrigorous but perfectly reasonable shorthand for that. I really have to ask; what practical mistake in inference do you think people are going to make as a result of the use of that shorthand? (I'd also note that contra your assertion, if you look at the paper, the confidence interval 8k-194k is constructed on the basis of the bootstrap, not a Gaussian or other continuous distribution).

(on a practical note as well, Yevgeny is a mathematician specialising in probability theory and thus will, most likely, end up in the risk management department of an investment bank. When you get to a risk management committee and the Head of Trading says something like "of course a daily return of 0.5% is much more likely than one of 25%", don't correct him in this manner!)

JoT,

Thanks for that radio link. Interesting that Roberts expected the secondary effects of war (disease, higher infant mortality due to disruption of services etc.) to cause more deaths than violence. This has a bearing on the claim (made by Shannon Love and others) that the cluster-sampling methodology was flawed from the outset, in that violence tends to be confined to small pockets.

Leave aside the fact that such a flaw means deaths are more likely to be undercounted; also the fact that those who advance this criticism do not seem to have a feasible alternative methodology to suggest; the interview makes it clear that Roberts chose a reasonable approach given his past experience.

In return, would you accept that the study did reject that hypothesis at the 95% confidence level

I would accept that the study rejected the hypothesis that the overall risk of death declined (at that confidence level.) Whether the invasion "made things better or worse" is not what the study pretends to quantify, and it certainly proves no such thing.

Toby, it is the idea that the study lets anyone "know with confidence" any precise figure or even range that is silly. The wide confidence interval accomodates too much spin - note Tim's example above. The odds of 88-108% are exactly the same as for 0-50%, and yet he imagines this is a point for "his side." Probabilistic measures so broad simply deserve more cautious treatment than this has received.

er 88-108k and 0-50k, obviously.

Pascal, your responses continue to be disingenous. You pick a narrow range around 98k to avoid admitting how likely 50k+ is. Yes, we don't know whether it is 50k or 100k or 150k or 200k, but most likely it is in there somewhare.

So Pascal, where are you advocating for a larger study to be done since you don't seem to like the one we have? An important function of such relatively small studies is to indicate whether there might be something serious to look into. If the result had been 10 K deaths extra with limits between -90000 and + 110000 the proper response may have been move on, but with these results and indications that it may be a large underestimate because of a small number of heavily impacted areas, the proper response is NOT nothing here, move on.

Perhaps Pascal should change his name to Derrida.

Yevgeny,

Since comments don't seem to be working at your own blog and since you seem to have a taste for exactitude bordering on the pedantic, I may as well note here that your post over there on the wrongness of Tim contains the following howler, which you may wish to correct:

"We know for 100% certainty that there are between 0 and 100 billion excess deaths as a result of the war."

We know nothing of the sort. One of the arguments used by the humanitarian proponents of the war, such as Norm Geras and his fans, was that it might actually save lives. Since Saddam was indeed a murderous dictator, as they frequently and quite needlessly pointed out, this was not a ridiculous notion.

Excess deaths are not restricted to the positive real numbers - although unlike you I am inclined to think they must be integers, whether positive or negative.

Since I was addressing Yevgeny, perhaps I should rephrase that: Excess deaths are not restricted to the non-negative real numbers; zero is of course a possible value.

Eli, I have no problem acknowledging that another more rigorous study should be conducted, along the lines recommended by Heiko Gerhauser on earlier threads. Where do I sign the petition?

Tim, I'm done debating whether "most likely" means "greatest of many small likelihoods" or "quite likely." My standards of epistemic rigor and exactitude are simply different from your own, as the "2,000-14,000" IBC example from the last thread already makes quite clear.

Pascal, there is no need to debate what the words mean. You described a 16.5% CI as "highly unlikely". So, using your definition, less than 50,000 excess deaths is highly unlikely. I expect you'll try to evade this point again.

"Where do I sign the petition?"

For the UK Government try here. I don't know of any US campaign on the issue.

dsquared, thank you for a critique that doesn't involve attacking people's credentials or calling them names.

You said that contra to my assertion, the 95% confidence interval is constructed from the bootstrap not a Gaussian distribution. If so, then there is no basis on which you could say that 98,000 is more likely than 8,000. In fact, 8,000 could be more likely (since you could have a distribution that is peaked on the left). I assume that at some point, they relied on the assumption of Gaussian standard errors to obtain that confidence interval. So, it's some transform of a Gaussian.

To Tim, no there is no Right-wing science or Left-wing science. But there is science that confirms the views of those on the Right and that which confirms the views of those on the Left. It seems you always, just by chance, choose only one type of science to defend and one type of science to attack.

In response to the point that there can't be half of a death. Of course there can not be half of a death. But the regression done with the bootstrap DOES contain continuous parameters, from which the confidence interval is obtained. So, as a strictly statistical point, every point in that interval is equally likely. We have to use the theoretical distribution since we have no clue whatsoever what the ACTUAL distribution is. If we did, we wouldn't need to do statistics.

In the end, I will acquiesce to Tim and dsquared and Donohue and agree that yes, you are right, as a "practical" matter, I suppose the 98,000 number is more "likely" whatever that means. I will correct my post.

A question: the authors seemed to have assumed a log-linear regression. What if it were cubic or quadratic? Wouldn't that seriously affect the conclusions? It's a bit odd that nowhere do they actually present the regression parameters. I mean, I'm sure it's common in epidemiology, but even in less scientific fields like economics and political science, it is common to display the full output of whatever stats package you are using. Maybe I'm misunderstanding what they actually did, but contra to your claims, dsquared, it is not clear at all in the paper. There is no data chart, for example, showing all of the parameters obtained from their calculation. It doesn't mean that their conclusions are false, but it makes me rather suspicious that people who are not familiar with standard epidemiological procedures, but are numerate nonetheless can't actually figure out what they did.

Also, for Mr. Donoghue, would you prefer it if I wrote between -6 billion and 100 billion?

Mr Vilensky,

Itsâs your blog. You can obviously see the point I am making, but nobody can force you to admit that you see it. However, it may help third parties to understand where I am coming from if we imagine an alternative universe in which these fascinating disputes about the interpretation of confidence intervals arise in a different way.

Imagine, if you will, that a group such as Al-Qaeda manages to introduce some potentially lethal bacteria into Americaâs water-supply. After a time the plot is uncovered and remedial action is taken. Alas, for thousands of Americans, it is already too late. Some are dead, some are infected but they donât yet know it and there is no hope of a cure.

What is the death toll? The President is not eager to enquire. He has other priorities, whatâs done is done: counting bodies will not reduce their number. He is running for re-election and has no intention of letting this tragedy dominate the news. Anyway, the effects of the bacteria are complex â it may even be the case that those who are infected and recover will have immunity to other diseases, so according to some authorities Al-Qaedaâs action may have saved lives. This is implausible, but by no means impossible, and the presidentâs admirers play it up for all they are worth.

But a small team of doctors decides that it is important to get some idea of the death toll and undertakes a study on a shoestring. To add to their problems the work is dangerous: working with the bacteria may cost researchers their lives. The study is constrained by these problems and they freely admit that they cannot be precise. The death toll may be as low as 10,000 or as high as 400,000. But they are able to reject the hypothesis of the Presidential Polyannas: the net effect clearly is that lives have not been saved.

Naturally the presidentâs supporters are livid. Bloggers attack the study without bothering to read it, the motives of the researchers are impugned. Foreign commentaters snigger at the ludicrously wide confidence interval. A Le Monde columnist describes it as âa dart-board.â A blogger who seeks to defend the study gets comments that start: âBâjour Tim, I am a physicist and I fart in your general direction....â

In this alternative universe, does the alternative Yevgeny Vilensky make hyper-pedantic points about the fact that the integral from the mean to the mean, of a continuous distribution, is necessarily zero? Or does he concern himself with matters of substance?

erratum: for "distribution" read "density function." One gathers that these things are important to Mr Vilensky.

As long as we are being pedantic, there are only ~6 billion people on earth, so we do have an upper bound.

Pascal,
Talking about our society being based on science is a bit like talking to fish about water. It simply dominates everything.

While the remark is obviously simplistic, in that no society as an entity has one single conceptual base, it is nonetheless fair to say that all the other major drivers (like capitalism or individualism or competitiveness) make very little difference without the huge amplifier effect created by science.

If you were a martian observing humanity over time, you would see a large number of societies existing like colonies on a petrie dish for a long time - at least ten thousand years - until very quickly one colony expands to cover the whole dish. The difference, I suggest, is the idea of experimentation - the systematic search for improvement, which is neither a kind of Brownian bricoleur approach or the working out of pre-existent dominant metaphors about self, society and the world, like totems.

Visualise an alternative and then find out if it fits. I am, of course, defining the notion widely enough to include engineering which I think is fair enough.

Yevgeny Vilensky,

The tone of my posts above (9:06 and 9:11) would have been less snarky if I had seen the second update to your blog post before writing them. Apologies for that. (I must make it a rule not to post comments after coming home from the pub.)

Like you, I would have liked the Lancet to provide a bit more information on their estimation procedures, pitched at a level suitable for those with just basic stats. But as I have said in other threads, that is a problem with lots of papers; Roberts et al. is better than most. They do list the three software packages they used, two of them are readily available and one of those (EpiInfo) is free. Their notes 15 and 16 also cite two textbooks so I'm sure that other researchers would have no difficulty. For the general reader, it is enough to note that the relative risk of death (2.5 with Fallujah and 1.5 without) can be got from their numbers with just a calculator. There is only one technical paragraph in the paper, so nobody should be put off.

Like you, I would have liked the Lancet to provide a bit more information on their estimation procedures, pitched at a level suitable for those with just basic stats. But as I have said in other threads, that is a problem with lots of papers

While this is a laudable aim, each paper in any journal (including the Lancet) has a strict page limit. If every paper on epidemeology had to be pitched at a basic stats level, every paper would then spend a number of pages re-covering the same basic stats grounding. The amount of the paper that could be devoted to an in-depth discussion of the new results is less. The paper will be easier for new readers to access, but wouldn't have the depth that people already in the field need to accurately critique the paper. And while in certain cases such as this, there is a lot more interest from people without a rigorous stats grounding, in most cases the people interested in the paper will have that grounding already and the pages spent on pitching the paper at people with a basic stats background would essentially be wasted.

I believe the current practice in academia is that while you provide sufficient references to indicate where your research fits within the field, it is still up to the reader to follow up those references and acquaint themselves with previous research. I would expect someone critiquing my robotics research, or Mr Vilensky's research, to have some idea of the state of the art, or if not, to read up on it and take mine (or his) work in context with it; I thought that was fairly common in academia, and would be the assumption for the Lancet.

Apologies. Both of the Qur'tuba comments were me figuring out trackback pings.