It never ceases to amaze me the way the Wall Street Journal combines superb news coverage with a completely clueless editorial page.
To balance an excellent news article by Carl Bialik on the first Lancet study, we have an innumerate article on the editorial page by Steven E. Moore. Moore claims that the sample size for the Lancet study is too small:
However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, “Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey,” the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn’t survey a junior high school, no less an entire country, using only 47 cluster points.
So how many clusters did Moore use to survey Iraq? In a 2004 LA Times article he wrote:
Polling in Iraq is done much as in any developing country. Interviews are conducted face to face by highly trained Iraqi interviewers. For a 1,500-person sample, for instance, 75 qada (the Iraqi equivalent of precincts) would be chosen at random, with interviews conducted in 20 randomly chosen households in each.
So 47 clusters is grossly inadequate but 75 clusters is plenty? Moore continues:
A 2005 survey conducted by ABC News, Time magazine, the BBC, NHK and Der Spiegel used 135 cluster points with a sample size of 1,711–almost three times that of the Johns Hopkins team for 93% of the sample size.
Funny how he didn’t mention his own survey’s 75 clusters here.
Moore then demonstrates that he really doesn’t know what he’s talking about:
Another study in Kosovo cites the use of 50 cluster points, but this was for a population of just 1.6 million, compared to Iraq’s 27 million.
No, the size of the population can be mostly ignored when you work out what size sample you need. Steve Simon explains:
The best analogy I have heard about sampling goes something like: “Every cook knows that it only takes a single sip from a well-stirred soup to determine the taste.” It’s a nice analogy because you can visualize what happens when the soup is poorly stirred.
Margaret Smith has an activity you can try if you, like Moore, don’t get this
Finally, it is easy to see whether or not the sample size was big enough — look at the confidence intervals on the estimates. If the sample size is too small, then the intervals will be too broad for any useful conclusions. This is not true for this survey.
Moore offers a second criticism:
And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don’t take my word for it–try using Google to find a survey that does not ask demographic questions.
Without demographic information to assure a representative sample, there is no way anyone can prove–or disprove–that the Johns Hopkins estimate of Iraqi civilian deaths is accurate.
This is an example of what Daniel Davies called the “devastating critique”, where any departure from the pure Platonic Form of the Epidemiological Study is grounds for dismissing the findings. Would it have been better to have recorded ages for the living people (they did ask about gender)? Sure, but it would have taken more time, and it actually doesn’t matter that much. For surveys where you have collected age and gender, weighting by age and gender usually gives you the same results as unweighted averages (which is what the Lancet study used).
So who is Steven E. Moore anyway? Turns out he is a political consultant at Gorton Moore International, a Republican-aligned political consultancy firm.
Update: Stats.org has more on Moore.