[On Oct 7 2002 I posted this to firearmsregprof and emailed it to Lott.]
However, that isn't what I was referring to when I wrote "mathematically impossible". Lott often goes on to claim that 3/4 of the times the DG User fires the shots are warning shots, that only 0.5% of DG uses involve a shot fired at the offender. Kleck's survey turned up about 200 incidents. Since Lott's survey had half the sample size of Kleck's, the most he could expect to find would be 100 incidents (slightly less if you allow for the slightly lower DGU incidence he found). 0.5% of 100 is 0.5. It is mathematically impossible for half a person to report firing at an attacker.
James Lindgren replies:
Tim, your comments seem to be based on an assumption that figures in national surveys are unweighted.
Yes, I assumed that the percentages Lott gave were based on unweighted numbers.
Most national studies use a multistage stratified sampling design, which typically requires that answers be weighted. Further, a well done sample design by telephone would weight by adult family size (otherwise small households are overcounted relative to large households). A respondent in a household in the US with 2 adults would usually be weighted about .98 and a person in a household with 1 adult would be weighted about .49 (which is awfully close to .5) Lott tells me that he did not do this sort of weighting for adult household size, but he claims to have used what is usually called post-stratification, though Lott didn't use the term. Lott said he weighted his results to match demographic information from his larger study in the book. That could easily lead to weights of half a person
I agree that it is possible that you could get a weight of one-half that way, so I should not say that the result is mathematically impossible if that is what he did.
(but, of course, one wouldn't normally report a rate of incidence for behavior for which only one person in a sample reported the behavior unless one had pretty close to 100% of the population in one's sample, which he doesn't)
However, that may well be what Lott has done. Another favourite Lott statistic, presented in dozens of opeds, speeches and in his book is that "the probability of serious injury from an attack is 2.5 times greater for women offering no resistance than for women resisting with a gun." If you examine the NCVS data this is based on, you find that 1 out of 80 women who resisted with a gun was injured, whereas if the injury rate had been the same as for women offering no resistance it would have been about 2 out of 80.
Further, Tim's computation of significance (not shown above) assumes that the same questions were asked and that Lott's sample was as efficient as a Simple Random Sample. With demographic weighting, no weighting for household size, and a sample taken from a CD-ROM of phone numbers, Lott's effective sample size would be considerably smaller than the stated sample size.
I'm not sure I follow you here. Surely the only one of those that effectively reduces the sample size would be the demographic weighting? And if he took a random sample of households, the weights would be mostly close to one, so that the effective sample size would not be much different.
Further, his results would be biased in favor of small households.
I looked at Kleck's data and the fraction who fired seems to be about the same if I calculate it without weights. (Actually there don't even seem to be weights in the data set.) So this cannot explain, even in part, Lott's results.
Lott's results would be almost impossible if he used the same questions to probe the behavior as other studies, had a sampling design as efficient as a Simple Random Sample, and was not biased against large families, but Lott's study does not have any of these characteristics, so Tim's computations of significance would not be accurate.
However, I think I'm within a factor of two, which is still enough to say that his results are almost impossible.
One must be extremely careful in making accusations to avoid error or overstatement (to the extent humanly possible). I'm sure that Tim (who has examined this far more than I have) will correct me if I'm in error--and I urge him to do so. I'd rather that we end up with the right answers and the issues clarified than defend my analysis--which might itself be based on misunderstandings.
I most grateful for your comments. As far as I know, Lott has told you more about the design of his survey than he has told anyone else.
Suspicions have arisen about Lott's survey because he has failed to answer reasonable questions about it.