Too good to be true: The results of Lottâs 2002 survey

When people raise questions about the mysterious 1997 survey, Lott's standard line of defence is:

"the survey was replicated, and I obtained very similar results."

So how similar are the results? Well, Lott claims that the 2002 survey gave a 95% brandishing number, quite close to the 98% he claims he found in 1997. However, the 2002 survey does not give a 95% number and is too small for the number to be reliable.

Very little attention has been paid to the other result that Lott claims comes from his 2002 survey---an estimate that there were 2.3 million defensive gun uses (DGUs) in 2002. This number is quite close to the 2.1 million number that Lott says he got from the alleged 1997 survey.

We can test if the difference between two results is statistically significant (Details here). The p value given by the test is 0.96. This means that two surveys that sampled exactly the same population would differ by more than this difference 96% of the time. Now, a small p value (less than 0.05 by convention) would mean that the differences were too large to be likely to be caused by chance, and we could conclude that the surveys were different in some way. A large p value like we got here suggests that the close agreement of the surveys is too good to be true. Properly conducted surveys of the size that Lott says he used would tend to differ by more than the amount that these two surveys supposedly do. Now it is possible that the close agreement is just chance, but there is only one chance in twenty-five chance that the surveys would have agreed this well, and given all the other behaviour by Lott, this seems a little suspicious.

Nobody can check Lott's calculations of the 2.1 million estimate that supposedly comes from the 1997 survey, but I can check his work on the 2002 survey. Using the weights I worked out while showing that he got the brandishing number from the survey wrong, I found that there were 15.8 weighted DGUs in this survey. Dividing this by 1015 (survey size) and multiplying by the adult population of the US gives a DGU estimate not of 2.3 million as Lott claims, but of 3.3 million.

Lott managed to get both the number of DGUs and the brandishing number wrong in his 2002 survey. Both of these errors were in the direction of making the 2002 survey agree more closely with the results he claims to have obtained from his 1997 survey. In the case of the DGU numbers it is rather unlikely that a random error would make it agree that closely.

Technical details: To test if the two surveys gave different results I used a Chi square test. This requires the weighted number of defensive gun uses in each survey. Lott does not tell us this number, but we can work backwards from the estimated number of DGUs. For 2002, this was 2.3 million in a population of 207 million adults and a survey size of 1015, so the weighted number is 1015*2.3/207=11. For the 1997 survey, a similar calculation yields 25 weighted DGUs in a survey of size 2424. Plugging these numbers into a 2x2 Chi-square with Yates continuity correction gives p=0.96.


More like this

A few days ago I observed that Lott had changed his story from his original, unworkable, claim that he had used 1836 categories (sex, race, age and state) to weight his data to the claim that he had used just six (sex and race). If this is indeed the scheme he used then two…
Lott's reply to Duncan's article raises some disturbing questions about Lott's honesty. See also James Lindgren's report on his attempt to find some evidence that Lott actually conducted a DGU survey. Where did that 98 percent come from? 98 percent claims before 1997 Way back in 1993 in…
[On Sep 14 2002 I posted this to firearmsregprof. I also emailed it to John Lott. ] Way back in 1993 in talk.politics.guns, C. D. Tavares wrote: The answer is that the gun never needs to be fired in 98% of the instances of a successful self-defense with a gun. The criminals just leave abruptly…
Otis Dudley Duncan This discussion is concerned with four topics: (1) Lott’s references to, remarks about, and discussions of DGU statistics originating in sample surveys or polls carried out by other investigators; (2) Lott’s claims about a survey he says he conducted in 1997; (3) Lott’s reports…

If I understand the "data set" correctly, the DGUs appear in clusters - e.g., there is some white guy who managed to use his gun to stop a crime on no less than 3 separate occasions in 2002.

If this is so, a close agreement is even less likely than it would appear from comparing the number of uses. For example, if Lott just happened upon one more or one less gun user in his 2002 survey, the DGU count would have swung up or down by more than two uses, on average.

On a related note: I doubt that a continuous test is accurate enough to be useful when dealing with such small numbers of positive answers. Some sort of a permutation test would probably be more informative.

By Pro bono mathe… (not verified) on 12 Mar 2005 #permalink

This 70m year old says to hell with all that math. I "brandished" and stopped three guys with a hammer from attacking me.

A girl friend "brandished" and got a guy who had broke into her and her childs apt to climb back out the window he had come in.

While she should have held for the cops, she did not do so.

The point is that these types of things go on all of the time and are never reported. So I do not give a damn how many times someone estimates it happens. GUNS SAVE LIVES AND PREVENT CRIME!

Neil C. Reinhardt

By Neil Reinhardt (not verified) on 17 Mar 2005 #permalink

Hey Neil:

Whereas it is true that guns save lives and prevent crime, it is also true that guns take lives and help commit crimes

So the question becomes which happens more?