When people raise questions about the mysterious 1997 survey, Lott's standard line of defence is:
"the survey was replicated, and I obtained very similar results."
So how similar are the results? Well, Lott claims that the 2002 survey gave a 95% brandishing number, quite close to the 98% he claims he found in 1997. However, the 2002 survey does not give a 95% number and is too small for the number to be reliable.
Very little attention has been paid to the other result that Lott claims comes from his 2002 survey---an estimate that there were 2.3 million defensive gun uses (DGUs) in 2002. This number is quite close to the 2.1 million number that Lott says he got from the alleged 1997 survey.
We can test if the difference between two results is statistically significant (Details here). The p value given by the test is 0.96. This means that two surveys that sampled exactly the same population would differ by more than this difference 96% of the time. Now, a small p value (less than 0.05 by convention) would mean that the differences were too large to be likely to be caused by chance, and we could conclude that the surveys were different in some way. A large p value like we got here suggests that the close agreement of the surveys is too good to be true. Properly conducted surveys of the size that Lott says he used would tend to differ by more than the amount that these two surveys supposedly do. Now it is possible that the close agreement is just chance, but there is only one chance in twenty-five chance that the surveys would have agreed this well, and given all the other behaviour by Lott, this seems a little suspicious.
Nobody can check Lott's calculations of the 2.1 million estimate that supposedly comes from the 1997 survey, but I can check his work on the 2002 survey. Using the weights I worked out while showing that he got the brandishing number from the survey wrong, I found that there were 15.8 weighted DGUs in this survey. Dividing this by 1015 (survey size) and multiplying by the adult population of the US gives a DGU estimate not of 2.3 million as Lott claims, but of 3.3 million.
Lott managed to get both the number of DGUs and the brandishing number wrong in his 2002 survey. Both of these errors were in the direction of making the 2002 survey agree more closely with the results he claims to have obtained from his 1997 survey. In the case of the DGU numbers it is rather unlikely that a random error would make it agree that closely.
Technical details: To test if the two surveys gave different results I used a Chi square test. This requires the weighted number of defensive gun uses in each survey. Lott does not tell us this number, but we can work backwards from the estimated number of DGUs. For 2002, this was 2.3 million in a population of 207 million adults and a survey size of 1015, so the weighted number is 1015*2.3/207=11. For the 1997 survey, a similar calculation yields 25 weighted DGUs in a survey of size 2424. Plugging these numbers into a 2x2 Chi-square with Yates continuity correction gives p=0.96.
If I understand the "data set" correctly, the DGUs appear in clusters - e.g., there is some white guy who managed to use his gun to stop a crime on no less than 3 separate occasions in 2002.
If this is so, a close agreement is even less likely than it would appear from comparing the number of uses. For example, if Lott just happened upon one more or one less gun user in his 2002 survey, the DGU count would have swung up or down by more than two uses, on average.
On a related note: I doubt that a continuous test is accurate enough to be useful when dealing with such small numbers of positive answers. Some sort of a permutation test would probably be more informative.
This 70m year old says to hell with all that math. I "brandished" and stopped three guys with a hammer from attacking me.
A girl friend "brandished" and got a guy who had broke into her and her childs apt to climb back out the window he had come in.
While she should have held for the cops, she did not do so.
The point is that these types of things go on all of the time and are never reported. So I do not give a damn how many times someone estimates it happens. GUNS SAVE LIVES AND PREVENT CRIME!
Neil C. Reinhardt
Hey Neil:
Whereas it is true that guns save lives and prevent crime, it is also true that guns take lives and help commit crimes
So the question becomes which happens more?