When people raise questions about the mysterious 1997 survey, Lott’s standard line of defence is:
“the survey was replicated, and I obtained very similar results.”
So how similar are the results? Well, Lott claims that the 2002 survey gave a 95% brandishing number, quite close to the 98% he claims he found in 1997. However, the 2002 survey does not give a 95% number and is too small for the number to be reliable.
Very little attention has been paid to the other result that Lott claims comes from his 2002 survey—an estimate that there were 2.3 million defensive gun uses (DGUs) in 2002. This number is quite close to the 2.1 million number that Lott says he got from the alleged 1997 survey.
We can test if the difference between two results is statistically significant (Details here). The p value given by the test is 0.96. This means that two surveys that sampled exactly the same population would differ by more than this difference 96% of the time. Now, a small p value (less than 0.05 by convention) would mean that the differences were too large to be likely to be caused by chance, and we could conclude that the surveys were different in some way. A large p value like we got here suggests that the close agreement of the surveys is too good to be true. Properly conducted surveys of the size that Lott says he used would tend to differ by more than the amount that these two surveys supposedly do. Now it is possible that the close agreement is just chance, but there is only one chance in twenty-five chance that the surveys would have agreed this well, and given all the other behaviour by Lott, this seems a little suspicious.
Nobody can check Lott’s calculations of the 2.1 million estimate that supposedly comes from the 1997 survey, but I can check his work on the 2002 survey. Using the weights I worked out while showing that he got the brandishing number from the survey wrong, I found that there were 15.8 weighted DGUs in this survey. Dividing this by 1015 (survey size) and multiplying by the adult population of the US gives a DGU estimate not of 2.3 million as Lott claims, but of 3.3 million.
Lott managed to get both the number of DGUs and the brandishing number wrong in his 2002 survey. Both of these errors were in the direction of making the 2002 survey agree more closely with the results he claims to have obtained from his 1997 survey. In the case of the DGU numbers it is rather unlikely that a random error would make it agree that closely.
Technical details: To test if the two surveys gave different results I used a Chi square test. This requires the weighted number of defensive gun uses in each survey. Lott does not tell us this number, but we can work backwards from the estimated number of DGUs. For 2002, this was 2.3 million in a population of 207 million adults and a survey size of 1015, so the weighted number is 1015*2.3/207=11. For the 1997 survey, a similar calculation yields 25 weighted DGUs in a survey of size 2424. Plugging these numbers into a 2×2 Chi-square with Yates continuity correction gives p=0.96.