A few days ago I observed that Lott had changed his story from his original, unworkable, claim that he had used 1836 categories (sex, race, age and state) to weight his data to the claim that he had used just six (sex and race). If this is indeed the scheme he used then two things follow:

  1. He has incorrectly calculated the brandishing number for his 2002 survey.
  2. It is impossible for him to have obtained a 98% brandishing number for his 1997 survey.

Here are the details:

In his new book, Lott tells us the brandishing number he gets after weighting:

“the survey I conducted during the fall of 2002 indicates that simply brandishing a gun stops crimes 95 percent of the time”

After Lott spent years claiming that the brandishing number was 98%, I can now claim victory. He has stopped making the 98% claim. As of this month he now claims:

Something like 95% of the time or so simply brandishing a gun is sufficient to go and cause a criminal to break off an attack.

I guess I should be pleased that he has stopped using a number based on nonexistent data and started using a number based on a real survey. I only have a few remaining quibbles:

  1. The sample size of just 7 defensive gun users is far too small for a point estimate like 95% to be at all meaningful.
  2. It is dishonest not to mention the markedly different estimates from professionally conducted surveys with samples containing hundreds of defensive gun users.
  3. Lott hasn’t even calculated the point estimate correctly. It should be 91%, not 95%.

Weighting Lott’s 2002 survey isn’t particularly complicated. I’ll go through the whole procedure here, so you can check my working. To work out he weights you need to count the number of adults in the US in each category, and the number of people in Lott’s survey in each category:

Category Population  # in sample  weight
Black Male 10,821,013 32 1.52
Black Female 12,790,596 55 1.04
White Male 72,496,265 333 0.98
White Female 77,992,817 424 0.83
Other Male 17,591,994 41 1.93
Other Female 17,586,464 54 1.46
Total 209,279,149 939

Looking at this we see that other race females make up 17,586,464/209,279,149=8.4% of the population, but only 54/939=5.8% of Lott’s survey. That means responses in that category must be given weight 8.4/5.8=1.46 to compensate for their underrepresentation in the survey. The other weights in the last column are calculated in a similar fashion.

Second, we look at all the defensive gun users in Lott’s survey. Here is the complete data set:

Category   Weight   DGU count   Weighted DGUs   # times gun fired   Weighted times fired
Other Female 1.46 1 1.46 1 1.46
White Male 0.98 2 1.95 0 0
Other Female 1.46 2 2.92 0 0
Black Female 1.04 1 1.04 0 0
White Female 0.83 2 1.65 0 0
Other Male 1.93 2 3.85 0 0
White Male 0.98 3 2.93 0 0
Totals 8.67 13 15.81 1 1.46

All we do is look at which category each case falls into to determine its weight and multiply that weight by the number of defensive uses and firings. The result is that there were 15.81 defensive uses after weighting and 1.46 of these involved firing, so the weighted firing percentage is 1.46/15.81=9%, or, in other words 91% brandishing, contradicting Lott’s 95% figure that he claims came from this survey. My calculations are also available in a spreadsheet.

Back in January, Lott responded to my question

Even Lott cannot possibly be sure that the correct result of his survey was 98% since there is no way to check his calculations. Why did he repeat the figure over and over again?

with

Is there any evidence to suggest that I can’t figure out a weighted average?

Well, there certainly is.

Even more troubling is the fact that Lott now recalls that in his 1997 survey there were two cases of firing. For these two cases to constitute 2% of the (weighted) 25 uses Lott must have had, their total weight must have been 2% of 25 or 0.5. That means that at least one of the weights must have been 0.25 or smaller. And to get a weight of 0.25, Lott’s 1997 survey would have had to have four times as many people in the appropriate category as occur in the general population. Clearly that is not even possible with a category like “White Male” which makes up 34% of the population. The most likely way this could have happened is with the smallest category, “Black Male”, which makes up 5% of the population. Five percent of 2424 people (the size Lott claimed for his 1997 survey), is 125 black males. If that is the expected number, what is the chance of getting 500, four times as many? I worked out the probability and it is less than one in a googol (that’s a one followed by one hundred zeroes). That counts as impossible in my book. In other words, if Lott used the weighting scheme he now claims to have used there is no way he possibly could have obtained the 98% brandishing figure from his 1997 survey.