Last September Lott told Lindgren that he "weighted his respondents by demographic information taken from his main national study in More Guns, Less Crime"
On January 14 he provided more details:
I did not weight the sample by household size but used the state level age, race, and sex data that I had used in the rest of my book. There where 36 categories by state. Lindgren hypotheses why you can get such small weights for some people and I think that this fine of a breakdown easily explains it. I don't remember who answered what after all these years, but suppose someone who fired a gun was a elderly black in Utah or Vermont.
I tried to weight the data in Lott's 2002 survey using this procedure. The first difficulty I encountered was that the age categories he said he used were 10-19, 20-29, 30-39, 40-49, 50-64, and 65+, but his survey collected age by decade. There is no way of knowing which category to assign someone in their 60s to. Also, the people in the survey were 18 or older, so using a category of 10-19 makes no sense.
There is, however, a far worse problem. In states with a small population there were only two or three people in the sample. With 36 categories for each state, that meant that most categories in those states had no data. It should be obvious that no amount of weighting can correct for the problem of having no data.
All that can be done for the categories with no data is give them weight zero. But this means that the weighting procedure is systematically biased against minorities and people from small states. An example might make this clearer: Suppose that we just weight by race and sex and 90% of the population is white and 10% is black. We take a sample of ten people, getting nine whites (four men and five women) and one black (a man). The four white men constitute 40% of the sample and 45% of the population so get weight 45/40=1.13. The five white women are 50% of the sample and 45% of the population so get weight 45/50=0.9. The black man is 10% of the sample and 5% of the population so gets weight 5/10=0.5. Notice that the total weight of the whites in the sample is 4x1.13+5x0.9=9, 18 times the total weight of the blacks, even though they are only 9 times as frequent in the general population. In this example whites were weighted twice as heavily, but with 1836 (=36x51 states+DC) categories Lott's procedure could cause even worse discrepancies.
I didn't notice this problem until I took his data set from the 2002 survey and tried to follow his procedure. With 6 out 7 people brandishing (86%), after weighting using his procedure, the weighted brandishing percentage is 99.8%. It turns out that the person who fired was a minority woman from a small state, and ends up with a weight about 100 times as small as that for the other defensive gun users. Obviously this is incorrect.
Now in Lott's new book, The Bias Against Guns (published in March) he claims to have weighted the data by race and sex (only six categories). Unlike the weighting procedure he claimed to have followed in September (and he repeated the claim in January), this is a reasonable procedure. (Though, of course, the sample size is too small to give even a rough estimate of the brandishing percentage).
One explanation for the change in Lott's story is that in January he had not done any weighting and it was only when he tried to do the weighting that he discovered that the procedure he described in January was not valid.