From: John Lott
Sent: Saturday, January 25, 2003 2:49 PM
Please post
Claims by archpundit (apparently reflecting similar claims made in other places):
Lott claims his new survey will solve the problems, but that assumes he oversamples DGU's. If he doesn't, his finding will be as pointless as the above. . . .
So he collected 36 demographic variables in his survey? Or are there 36 breakdowns of a smaller number of demographics collected? Either way this was a pretty involved survey that had to take a lot of time on the phone to collect all that data. Surveying isn't nearly as quick as it sounds and collecting that many distinct variables plus gun use information would entail significant amounts of time on the phone.
Using a professional center I'm guessing he would have looked at least $20,000. Even using cheaper labor (less reliable), this had to cost a lot of money. Gathering that much demographic data with untrained surveyors without the benefit of computer assisted surveying techniques would likely be 10 minutes a call (this is charitable). At 2400 respondents that is 400 hours of labor not including non-responders and no answers. At $7/hr this is $2800 alone, plus charitably the long distance of $1200 at $.05 a minute. So he paid $4,000 out of his own pocket?
Does he really think that anyone buys that he didn't at least deduct this expense from his taxes?
This adds another significant amount as even a simple database for 2400 respondents would take time to construct--especially with 36 categories.
Of course, all of this should have been overseen by a competent researcher, which Lott appears not to be. The stunning thing is that if it is true Lott did all of this, he wasted a fair amount of money on a useless survey that lacked even rudimentary controls to ensure a sample of any quality.
It may be that Lott didn't just make up the number, but he might as well have. One can complain all they want about how this is only about one sentence. Strictly speaking it is, but it also shows an incredible lack of understanding about data collection and analysis. His other results aren't very impressive once one looks at the standard errors anyway. I am beginning to wonder if Lott uses these side issues to avoid discussing his actual results. By focusing on claims of fraud and providing just enough information to be plausible, no one ever gets around to questioning the actual theory and statistical results.
I have already discussed these issues, but it is obvious that I need to repeat what I have previously sent out.
I have previously posted the survey questions used (see below for another copy). The 36 demographic categories for the 1997 survey were the exact same ones that I used in all the regressions in my book More Guns, Less Crime. The breakdown is by age (six age categories -- 10 to 19, 20 to 29, etc.), by sex, and by race (black, white, other). Information on where the person lived is immediately available because we have the person's area code and address from the telephone CD that we used. As to demographic questions, you will see that we asked two race and age. The student conducting the survey would fill in sex on their own unless there was a question.
"This adds another significant amount as even a simple database for 2400 respondents would take time to construct--especially with 36 categories."
This is simply incorrect. The demographic information is a product of only four variables. First you merge the census demographic and population information with the survey file. Since the 36 demographic categories are a function of only three answers to the survey, a simple set of "If, then" statements tell you in which of the 36 categories each respondent falls in. That quickly gives you the share of that states population represented by people who answered the survey in that demographic group, and you also figure out that state/demographic group's share of the national population.
"10 minutes a call (this is charitable)"
When you look at the survey you will see that it is very short. Well over 90 percent of the people would answer no to the first question and then only have the two demographic questions to answer. In this case, the survey would only take about thirty seconds or so. Even for those who answered "yes" only a fraction would have to answer all the other questions. Overall, only about one percent of those surveyed would be asked all the questions in the survey and even then the entire survey would only take a couple of minutes.
The 2002 survey of over one thousand respondents was completed over just eight nights. Students were often able to survey over 20 people per hour.
As to deducting these costs on my income taxes, my 1997 tax form, which I have shared with many others, shows that $8,750 was deducted for research assistants (the heading was under "legal and professional services"). We do not keep the supporting documents past the three years required by the IRS and the $8,750 does include the expenses for other projects. On the other hand, I am sure that I did not keep track of all of my expenditures so the $8,750 is a sizeable underestimate of what I spent.
The survey telephone numbers were obtained from a CD directory for the entire United States. The numbers were selected randomly so that all states were represented in proportion to their population. (See attached below for a very similar discussion relating to the 2002 survey.)
Between the two surveys 3,439 people have been interviewed. Can more information be obtained? Sure, but given my personal resources and that these surveys are such a trivial portion of my overall interests I have spent about as much time as I plan on spending on this issue.
Detailed information on the survey will be provided in my book that is due out the end of March.
D. Survey on Defensive Gun Use
Below is the survey that was used to identify the rate of defensive gun use.
Hello, my name is _______, and I am a student at ________ working on a very brief survey on crime. The survey should take about one minute. Could I please ask you a few questions?
1) During the last year, were you ever threatened with physical violence or harmed by another person or were you present when someone else faced such a situation? (Threats do not have to be spoken threats. Includes physically menacing. Attacks include an assault, robbery or rape.) a) Yes b) No c) Uncertain d) Declined to answer (Just ask people "YES" or "NO." If they answer "NO" or "Decline to answer," go directly to demographic questions. If people are "Uncertain" or say "YES", proceed with question 2.)
2) How many times did these threats of violence or crimes occur? _____
3) Which of the following best describe how you responded to the threat(s) or crime(s)? Pick one from the following list that best described your behavior or the person who you were with for each case faced. a) behaved passively b) used your fists c) ran away d) screamed or called for help e) used a gun f) used a knife g) used mace h) used a baseball bat or club i) other (Rotate these answers (a) through (h), place a number for 0 to whatever for each option. Stop going through list if they volunteer answer(s) that account for the number of threats that they faced.)
4) This is only done if they respondent answers "e" (a gun) to question 3 If a gun was used, did you or the other person you were with: a) brandish it b) fire a warning shot c) fire at the attacker d) injure the attacker e) kill the attacker (Again, place a number for 0 to whatever number is appropriate for each option. Rotate answers.)
5) Were you or the person you were with harmed by the attack(s)? a) Yes b) No c) Refused to answer (We obviously have the area code for location, write down sex from the voice if possible, otherwise ask.)
Two demographic questions asked of all participants.
What is your race? black, white, Hispanic, Asian, Other.
What is your age by decade? 20s, 30s, 40s, so on.
Question for surveyor: Is there any reason for you to believe that the person was not being honest with you? a) Didn't believe respondent at all b) Had some concerns c) Had no serious concerns
Write up by James Knowles of the discussion of the survey:
We had a small army of interns and AEI staff making phone calls. The callers for any given night varied according to who was available/willing to make phone calls. I was here every night supervising from my office at AEI. The survey was conducted over eight nights. Calls were made between 7pm and 9pm local time. Here are the list of callers and their email addresses, I can try to track down phone numbers if need be.
Susan Follett- xxx Carl Westine- xxx Andrei Zlate- xxx Radek Muron- xxx Arnaud Bonraison- xxx Craig Morehead- xxx Melissa Robe- xxx Salma Sallam- xxx Jill Mitchell- xxx Matt Trager- (No email yet, I will have to correspond through Carl Westine) and Myself- xxx
[e-mail addresses have been provided to Dan Polsby, Eugene Volokh and several others.]
We used a phonebook program from a company called infoUSA the program was called Select Phone Pro version 2.4. The program has a random function. First, we calculated how many numbers should be drawn from each state, we decided that we would pull 4,000 numbers (based on how much the PhonePro program gives us for free). Then we took the populations of each state from the Census and assigned the quantity of numbers that we indended to get from the program. Then, in PhonePro, we picked a state, then sorted the state's list by area code, then randomly generated a number (using excel's analysis pack) as a starting point, then the Phone Pro program would export a number every so often from the list until it reached the desired number of listings exported from the state. This may be something that is easier explained in a conversation, my direct line is xxx.