Gun Use Surveys: In Numbers We Trust?

Otis Dudley Duncan, University of California, Santa Barbara

(from The Criminologist Vol 25, No 1 Jan/Feb 2000 pp 1-7)

We who work hard to produce statistics for public consumption would do well to acquire a little historical perspective. Theodore Porter's wide-ranging Trust in Numbers: The Pursuit of Objectivity in Science and Public Life (1995) takes note of 19th-century developments illustrating the "creative power of statistics.... Every category has the potential to become a new thing." Crime did not originate in that century, but "it may be doubted whether there were crime rates" before then (p. 37). To the examples given by Porter we might add the relatively new "thing" in the domain of what used to be called "moral statistics"---the production of counts and rates of DGU (defensive gun uses, or users, according to context). Porter concludes that democratic politics gives rise to distrust of personal judgments and statistical methods are brought into play to nurture confidence where personal knowledge is an insufficient basis for political decisions. A somewhat paradoxical outcome is that "There is a strong incentive to prefer precise and standardizable measures to highly accurate ones" (p. 29). But it is arguable that the study of DGU is not yet at a stage where such a choice is necessary, or even possible. I shall point out some problems of communication that stand in the way of achieving either desidratum soon.

Let me begin with illustrations of the problem of public understanding of DGU statistics. The writer of a letter appearing on the editorial page of USA Today discusses the "order of actions that any prudent and decent person would take to thwart an assailant. As such, firearms are used---not necessarily fired---over 2.5 million times a year by law-abiding citizens in defense." This statistic is undoubtedly the one generated by the 1993 National Self Defense Survey (NSDS) (Kleck and Gertz 1995). The letter is valuable as a clue to what the public, or at least one of its sectors, might like to have by way of DGU statistics. But NSDS did not purport to count the number of DGUs in which prudent, decent, law abiding citizens are involved. At one point (p. 163) the authors were quite explicit: "We made no effort to assess either the lawfulness or morality of the Rs' defensive actions." As for prudence, they note that "This survey did not attempt to compare the effectiveness of armed resistance with other forms of victim self-protection" inasmuch as "this sort of work had already been done" (p. 174).

The data collected in NSDS could, conceivably, shed some light on the lawfulness and prudence of the defenders, taken in the aggregate. For example, some 64 percent of respondents indicated that the police were informed of the incidents in question, or otherwise found out about them. But the authors advise caution in the interpretation of this figure, "since victims presumably want to present their use of guns as legitimate" (p. 177); moreover, "a large share of the incidents covered by our survey are probably outside the scope of incidents...likely to be reported to...the...police" (p. 167). Further, as to the matter of "law-abiding" users, Kleck and Gertz emphasize that NSDS, unlike the National Crime Victimization Survey (NCVS), encouraged the reporting of acts that are "extremely sensitive and legally controversial," in that the defensive act in itself could be construed as "unlawful assault" and the mere possession of a gun at the time and under the circumstances of the DGU "might itself be unlawful" (p. 155). In the latter connection, Kleck and Gertz, noting that about one-fourth of defenders stated that there was no gun in their household at the time of the interview, drew attention to the possibility that some were falsely denying ownership of "incriminating evidence" (p. 177).

There is one bit of survey evidence rather directly to the point of whether DGUsers are "law abiding." The 1994 National Survey of Private Ownership of Firearms (NSPOF) (Cook and Ludwig, 1996, Table 6.7) included a direct question on whether the respondent had ever been arrested for a non-traffic offense. Affirmative answers were given by 20.5 percent of persons with a DGU in the preceding five years, by 7.8 percent of gun owners who had never had a DGU, and by 4.9 percent of non-owners of guns who had never had a DGU. A history of arrest does not necessarily imply, of course, that the particular incident reported as a DGU in the survey was itself unlawful.

In another letter appearing in the Santa Barbara News-Press the writer opines that "in the hands of ... well-trained, competent, morally responsible individuals firearms are the single best tool for defense against criminal victimization---a fact borne out by statistics, as they are used some two million times a year for this purpose. And the most significant point here is that only an extremely small portion of these involve actually firing the gun." It seems very likely that the number quoted and the "small portion actually firing" originate with John R. Lott, Jr., concerning whose work more will be said. Indeed, the timing of the letter (8/26/99) strongly suggests that the writer is relying on statements made in a TV appearance by Lott on 8/18/99. (I am indebted to Elaine Cressey-Ohlin for securing a tape of this show which I was thereby enabled to view.) Although Lott has repeated this claim concerning frequency and kinds of DGU in a variety of contexts, he has not, to my knowledge, asserted that the data mentioned in the TV show shed light on the training, competence, or moral responsibility of DGUsers.

Again, for a tidbit of direct evidence, one might turn to NSPOF, which did include questions about training in the use of firearms. However, so far as I know, neither Cook and Ludwig nor anyone else has produced a set of DGU rates for persons who have had various kinds of training, although it would take only a simple tabulation of the NSPOF data to create this information.

In sum, it is my impression, based on the items quoted and other reading, that there is a demand for statistics that are accurate, comprehensive, and carefully tailored to provide reliable information on how often trained, prudent, moral, law-abiding citizens use firearms to defend themselves effectively against indubitably criminal acts. Both advocates and opponents of gun control---that is, the "true believers" on both sides---will make the statistics mean what they want them to mean. Does this not suggest that researchers should lean over backwards to be as clear about what their numbers do not show as they are about the supposed policy implications of their findings?

Between the researcher and the general public there is often the third party, the writer or media personality who encapsulates and propagates information obtained by researchers. On a personal note, it was actually a column by George Will (Santa Barbara News-Press, 1/24/99; from The Washington Post) that drew my attention to DGU statistics. I had not seen the earlier (11/15/93) article by Will in Newsweek, where he wrote, quoting Jeffrey Snyder, "Florida State University criminologist Gary Kleck, using surveys and other data, has determined that armed citizens defend their lives or property with firearms against criminals approximately 1 million times a year. In 98 percent of these instances, the citizen merely brandishes the weapon or fires a warning shot. Only in 2 percent of the cases do citizens actually shoot their assailants." (Snyder's article is re printed in Dizard et al., 1999.) Will's later column had a curiously parallel construction: "[A] University of Chicago law professor, John B. Lott [actually, John R. Lott, Jr., then a Fellow, not a Professor, in the University of Chicago Law School] argues that Americans supplement police services and save municipalities large sums by using guns defensively against criminals 2 million times a year, 98 percent of the time just by brandishing guns."

Apparently Will is among those who put their trust in numbers. As of 1993, having been a proponent of gun control, Will was unpersuaded but nevertheless moved by Kleck's estimates, as summarized by Snyder. By 1999, having discerned a connection between gun control advocacy and judicial activism. Will was ready to assert that it is a "problematic" assumption that private ownership of handguns produces a net cost to governments. He did not pause to inquire how it may have happened that DGU per year seemingly had increased from 1 to 2 million or to note the shift in meaning of the category accounting for "98 percent" of users, from "merely brandishes the weapon or fires a warning shot" to "just by brandishing guns." Nor did he consider the possibility that one set of numbers may be more credible than the other. And thereby hangs the remainder of this tale.

As to the statistics cited in the 1993 article, there is a somewhat complicated story to tell. Snyder's source evidently was Kleck's 1988 article, from which we learn that the survey data came from "the 1981 Hart poll of 1,228 registered voters" (p. 2). It happens that this information has been preserved only in Kleck's files. The figures cited were given to Kleck in a telephone conversation with a member of the staff of Peter D. Hart Research Associates, Inc., in 1986; but "the original data set has been lost and only a record of the marginals remains." (For further details of this circumstance, see Blackman, n.d.) What Kleck had to work with was the distribution of responses to the question, "Within the past five years, have you yourself or another member of your household used a handgun, even if it was not fired, for self- protection or for the protection of property at home, work, or elsewhere, excluding military service or police work?" If "yes," then "Was this to protect against an animal or a person?" Some 2 percent replied "animal," 3 percent "person," and 1 percent "both," or altogether 4 percent against a person.

Applying the 4 percent to the 1980 census figure of 80,622,000 households, Kleck estimated there were 3,224,880 households in which at least one person used a handgun defensively in the five-year period preceding the survey. What he wanted, though, was an estimate for a one-year period covering uses of all guns, not just handguns. From other sources he estimated that in 1978 there were 21 million handguns and 33 million guns of all types owned primarily for protection or defense. Kleck divided 3,224,880 by 5 to obtain the estimate of 645,000 handgun defensive uses annually and multiplied by 33/21 = 1.57 to extrapolate to all gun uses and thereby "to very roughly estimate that guns of all types are used for defensive purposes about one million times a year."

Here and in the sequel it is well to keep in mind that the only direct evidence of DGU frequency in these calculations is the single-digit 4 percent of households. If Hart was following the usual conventions of rounding, that 4 percent could have been as low as 43/1,228 = 3.50+ percent or as high as 55/1,228 = 4.48 percent. That is, we have no basis for preferring the rough estimate of 1 million to the likewise rough estimates of 890,000 or 1.13 million. This is a matter of simple arithmetic, not of the probability theory relating to sampling variability. Any estimate in this range would itself be subject to uncertainty from this additional source.

By the time Snyder and Will were citing Kleck's estimate, he had revised it downward, to 783,000 +/- 177,000, in this presentation showing the 95% confidence interval as well as the point estimate (1991, p. 107). The change was due to a revision of the estimated ratio of all gun to handgun uses. The new ratio, 1.214, was derived from 1985 National Crime Survey statistics given in Kleck's Table 2.9 (not 2.11, as stated on p. 106). These data pertain to violent crime incidents, estimated by NCS as numbering 657,119 incidents in which the offender used any kind of gun, 541,271 using handguns. The assumption, then, is that defenders select their weapons in the same proportion as offenders.

Snyder, like Will following him, did not take note of Kleck's estimates (p. 107 and Appendix 4) from other surveys. Three of them yielded larger figures, one smaller, than the one based on Hart's data. Nor did they comment on the disclaimer that all the estimates required "adjustments" of the raw survey percentage, and "these numerous adjustments are all subject to question" (p. 468).

I have read that old Roman coins were still circulating in parts of rural France as late as the 19th century. Statistics, too, have a life of their own. I have seen the 1 million estimate within the past year in a letter to the editor. Meanwhile, Kleck has revised it yet again, to no less than 1,797,000. In part, the increase is attributable to population growth. Whereas the two earlier figures developed from the Hart result were projected to the number of households in 1980, the one put out by Kleck and Gertz in 1995 (but see Kates and Kleck, 1997, pp. 187-190 for explanation of the adjustments, which differ from those described on p. 158 of Kleck and Gertz 1995) was projected to the census household estimate for 1993, for comparability with the NSDS results. Had Kleck projected again to the 1980 base, the DGU count would have been 1.5 million, not quite twice the size of the 1991 estimate. There was a small revision of the ratio of all guns to handguns, changing 1.214 to 1.2547, the latter derived from the sample counts in NSDS. But the more important revision was in the derivation of the annual from the adjusted 5-year estimate. Again, NSDS provided the adjustment factor, DGUs in the past year divided by DGUs in the past five years, .37155. No detailed argument supporting this choice was offered, but Kleck observes (1995, p. 172) that NCS results suggest that "the ratio of memory loss errors over telescoping errors increases as the recall period lengthens." (Cook and Ludwig, 1996, Table 6.2 present two sets of one-year and five-year DGU frequencies that yield ratios close to Kleck's. These pertain to counts made on a person rather than household basis.)

Regardless of the plausibility of Kleck's adjustment factors, I would find the comparison of Hart and NSDS results more convincing if Kleck had simply confined himself to observing that Hart found 4 percent (understood as 3.5-4.5 percent) of households experiencing defensive handgun use in the five-year period preceding the 1981 survey, as compared with 2.974 percent of households in the 1993 NSDS. Given the uncertainty in regard to what the Hart survey actually found, not to mention sampling variability in both surveys, there is no difference worth discussing. It follows that the apparent increase in DGU frequency, from Kleck's 1 million in 1981 to his 2.5 million in 1993, apart from the contribution of population growth, is explained by the shift from the five-year period, household basis, of reporting in Hart's 1981 survey to the one-year period, person basis, in the 1993 NSDS.

Kleck's figure of 98 percent of DGUs in which the "citizen merely brandishes the weapon or fires a warning shot" was not derived from survey evidence and thus lies outside the scope of this essay. In the 1991 presentation he noted that his estimate of 10,000-20,000 legal shootings of criminals by civilians would amount to "less than 2% of all defensive gun uses" (p. 116). In the recent review of this topic Kleck (1997, p. 164) noted that "reported legal shootings of criminals ... would be less than 1% of all DGUs. The vast majority of DGUs, then, involve neither killings nor woundings but rather misses, warning shots fired, or guns used to threaten, by pointing them or verbally referring to them."

Turning to the figure of 2 million DGUs attributed by Will to Lott, there are no technical details to discuss be- cause Lott has provided none. We might conjecture that this number pertains to persons, rather than households, but this is not specified in any of the public statements quoted below, which include all that I am aware of. (I am indebted to Dan Armstrong for locating Lott's newspaper articles reproduced at www.tsra.com/LottPage.htm, updated as of 3/28/99.)

1. "If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack." Lott's book (1998), p. 3. "Fifteen national polls, including those by organizations such as the Los Angeles Times, Gallup and Peter Hart Research Associates, imply that there are 760,000 to 3.6 million defensive uses of guns per year. "(p. 11)
2. "Polls by the Los Angeles Times, Gallup and Peter Hart Associates show that there are at least 760,000, and possibly as many as 3.6 million, defensive uses of guns per year. In 98 percent of the cases, such polls show, people simply brandish the weapon to stop an attack." Chicago Tribune, 8/6/98.
3. "...the over 2 million times each year that Americans use guns defensively." Star Tribune Minneapolis), 8/16/98.
4. "98% of the time when people use a gun defensively, merely brandishing the weapon is sufficient to stop an attack. In less than 1% of the cases is a gun even fired directly at the attacker." Wall Street Journal, 11/11/98.
5. "Americans use guns defensively more than 2 million times a year and 98 percent of the time merely brandishing the weapon is sufficient to stop an attack." Chicago Tribune, 11/17/98.
6. "Americans also use guns defensively about 2.5 million times a year, and 98% of the time merely brandishing the weapon is sufficient to stop an attack." Los Angeles Times 12/1/98.
7. "People used guns defensively to stop violent crimes over 2 million times in 1997....98 percent of the time, when people use guns defensively, simply brandishing a gun is sufficient to cause a criminal to break off an attack. In less than 2 percent of the time is the gun fired....About three-quarters of those are warning shots." TV show, "Hardball," CNBC, 8/18/99.
8. "...in 1997, Americans also used guns to defend themselves against attackers about 2 million times." Newsweek, "Letters," 9/13/99.

We see that the 98% brandishing figure is variously associated with or attributed to "national surveys"; to polls by three named agencies and, by implication, as many as fifteen national polls; and to unspecified source or sources which estimate DGU frequency at "about 2 million" or "more than 2 million" or "about 2.5 million." In none of the statements is any specific survey or other source cited. In only the most recent statement does Lott give the date, 1997, for the statistics cited. This is later than any of the "fifteen national polls." It is especially noteworthy that Lott does not credit Kleck with the estimate of 2.5 million. In my casual perusal of the letters columns I have seen that estimate credited to Lott himself at least three times. Most recently, in Business Week, 9/6/99, the writer states, "According to the Lott study ... the legal use of firearms by private citizens prevented 2.5 million crimes, such as rape, robbery, assault, and murder." (Never mind that Kleck does not restrict his respondents to reports of "legal use" and that no more than three-fifths of the incidents reported in NSDS pertained to such violent crimes.)

Lott does not distinguish between household-based and person-based DGU counts. Hart, as we have seen, reports for households, Gallup and Los Angeles Times for persons.

The "fifteen national polls" evidently include the thirteen summarized by Kleck and Gertz (1995), which is referenced by Lott, and by Kleck in Kates and Kleck (1997). The summary in Kleck (1997) adds to these the NSDS and NSPOF results, to make fifteen altogether. As so often happens, Lott's citations are elliptical or inaccurate.

There is a mixture of household- and person-based figures in this collection. Only twelve of the fifteen are national, the others being surveys in California, Illinois, and Ohio. Kleck does not attempt to provide comparable estimates of DGU frequency for two of the surveys. None of the three polls Lott mentions by name---Hart, Gallup, and Los Angeles Times---includes information on percent firing weapons. Altogether, there are five polls providing estimates of percent firing that include DGUs against animals. Not surprisingly, these percentages are far higher than the 2% or "less than 2%" implied or stated by Lott.

The only two surveys among the "fifteen national polls" that provide estimates of percent of defenders firing, with defenses against animals excluded, are NSDS and NSPOF. The figures based on the most recent DGU in the past five years are remarkably similar: 23.9% in NSDS and 27% in NSPOF. For the one-year reporting period, Kleck simply assumes the same percentage. In a personal communication he states that the higher rate of firing implied by the NSDS figures for the one-year period in Table 5.1 of Targeting Guns is in error. In that source, he shows for NSPOF a one-year estimate of 1.44% using guns, 0.70% firing, during the preceding 12 months; this implies that 48.6% of defenders fired. I do not find this particular result in the Technical Report on NSPOF, but Table 6.10a therein shows 8 of 19, or 42%, of defenders firing, in a sample that has been purged of what Cook and Ludwig regard as questionable reports of DGU.

In addition to the nongovernmental surveys there is one other survey, NCVS, that collects relevant information. Both Kleck and Lott (who follows Kleck, but invents a non-existent defect in the NCVS sample design) strongly criticize that survey for what they regard as an extreme undercount of DGU. But Kleck has nonetheless cited NCVS data repeatedly to support the claim that gun use is the most effective means of self-defense against robbery and assault. While he did not have data on how many NCVS victims actually fire, that is surely relevant to the question of effectiveness. Inasmuch as NCVS does not routinely publish data on gun use by victims, it has been left to individual investigators, like Kleck, to create the necessary tabulations. McDowall and Wiersma (1999, Table 26.2) provide data on the estimated 258,460 uses by persons in the population covered by NCVS from 1987 through 1990. These figures pertain not only to victims of violent crimes but also victims of personal larceny, burglary, household larceny, and motor vehicle theft. Altogether, 27.7% of gun users said they fired their weapon.

Thus, the three largest, most comprehensive, and technically most sophisticated surveys in this domain are unanimous in being radically inconsistent with Lott's claim that 98% of defenders "merely brandish" their weapons. Not only that, he rebuts his own assertion quite effectively. On p. 190 of his book, he takes note of Kleck's observation (Targeting Guns, p. 162) that "Data from the NSDS indicate that no more than 8% of the 2.5 million annual DGUs involved defenders who claimed to have shot their adversaries, or about 200,000 total." Lott does concede that Kleck and Gertz (1995, p. 173) express reservations about the rate of wounding. Nevertheless, this estimate "seems somewhat plausible." How "plausible" is it, if only 2% of defenders fire, and fire mostly warning shots at that? Lott's figure accounts for no more than 25,000 defenders (1% of 2.5 million) intending to shoot their adversaries.

There is at least one more major problem with Lott's figures. In the TV presentation, he specified that people defended against violent crimes over 2 million times. NSDS data (Kleck and Gertz, 1995, Table 3), show 20.5 percent defending against robbery, 8.2 percent against rape or sexual assault, and 30.4 percent against other assault. The total of 59.1 percent is an upper limit on the proportion who thought they were becoming victims of a violent crime; the three categories are not mutually exclusive. Either Lott restricted his inquiry to defenses against violent crimes, or the total of DGUs in his data must considerably exceed 2 million, or his study design is quite different from that of NSDS. Moreover, Lott regularly describes the outcome of a DGU as stopping or breaking off of an "attack." According to NSDS data, there is "no threat or attack" in 46.8 percent of DGUs; 32.3 percent of defenders were "threatened only." Only 20.8 percent were actually "attacked," including the 5.5 percent who were injured. NSPOF data (Cook and Ludwig 1996, Table 6.6) likewise show that 46.1 percent of defenders were neither threatened, attacked, or injured. It is hard to see how one can "break off an attack" when there has not even been a threat.

I infer that the National Rifle Association does not accept Lott's figure of 98% merely brandishing. In response to my direct inquiry about that figure, they sent me a selection of Fact Sheets and leaflets. One of them, while endorsing Lott's study of the effects of Right to Carry laws (i.e., his 1998 book), does not mention this statistic. I also received the NRA leaflet by Blackman, summarizing the method and findings of NSDS and reproducing Kleck's summary (1997, Table 5.1) of the fifteen nongovernmental surveys, with slight editorial changes and one typographical error. Not a word about Lott's 98 percent merely brandishing. Four illustrations, in color, leave little doubt that NRA members are expected to shoot, if necessary.

I cannot forego a remark pertaining to rhetoric rather than statistics. In 1988 Kleck (p. 5) wrote that "over 98 percent ... of the one million estimated defensive gun uses ... involved neither killings nor woundings but rather warning shots fired or guns pointed or referred to." In 1993, Snyder paraphrased thus: "In 98 percent of these instances, the citizen merely brandishes the weapon or fires a warning shot [emphasis added]" (in Dizard et al., 1999, p. 189). In 1993 Will copied that language. Lott may well have read Will, inasmuch as Will's article is in the bibliography of More Guns, Less Crime. In that book, Lott wrote, "If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack [emphasis added]." Did Lott borrow the "98 percent" from Kleck - radically changing its meaning, to be sure, and erroneously attributing it to unnamed "surveys"---and the language, "merely ... brandishing," from Snyder, via Will? Even if that account explains part of the puzzle, the question remains, Where did the 2 million come from? In this discussion 1 have not reviewed the polemical literature criticizing gun-use surveys and rebutting the criticisms. Items in point, with ample references to others, are Cook, Ludwig, and Hemenway (1997) and Kleck and Gertz (1997). Let me simply observe that it is a sad day when parties to a scholarly controversy (not the ones just mentioned), writing for a Journal with impeccable academic standards, have to exchange compliments on their opponents' politeness and grace or restraint and professionalism. I mention three important issues not emphasized in this literature.

1. The black hole of nonresponse, including large numbers of outright refusals to be interviewed in telephone surveys, is large enough to swallow the expressed concerns about false positives and false, negatives. Completion rates do not measure the biases of nonresponse but only indicate the possibility that they exist. Everything depends on the nature and magnitude of the biases, which may vary greatly from one item to another. Does anyone have a clue about them, or a way to learn more?
2. The larger surveys, NCVS, NSDS, and NSPOF, have sample sizes that would support the discriminating multivariate analyses desperately needed to infer---to take but one example among many---the reasonableness of gun defense in the light of the defender's estimate of the seriousness of the perpetrator's intention. I have mentioned that substantial fractions of DGUsers are defending against nonviolent crimes and experience no threat of attack. NSDS and NSPOF agree that only about one-fifth of perpetrators had a gun. Approximately half of defenders in both surveys confronted a single offender. Wouldn't it be interesting to know whether the roughly one-quarter of defenders who fire their guns are disproportionately represented among those threatened or attacked, those expecting to become victims of a violent crime, those facing armed offenders, those confronting two or more perpetrators, or some combination of these hazardous situations? Why doesn't NCVS regularly give us the cross-classification of offender's weapon use by victim's mode of self-protection?
3. There is an unfortunate fixation in the statistical part of the gun debate on the question of "How many DGUs?" The tacit assumption on both sides is that there is some true number such that any quoted estimate of it can be criticized as too low or too high. But isn't it obvious that the number of events, of almost any kind, depends in the first instance on how the event is defined and delimited? Of course, there is a vital distinction between the nominal definition---what the investigator has in mind---and the operational definition, which can be specified fully only in terms of the entire complex of survey procedures that elicit information from informants. Kleck and Gertz (1995, pp. 157-8) offer a valuable assessment of the heterogeneity of the 13 surveys listed in their Table 1. The discussion is summarized in the sage remark that "each of the surveys ... was measuring something different." The moral of the story is then taken to be that all the several "flaws" in previous surveys could be overcome with a new design, to wit, that of NSDS. But that design too incorporates its own definition, not some manifestly correct definition of DGU. To mention a single facet of the matter, the wording of the questions put to NSDS respondents excludes defense against animals. One can certainly imagine points of view from which that exclusion seems like biasing the question in the interest of reducing the DGU count. Or take Kleck's remark that the question used in the Cambridge Reports 1978 survey---which asked handgun owners if they "ever had to use" their weapon defensively---was "oddly worded" (Kates and Kleck 1997, p. 191). He was overlooking the same phrasing in two other surveys, Bordua (Illinois) 1979 and Ohio 1982. This wording could certainly bear upon issues made salient in the gun debate, "prudence," for example. Of course, it is no simple matter to isolate the effect of question wording on response frequency. My only point here is that arguments about study design should be recognized as arguments about the concept and its theoretical, practical, or even rhetorical relevance, and not be presented as assessments of mere techniques, all of which are supposedly trying to get at the "same phenomenon" (Kleck and Gertz, 1995, p. 153) and succeeding in greater or lesser degree. At this stage of the game, with no small accumulation of experience, that naive idea is no longer helpful.

It was W. Edwards Deming, the guru of both statistical quality control and sample survey methods, who argued tirelessly against the idea of a "true value" of any quantity, definable without reference to the method of measuring it. To get back to fundamentals, we should periodically review the chapter on errors in surveys in Deming's classic text, now half a century old, recalling such cautions as these: "A sample is no longer a probability-sample if it is ruined by nonresponse or any other difficulty of execution" (p. 35). "...a bias...is not necessarily partially or wholly compensated by another bias in the opposite direction" (p. 38). "Sampling errors, even for small samples, are often the least of the errors present" (p. 47). And, above all, the chapter's epigraph: "It is far easier to put out a figure than to accompany it with a wise and reasoned account of its liability to systematic and fluctuating errors. Yet if the figure is ... to serve as the basis of an important decision, the accompanying account may be more important than the figure itself." Porter's monograph reminds us of the long, painful social process that was required for such principles to evolve and to be widely accepted. We jettison them at the price of relinquishing any plausible claim to the support of the society in which we work.

I am most grateful to Lloyd Ohlin for his wise and gentle counsel supporting my work on this essay, and to Gary Kleck, who patiently answered a myriad of questions about the research domain in which he was a pioneer.

References

Blackman, Paul H.
n.d. "Armed Citizens and Crime Control" (leaflet, 4 pp.). Fairfax, VA: National Rifle Association.

Cook, Philip J., and Ludwig, Jens
1996 Guns in America: Technical Report. Washington: Police Foundation.

Cook, Philip J., Ludwig, Jens, and Hemenway, David
1997 "The Gun Debate's New Mythical Number: How Many Defensive Uses Per Year? Journal of Policy Analysis and Management, 16:463-469.

Deming, W. Edwards
1950 Some Theory of Sampling. New York: Wiley.

Dizard, Jan E., Muth, Robert Men-ill, and Andrews, Stephen P., Jr.
1999 Guns in America: A Reader. New York: New York University Press.

Kates, Don B., and Kleck, Gary
1997 The Great American Gun Debate. San Francisco: Pacific Research Institute for Public Policy.

Kleck, Gary
1988 "Crime Control Through the Private Use of Armed Force." Social Problems, 35: 1-21.
1991 Point Blank: Guns and Violence in America. New York: Aldine de Gruyter.
1997 Targeting Guns: Firearms and Their Control. New York: Aldine de Gruyter.

Kleck, Gary, and Gertz, Marc
1995 "Armed Resistance to Crime: The Prevalence and Nature of Self-Defense with a Gun." Journal of Criminal Law and Criminology, 86: 150-187.
1997 "The Illegitimacy of One-Sided Speculation: Getting the Defensive Gun Use Estimate Down." Journal of Criminal Law and Criminology, 87: 1446-61.

Lott, John R., Jr.
1998 More Guns, Less Crime. Chicago: University of Chicago Press.

McDowall, David, and Wiersma, Brian
1999 "The Incidence of Defensive Firearm Use by U.S. Crime Victims, 1987 through 1990." Reprinted in Dizard et al. (1999), from American Journal of Public Health, 84: 1982-84(1994).

Porter, Theodore M.
1995 Trust in. Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton: Princeton University Press.