0. Introduction
Volume 87:4 of the Journal of Criminal Law and Criminology contains
three articles on the issue of the frequency of defensive gun use.
The first presents David Hemenway's critique of Gary Kleck's 2.5
million estimate, the second is Kleck and Gertz's reply and finally
Tom Smith of the National Opinion Research Center comments on both
papers.
I'll try to summarize the arguments and comment where I think they are
wrong.
1. Hemenway's critique
1a. False Positives
Hemenway's critique has two main arguments. The first is the problem
of false positives. Let me define some terms first:
FPR is the the false positive rate (the percentage of people without a
DGU who claim to have had one).
FNR is the false negative rate (the percentage of people with a DGU
who deny having had one).
P is the number of true positives (people with a DGU, whether or not
they admitted it).
N is the number of true negatives (people without a DGU, whether or
not they claimed to have had one).
FP=FPRxN is the number of false positives.
FN=FNRxP is the number of false negatives.
Even if FNR is 38 times greater than FPR, Kleck's estimate is too high
by a factor of 17. This is because what determines whether an
estimate is too high or too low is the relative sizes of FP and FN,
not the rates. Even if Kleck is right and DGU is common, the great
majority of people have not had a DGU in the past year. N is at least
75 times P and hence FP is larger than FN.
To put it another way, even if FPR is quite small (1.3%), Kleck's
would be an extreme overestimate. Hemenway then gives several
examples of surveys (for example, on personal contact with space
aliens) with FPRs of this magnitude or higher.
1b External validity
The false positive argument does not prove that Kleck's number is an
extreme overestimate. It merely demonstrates that it is probable.
Hence we come to Hemenway's second argument: All checks for external
validity show that the estimate is highly exaggerated.
The checks that the Kleck estimate fails are with:
NCVS estimates of burglaries,
Atlanta study on home invasion crimes,
NCVS estimates of sexual assaults,
NEISS estimates of gunshot wounds, and
UCR counts of homicides.
These arguments allow Hemenway to conclude that Kleck's estimate is
not reasonable.
2. Kleck and Gertz's reply.
They open with an ad hominem argument: Don't trust Hemenway - he's
associated with HCI. I hope I don't need to point out what is wrong
with this argument. Kleck and Gertz are rather heavy with the
rhetoric throughout their reply. For example, they complain about the
"illegitimacy" of Hemenway's "idle speculation". They assert that
Hemenway's critique is neither honest nor scientifically-based. They
claim that the NCVS based DGU estimate is "dead" and Hemenway is the
only one left who believes it isn't.
2b False positives
Anyway, let's consider their response to Hemenway's "false positive"
argument. Kleck and Gertz concede that there may be some false
positives but argue that there are even more false negatives.
Hemenway finds a FPR of 1.3% and a FNR of 50% plausible. Kleck and
Gertz counter by stating that a FPR of 0.2% and a FNR of 50% is more
realistic. It seems that they only disagree about the FPR. Kleck and
Gertz argue that:
-
Most DGUs involve illegal behaviour on the part of the defender
-
Therefore the FPR and the FNR will be similar to that for other
illegal behaviour such as illegal drug use. -
Studies of illegal drug use show that FN is much greater than FP.
There are some problems here:
-
Kleck and Gertz offer no evidence for their claim that gun
ownership is usually illegal and hence most DGUs involve some
illegality. -
Even if DGUs are usually illegal it does not follow that the FPR
will be similar to that for behaviours that are always illegal. If
you are inventing a DGU it as possible to invent one that involves no
illegal behaviour on your part. This is not possible for illegal drug
use. -
Kleck and Gertz present two studies that measured the FPR for
illegal drug use. Pooling them, I find that the FPR was 1 in 70 or
1.3%. That is exactly the rate suggested by Hemenway and much greater
than the rate that Kleck and Gertz claimed the studies supported. The
reason why FN was greater than FP in these studies was that over half
the population studied (convicts) were users of illegal drugs. Not
even Kleck believes that over half the population of the US has a DGU
each year.
To summarize: even the data presented by Kleck and Gertz on the "false
positive" question supports Hemenway's position.
2b External validity
Kleck and Gertz crank up the rhetoric even further here. They assert
that checks for external validity "have repeatedly confirmed our
estimates". They claim that Hemenway's "fallacious argument" is based
on what is merely a "misperception" of inconsistency.
Anyway, Kleck and Gertz's resolution of the inconsistency between
their survey and the NCVS counts of crimes such as burglary is quite
simple -- the NCVS is wrong and their survey is right. They argue
that DGUs typically involve unlawful gun possession and hence that the
crimes defended against will not be reported to the NCVS at all.
There are some problems with their argument:
-
Kleck and Gertz offer no evidence for their claim that gun
ownership is usually illegal and hence most DGUs involve some
illegality. -
64.2% of their DG users said that the police were informed of the
incident. If they are going to tell the police (who might arrest them
for illegal behaviour) about it, why wouldn't they tell the NCVS (who
can't arrest them) about it? -
Kleck and Gertz included a question about burglary in their survey.
The estimate of the frequency of burglary that you get from this
agrees with the NCVS count. Apparently we are supposed to believe
that because respondents don't want their illegal DGU to be found out
by Kleck and Gertz's interviewers they refuse to mention the burglary
during which it occurred. But that they will tell the interviewers
about this illegal DGU when asked if they've had a DGU. -
Even if we accept the dubious premise that none of Kleck and
Gertz's DGUs against burglary are counted in NCVS burglaries,
Kleck-Gertz's result is still inconsistent with the NCVS. Instead
of an impossible 80% of "at-home" burglaries involving a DGU, we get
an impossible 40%.
Kleck and Gertz also claim that Hemenway's logic is fallacious in
drawing implications about the validity of the survey from estimates
(such as that for the number of DGUs against burglary) made from
smaller subsets of the data. They claim that the estimate of the
total number of DGUs is based on a very large sample (n=4977) and
hence is reliable, while the estimate for DGUs against burglaries is
based on a far smaller subsample (194 DGUs, 40 against burglars) and
hence is less reliable. Kleck and Gertz forget to mention that their
2.5 million total DGU estimate is based on the 67 out of the 194 DGUs
that were personal and occurred within the past year. Since 40 is less
than 67, the estimate for DGUs against burglary is indeed less
reliable, but not by much.
In any case, whether an estimate is more or less reliable than another
estimate is irrelevant. What is important is whether or not the
estimate, after allowing for its unreliability, is inconsistent with
the external check. A 95% confidence interval for the estimate of the
number of criminals shot (the least reliable of the estimates) is
100,000-300,000. The low end of that estimate is still wildly
inconsistent with NEISS counts of gunshot wounds.
Kleck-Gertz's final counter to the external validity argument is to
claim that there is no inconsistency with the NEISS count of gunshot
wounds treated in hospitals because criminals would avoid treatment
for minor DGU related wounds. However, the NCVS indicates that 90% of
assault gunshot wounds are treated in a hospital, and the NEISS shows
that about half of gunshot wounds are serious enough to require
hospitalization. Consequently there should have been about 90,000 DGU
related woundings serious enough to require hospitalization. Needless
to say, this is still inconsistent with the NEISS.
3. Smith's comments
Smith's article (entitled "A call for a truce in the DGU war")
attempts to reconcile the Kleck-Gertz and NCVS estimates for DGUs.
3a NCVS estimate of DGUs
Smith believes that the NCVS undercounts DGUs for two reasons. The
first reason is that DGUs against activities like trespassing are not
counted. The second is that it does not ask a direct question about
gun use. Smith dismisses Kleck and Gertz's speculation that people
would be more likely to tell their interviewers about sensitive
incidents than Census Bureau employees, noting that the survey
literature does not support their speculation.
3b Kleck-Gertz estimate of DGUs
Smith believes that Kleck-Gertz over-counts DGUs for several reasons.
The first reason is telescoping. While this would be partly balanced
by forgetting, Smith cites studies that indicate telescoping is
greater than forgetting. The second reason is that the other studies
that roughly agree with Kleck-Gertz tend to give lower estimates. The
third reason is false positives, though Smith notes that neither Kleck
and Gertz nor Hemenway have provided conclusive evidence on what the
false positive rate would be, so it is hard to quantify this effect.
The fourth reason is all the internal and external inconsistencies of
the Kleck-Gertz survey (too many DGUs by respondent as opposed to
other household members, too many DGUs within one year as opposed to
within five years, too many woundings, too high a proportion of DGUs
by women, 20% of DGUs occurring in gun-free homes and so on.)
3c Resolving things.
Smith states that more research is needed to resolve the issue and
suggest that adding a few questions to the NCVS would be the best way
to do this.