Scholars cannot replicate Lott's results

Despiting having no supporting data, Lott claimed over and over and over again that merely brandishing a gun was sufficient to scare off a criminal 98% of the time. In 2002, he conducted a survey that he claims gave a very similar number -- 95%. But nobody can replicate this result. And by "replicate" I mean using exactly his data and the methods he said he used. You can read about my attempt to replicate here and also download a spreadsheet with my calculations. (You have to get the data from Lott because he doesn't allow redistribution of his data.)

And I'm not the only one. In
a peer reviewed article(subscription required) in Public Opinion Quarterly David McDowall reports on Lott's 2002 survey:

Among the 1,015 respondents, 7 persons reported a total of 13
incidents of armed defense. Given the goal of Lott's research, the
incident count is of greater relevance than the person count. For the
13 reported defensive incidents, 12 involved only brandishing. This
implies a 92 percent brandishing rate, less than the 98 percent or 95
percent figures that Lott has cited.

Lott (2003a, p. 259, 2003c) says that "the estimates were weighted by
race (black, white, other) and gender," and this may be responsible
for the difference between the sample value and his larger
figures. Table 2 presents estimates with weights for the six
combinations of race and gender that Lott apparently applied. [A footnote here says that Lott "did not respond to an e-mail asking him about the weights he used." I've asked him this as well, also without response.] This
produces a brandishing rate of about 91 percent, almost the same as
from the unweighted data.

In commenting on an earlier version of this review, Lott said that he
did not employ the weights that he had described in his written
accounts. Instead, he reported that he used "national weightings by
age, gender, and race for the age groups from 10 to 19, 20 to 29, and
so on." This alternative weighting method is obviously not a good fit
to the youngest group in his survey, which includes only a handful of
respondents aged less than 20 years. These respondents likely appeared
in the sample by mistake, but they would receive large weights because
of their rarity. An analysis that I conducted using this scheme
nevertheless produced a 93 percent brandishing rate, a value similar
to those from the other methods. ...

Many of Lott's statements during 2003 and 2004 strongly suggested that he had revised his brandishing estimate downward. In an article in the American Rifleman, for example, he wrote, "In 2002, some 90 percent of the time when people used guns defensively, they stopped the criminals simply by brandishing the gun" (Lott 2004, p. 60). In commenting on an earlier draft of this review, however, Lott strongly affirmed his continuing support for the 95 percent figure.

More like this

Lott is full of shit. It's extraordinary that people keep on quoting him. Suing Steven Levitt because he said, quite reasonably, that Lott's results haven't been able to be replicated, just shows what a contemptible piece of work Lott is.

And all this ignores the central flaw of his survey: finding out how successful a method of defense is by calling people up and asking them if they weren't maimed or killed while using it, is really, really dumb.

There does seem to be something akin to a self-selection bias going on there.

If someone does branish a weapon and it is unsucessful how is it reported? Since that event also carries the possobility of the death of the victim do you need to weight the results to account for it?

Can it be accounted for, and if not would that invalidate the results of the survey?

"I prefer to watch the criminal cowards scurry on surveillance videos when their intended target turns out to be armed. All the evidence I need."

Do you watch all the videos that show crimes fail when victims aren't armed? Oh, wait, those aren't as exciting and wouldn't meet your pre-conceived notions.

All this assumes that the scared off were indeed criminal.

Youtube has all the evidence I need to prove that Guns are awesome.

Ben isn't a parody troll, is he?

All this assumes that the scared off were indeed criminal.

In the video I posted they are indeed criminals. Doesn't prove a damn thing, but it is interesting.

ben comments like American science fiction writers have their characters talk.

In ben's imagined world, IMO, scientific papers are peer-reviewed for self-satisfaction and machismo. The final word on publication comes from the RNC or Cato.

By Marion Delgado (not verified) on 13 Feb 2009 #permalink

Marion Delgado writes:

ben comments like American science fiction writers have their characters talk.

I'm an American science fiction writer, and my characters don't talk that way.

I think the larger point Ben is alluding to (albeit crudely), is that Lott's sloppiness aside, even the lowest figure mentioned in the artcile (90%) is enough to blow out of the water all these "a gun is more dangerous to the carrier" arguments. You the ones I'm talking about, those that pretend the scenes in movies where the bad guy always takes the gun away from the innocent person represents reality. The bad guy running away is too boring (except of course when it follows "My name is Inigo Montoya...")

You actually believe the 90% number? I figure if we wait a while Lott will be retreating to 85%, 80%, etc.

And his claimed survey results point out the rarity of good guys pulling guns on bad guys, even as self-reported. Gun accidents seem to be unfortunately more common.

I suppose if you went through enough footage you'll find almost any outcome. And I'd certainly run like hell if someone pointed a gun at me.

But people with self-defense fantasies (and they sure sound fantastic) seem to have seen too many old movies. If the bad guy gets the 'drop' on me, I'm done, gun or no gun.

Of course there are people who should carry a gun. But considering the number of police who are shot with their own guns, I'm disinclined to think I could get the drop on a criminal. Better to use my judgment to steer clear of such confrontations.

You can't just go carrying a gun around without knowing how to use it, and then expect it to magically save you in a desperate situation. You've got to know some self-defense tactics (and practice them) before you really have a fighting chance when mugged or under the gun in some way. Also, if you are very well-trained, you could save someone else's life, which is a bonus.

By defenseless and dead (not verified) on 16 Feb 2009 #permalink

I don't think the critics are getting it. Self-reporting by Iraqis of violent deaths in their families is unreliable; they are obviously making stuff up to please the investigators or let off steam. Self-reporting by American gun-owners of incidents of heroic self-defense are reliable because they are American gun-owners.

Scholars are a bunch of homos.

If they really wish to replicate John's results they can hang around high crime areas with a concealed firearm and wait for things to happen. After 100 such incidents they will realize that in 98 cases simply brandishing the firearm will end the crime.

This comment is unrelated to this specific post, but it is related to Lott. I was actually looking for some information on a person Lott cited on his blog and, without typing anything about Lott, came up with this site. It's good to know someone else thinks John Lott is an idiot.

It is wrong to encourage gun owners to 'brandish' their weapons.

If you have a gun, and you draw it, you should be prepared to use it.

The other guy is just as likely to pull a gun and shoot you, or rush you, if you brandish a weapon.

Lott is guilty of encouraging gun owners in very bad and dangerous habits.

Excellent discussion on Making Light blog last year, by a former US Marine, advising women in a divorce-law attorney's firm (as you can imagine, they get a lot of potentially violent, and certainly irate, exes storming their office) as to whether they should own guns.

His advice: train like the blazes, and if you pull, use.

People who brandish weapons are fools and Lott criminally negligent to suggest they act in that way.

By Valuethinker (not verified) on 01 Mar 2009 #permalink

I just noticed Mary Rosh commented here. That's John Lott's pseudonym.

And that also explains how Lott got his results. He went to a high crime area with a concealed weapon and waved it around.

By Steven Kippel (not verified) on 22 Dec 2012 #permalink

Yes, but the "Mary Rosh" was just making fun of Lott.

By Tim Lambert (not verified) on 22 Dec 2012 #permalink