Dubious figures about guns

Carl Bialik, the Wall Street Journal‘s “Numbers Guy” writes about some of the dubious numbers used by both sides on the debate about guns.

This one should be familiar to my readers:

Another number that has emerged from the antigun-control camp ties multiple-victim public shootings to restrictions on carrying concealed weapons. John Lott Jr., visiting professor at SUNY, Binghamton, and University of Chicago economist William Landes counted references to multiple public shootings — more than one killed or wounded at one time — in the Lexis/Nexis news database for a 2000 book. They matched trends from 1977 to 1999 with right-to-carry laws, and found that when states allowed the carrying of concealed weapons, the rate of these attacks declined by 60%.

But another study, published in 2002 in the journal Homicide Studies, found “virtually no support for the hypothesis that the laws increase or reduce the number of mass public shootings.” This later study counted only shootings with four or more murders, used FBI crime data to supplement news reports and, unlike the Lott-Landes work, included shootings that were byproducts of other crimes, such as gang murders.

Grant Duwe, a researcher on the later study, said the news-archive approach was likely incomplete, because the media don’t always give publicity to multiple shootings.

Prof. Lott wrote in an email that he counted less-severe incidents to get enough data for statistically significant results. He justifies his exclusion of gang murders because gun usage by chronic criminals “would not be directly affected by the passage of right-to-carry laws.”

That seems to be precisely the reason to include them for a full picture of the effect of these laws. Of course, the complete picture frequently goes missing in this debate.

Bialik has further discussion on his blog.

Comments

#1 ben
April 20, 2007

He justifies his exclusion of gang murders because gun usage by chronic criminals “would not be directly affected by the passage of right-to-carry laws.”

That seems to be precisely the reason to include them for a full picture of the effect of these laws. Of course, the complete picture frequently goes missing in this debate.

Depends if the murder victims of gang members are gang members themselves, or otherwise innocent victims. Gang members tend to carry weapons all the time, and so gang on gang violence could not be affected by concealed carry laws.
#2 ben
April 20, 2007

And I forgot to include, that if I’m not mistaken, most victims of gang murder are themselves gang members. Hence the conclusion above is wrong.
#3 Rob
April 20, 2007

Ahh Ben. Throwing out a bunch or murders because they won’t be affected by law changes does bias the results. Imagine you have two pots of ink. One is all white and the other all black. Now you add black ink to both. The white ink turns gray while the black ink stays black. Since they black ink is unaffected, you throw that out and say 100% of the ink turns gray!
#4 Brian Schmidt
April 20, 2007

Bialik is saying that it’s incorrect to say the rate of attacks decline by 60% when you’re only examining a subset of the overall level of attacks.

I’m curious whether this limitation in the Lott study was disclosed by Lott in advance, or if it’s just his first point of retreat once the limitation was exposed by someone else’s legwork. I suspect there are quite a few more flaws in anything that Lott produces.
#5 mario
April 20, 2007

I am so embarrassed that my alma mater, SUNY Binghamton, has hired John Lott as a visiting prof.

the shame,
the shame…
#6 Robert S.
April 20, 2007

The trouble with trying to analyze any of this is just like trying to analyze any complex system you can’t experiment on.

What happens if you elimate or double something with everything else equal at some snapshot momment to really know what’s going on? We can’t answer it with right-to-carry any more than we can with CO2. Trying to make sense of complex systems of any kind is difficult at best — I’m not surpised anything gives us dubious numbers. Too inexact.
#7 LogicallySpeaking
April 21, 2007

“Trying to make sense of complex systems of any kind is difficult at best — I’m not surpised anything gives us dubious numbers. Too inexact.”

Translation: It seems really complicated so I guess we can’t know much about it.

Is this your stance on everything? Just because you’re unable to do something, or unable to think of how it might be done, doesn’t mean everybody else is equally incapable.
#8 Davis
April 21, 2007

Trying to make sense of complex systems of any kind is difficult at best — I’m not surpised anything gives us dubious numbers. Too inexact.

It’s all really hard, so let’s just throw our hands in the air and say we can’t understand it? What a convenient way to let you just reject pretty much any part of modern science you don’t like, since everything we study is a complex system.
#9 trrll
April 21, 2007

There is not necessarily anything wrong with excluding a population that your model predicts will be unaffected by the factor being studied. If gang related killings are unaffected by gun laws, then they add an additional source of variance that could obscure the “signal.”

However, when I see a group excluded for reasons that sound kind of arbitrary (if you believe that more guns around will deter or at least reduce the death toll from attacks by people who are insanely suicidal, then why would you assume that the same factor will not also affect attacks by people who are relatively sane and motivated by territorial and illegal business disputes?) I can’t help suspecting that “data snooping” is going on.

The rule is that you have to decide who to include and who to exclude before you look at the data. It is data snooping (and a source of bias) to do the following:

“Oh, darn, it’s not significant. Wait, let’s try lowering the threshold for deaths from 5 to 3–still not significant? OK, let’s try excluding gang-related shootings–there we go, now it’s significant.”
#10 z
April 22, 2007

But… if any group would be expected to be fully “packed” as they say, it would be a bunch of gang members; and therefore multiple shootings of said group would be predicted by the Lott model to be maximally ineffective, and therefore maximally discouraged. My offhand impression is that this is not the case, but they are certainly a terrific control group for his study.