More erroneous claims about Kellermann

Posted by Tim Lambert on July 26, 2004
(15)
More »

Kellermann’s studies on guns frequently get criticized by people who do not seem to have read them. The latest to do so is Michael Krauss, who writes

Notwithstanding all this data, the press gave extraordinary publicity to a 1993 article by one Arthur Kellerman in the New England Journal of Medicine. Kellerman’s “study” concluded that the presence of a gun in one’s home dramatically increased one’s chances of being killed by gunfire. As has since been widely noted, though, the study had stupendous methodological flaws that would surely have precluded its publication, were the NEJM not blinded by its fear and loathing of guns.

As we shall see below, Krauss doesn’t seem to have actually read Kellermann‘s study.

The study consisted of going to homes where a homicide occurred, and asking whether there was a gun in the house. Such a study by design and definition excluded successful uses of the gun (i.e., where the attacker is scared off and no one is killed).

Not so. Krauss is apparently unaware that the study was a case-control study. That means that as well as visiting the houses where there was a homicide (the cases), they also found similar homes where there wasn’t a homicide (the controls). Successful uses of guns that prevent homicides show up in the controls.

Even if the homicide victim was someone who did not live in the house, and who was stabbed to death, the answer “yes” to the question, “Was there a gun in the house?”, would increase the correlation between guns and homicide.

Krauss does not seem to have even bothered to look at the abstract of the study. It states: “we identified homicides occurring in the homes of victims”.

Moreover, the fear of being killed by a stalker or a gang might well contribute to one’s decision to purchase a firearm. If the fear is well-founded, then we would expect gun purchasers to be more likely victims of murder than others. But that does not establish that the firearm ownership caused the crime. Analogously to Kellerman’s dishonest methodology, I could “prove” that visiting a hospital correlates with dying. This does not show that the hospital visit caused the fatal illness.

Krauss seems to be unaware that Kellermann’s study controlled for many factors such as age, sex, neighbourhood, drug use and criminal history. To take the hospital example, if we compare people with the same disease and the same severity of disease and find that the ones who go to hospital are more likely to die than the ones that don’t then you have evidence that the hospital is making things worse. Furthermore, Kellermann’s study found that gun ownership was not associated with a higher risk of being murdered by other means, just with a higher risk of being murdered with a gun. Are we supposed to believe that people only get a gun to defend against potential killers that are going to use guns and not against people who might attack them with a knife?

I would have hoped that before making serious charges of dishonesty against Kellermann, Krauss would have looked at Kellermann’s study, but he does not seem to have done so. And Krauss is a law professor.

(15)
More »

Comments

  1. #1 Michael Krauss
    July 26, 2004

    I was surprised and disappointed to learn of this posting. First off, I never “made a serious charge of dishonesty against Kellerman”, as you state in your blog. To the contrary, your blog posting is itself a serious mischaracterization of my opinion, and therefore possibly dishonest.

    I asserted, and still do, that Kellerman’s study is methodologically flawed (that is very different from saying it is dishonest), in that it did not sufficiently isolate gun ownership as an independent variable in death. The tremendous differentials in the extent of gun ownership across the USA are ample testimony to the existence of other overwhelming variables, that I don’t think are adequately controlled for by Kellerman.

    Importantly, you might have pointed out to your readers that my mention of the Kellerman study was part of a much longer, 10-step dialogue on a totally different issue, that of American federalism and the nexus of product liability law (state-based) and federal liability exemptions. I did not devote much time to the Kellerman piece, not because I hadn’t read it or was giving it short shrift but because the study was not central to the dialogue in question.

    Thank you for the opportunity to respond to your criticism of me.

    Respectfully,
    Michael Krauss

  2. #2 Tim Lambert
    July 26, 2004

    Dear Professor Krauss,

    You didn’t just say that Kellermann’s methodology was flawed, but that it was dishonest. You wrote (my emphasis):

    Analogously to Kellerman’s dishonest methodology, I could “prove” that visiting a hospital correlates with dying.

    And since you don’t know what Kellermann’s methodology was, your claim that it was flawed lacks merit.

    I quoted your remarks about Kellermann in their entirety and included a link to your post so that my readers could see them in context. The fact that your comments were not central to the dialogue does not excuse the gross inaccuracies in them.

  3. #3 Mark
    July 26, 2004

    In addition, though Tim is generous enough not to say it, Michael, your response does not address any of the specific errors in your analysis which he has identified. Can we take it that you are tacitly conceding that Tim was spot-on on these matters?

  4. #4 ChrisPer
    July 27, 2004

    Is that the Kellerman study that found alcohol use and renting (not owning) the home were better predictors of gun violence than gun ownership, but did not discuss them in the conclusion?

    I have also seen writers comment that the case-control method assumes that the cases actually match in controlled variables such as income, race, drug use etc; but when the control cases in this study were checked (years later when the data were released) they were actually quite poorly matched.

    The assessment of dishonesty may be strong for one or two points but certainly applies to Kellerman’s work as a whole. This particular study is widely regarded as a benchmark of state-funded advocacy research on guns.

  5. #5 Terry Josiah
    July 27, 2004

    “we identified homicides occurring in the homes of victims in three metropolitan counties.”
    That is like going to a prison and surveying who has used pot and concluding, using pot will lead you to prison 100 percent of the time.
    Asinine.
    I had two childhood friends and three relatives who owned guns.
    One of my childhood friend, I didn’t know his father owned a gun until he told me fifteen years later.
    None committed a homicide in their home. No homicide was ever committed in their home or on their property.
    Because of this, Kellerman’s study would NEVER include them.
    EVER.
    Common sense tells you, if you have it, how utterly ridiculous the study is.

  6. #6 Tim Lambert
    July 27, 2004

    1. Kellermann did not find that alcohol use and renting were better predictors than gun ownership. Alcohol use was not an independent risk factor. Renting was, but these were independent risk factors — that means that just because renting was a risk factor it doesn’t mean that gun ownership was not also a risk factor. They did not discuss renting in the conclusion, but that is because it pretty obviously is not a direct causal factor. They did discuss alcohol, illicit drugs, and a history of violence as well as firearms in the conclusion.

    2. It is not true that controls have to match cases on every conceivable characteristic — that is generally impossible. Kellermann’s controls were matched on sex, race, age and neighbourhood. If you bothered to look at the data you would see that they are not poorly matched.

    3. It is disappointing that like Krauss, you recklessly accuse Kellermann of dishonesty when you have managed to get every point you attempt to make about his research wrong.

    4. Neither you nor Krauss even managed to spell Kellermann correctly.

  7. #7 Tim Lambert
    July 27, 2004

    Terry, you’re wrong. I already addressed your point in my post. Homes where there was no homicide were included in the controls.

  8. #8 ben
    July 28, 2004

    A much better critique of KellerMANN’s work can be found here. I haven’t gone through it to rigorously verify the results, but it looks a lot better than the other criticisms I’ve seen. Note that the absolute risk from a gun in the home is puny.

    Kellermann critique from guncite

    enjoy!

  9. #9 ChrisPer
    July 28, 2004

    Gosh Tim its good to work with someone who knows his onions. Raises the bar after all these years of arguing with feelgood people with the intellectual strength of the Australian Democrats after wine.

    The gun lobby needs you.

  10. #10 John Quiggin
    August 1, 2004

    I must say I’m struck by Krauss’ behavior here. He’s obviously aware of your post and takes immediate objection to it, but when his sole substantive point is refuted (conclusively as far as I can see), he neither responds nor apologises (he’s had a week or so to do it).

    And his post is still up at the Manhattan Institute website without amendment or apology. As you say, very poor behavior from a professor at a reputable university.

  11. #11 Tim Lambert
    August 2, 2004

    I’ve exchanged several emails with Krauss. He would not even acknowledge the existence of the points I made in my post, complaining that I was “disagreeable”. I asked if I could post our exchange, but he refused permission.

    Krauss makes similar claims about Kellermann’s research in his book “Fire and Smoke: Government Lawsuits and the Rule of Law” (though in his book he stops short of accusing Kellermann of fraud).

  12. #12 Brett Bellmore
    August 19, 2004

    One point that concerns me about Kellerman’s use of the case control study format: It’s that whether a gun was owned in the case houses was likely known to some degree of precision due to police investigation, while for the control houses you had to take someone’s word for it. Isn’t this likely to skew the results, if gun owners don’t all admit to owning guns?

  13. #13 Tim Lambert
    August 28, 2004

    Kellermann use the same method for determining gun ownership in the cases and controls — asking.

  14. #14 Brett Bellmore
    August 28, 2004

    Not quite the same; The controls got asked directly, and the cases he relied on proxies. Even Kellerman admitted that the controls might have been understating their rate of gun ownership.

    I’m just curious: How much would they have had to understate it, to produce Kellerman’s results?

  15. #15 Tim Lambert
    August 28, 2004

    The controls also relied on proxies. (Though some of them wer e asked directly the gun ownership rate was the same.)

    If I recall correctly the gun ownership rates were soemthing like 44% cases 34% controls.