Lott's claim that Ayres and Donohue's results are not significant

In Lott's 8/20 blog entry he writes:

There is a pretty obvious reason why these guys have choosen to publish their work in nonrefereed publications. Despite their continuing claims to the press, Ayres and Donohue's own papers do NOT provide any statistically significant evidence that violent crimes increase (for a brief discussion see point 2 here). Even most of their own results show that violent crime rates decline after right-to-carry laws are passed.

And this is his brief discussion:

the bottom line is that Ayres and Donohue fail to discuss the statistical significance of the overall effect. They simply note what way individual states go and discuss the weighted average for the effect, but there is no discussion of whether this average effect is statistically significant -- and of course it is not. In statements from the LA Times to their letter to the Post-Dispatch, they claim that there is an increase, but they never offer any statistical evidence to support this claim.

If we examine Ayres and Donohue's paper and look at table 13 (which they describe as their most definitive results), we find, contrary to Lott's claim, that they do have statistically significant evidence of violent crime increases. With 24 states and 4 violent crime categories, there are 96 different tests. Of those, 25 show statistically significant increases and only 7 significant decreases. (Since there were 96 tests, by chance you would expect about 5 significant ones, with a less than 5% chance of 10 or more, so the 7 decreases are not really significant, while the 25 increases are.)

Another way to look at it is to compare the number of decreases with the number of increases. If there was no effect then the numbers should be roughly equal. However, increases outnumbered decreases by 68 to 28. (And yet Lott claims that most of their results show decreases.) There is only a 0.00003 chance of this happening if increases and decreases were equally likely, so once again we have a statistically significant result.

Lott's claim that this is the "obvious" reason that they published in nonrefereed places also makes no sense whatsoever. Even if Lott's claims about no significant increases was true (which it isn't), how would publishing their research in a refereed journal have stopped them from making claims to the press?

More like this

Lott's 6/13/03 entry on his blog links to a letter from David Mayer printed in the Columbus Dispatch replying to a letter from Donohue. Mayer asserts: The recent letter by Stanford law professor John Donohue (June 7) nicely illustrates the propensity of gun-control advocates to play…
Helland and Tabarrok's paper 'Using Placebo Laws to Test "More Guns, Less Crime"' has been published in Advances in Economic Analysis & Policy. Their objective was to correct for serial correlations in the crime data. I explained earlier how, if crimes rates in adjacent…
Lott has a new posting where he responds to a letter from John Donohue to the Columbus Dispatch replying to a Lott op-ed. I earlier posted a link to the op-ed and a letter from Michael Maltz replying to it. I'll post more on Lott's comments later, but for now I want to point to the most…
Lott has a new posting where he has some more about the important matter of the coding errors in his data. Sandwiched between some more complaints about unfair the Stanford Law Review has been and some imaginary errors in Ayres and Donohue, we have: Of course, this is nothing new with…