This is a long post, so I'll start with two summaries. One sentence summary: It looks as if Lott might have been caught cooking his "more guns, less crime" data.
One paragraph summary: Ian Ayres and John Donohue wrote a paper that found that, if anything, concealed carry laws lead to more crime. Lott, (along with Florenz Plassmann and John Whitley) wrote a reply where they argued that using data up to 2000 confirmed the "more guns, less crime" hypothesis. In Ayres and Donohue's response to that paper, they found that Lott's data contained numerous coding errors that, when corrected, reversed the results. Furthermore, this was the second time these sorts of errors had been found in Lott's data. Lott had presented to the NAS panel figures showing sharp declines in crime following carry laws. Declines which disappeared when the coding errors were corrected. Finally, when Lott saw Ayres and Donohue's response he had his name removed from the final paper.
The gory details: In a critique of econometric studies such as Lott's Ted Goertzel makes an important point---for any study that involves a multiple regression that finds a significant association, it seems that there is another study that applies a different model to the same data and gets a different answer. Goertzel argues that
When presented with an econometric model, consumers should insist on evidence that it can predict trends in data other than the data used to create it. Models that fail this test are junk science, no matter how complex the analysis.
In the case of Lott's model we are in the fortunate position of being able to test its predictive power. Lott's original data set ended in 1992. Between 1992 and 1996, 14 more jurisdictions (13 states and Philadelphia) adopted carry laws. We can test the predictive power of Lott's model by seeing if it finds less crime in those jurisdictions. In their just-published paper Ayres and Donahue have done this test. They found that, using Lott's model, in those later-adopting jurisdictions, carry laws were associated with more crime in all crime categories . Lott's model fails the predictive test.
Ayres and Donahue go on to examine all the states adopting carry laws using data up to 1999 and find that:
The best, albeit admittedly imperfect, evidence suggests that, for the majority of states, shall-issue laws are associated with higher levels of crime.
Lott's reply offered two main arguments. First:
Ayres and Donohue have simply misread their own results. Their own most generalized specification that breaks down the impact of the law on a year-by-year basis shows large crime reducing benefits. Virtually none of their claims that their county level hybrid model implies initial significant increases in crime are correct. Overall, the vast majority of their estimates based on data up to 1997 actually demonstrate that right-to-carry laws produce substantial crime reducing benefits.
And second:
Analyzing county level data for the entire United States from 1977 to 2000, we find annual reductions in murder rates between 1.5 and 2.3 percent for each additional year that a right-to-carry law is in effect.
Ayres and Donohue's response to this is devastating. This is from their introduction:
In our initial article Shooting Down the More Guns, Less Crime Hypothesis---we reached two main conclusions: First, that there was no credible statistical evidence that the adoption of concealed-carry (or "shall-issue") laws reduced crime; and second, that the best, although admittedly quite imperfect, data suggested that the laws increased the costs of crime to the tune of $1 billion per year (which is a relatively small number given the total cost of FBI index crimes of roughly $114 billion per year). In their response to our article, Florenz Plassmann and John Whitley (PW) offer two sets of evidence in support of their view that that concealed-carry laws are beneficial: First, they argue that some of our regression specifications really buttress their position; and second, they analyze some new county data for the period 1977-2000.Their first method of proof fails because it simply overlooks---without even a single word of commentary!---the entire thrust of our paper: that aggregated specifications of the effects of these laws are badly marred by "jurisdiction selection" effects. We did not misread these aggregated estimates, as PW suggest; we simply showed that the PW claims based on these aggregated estimates are inaccurate and misleading. The data at every turn reject the idea that concealed-carry laws passed in different jurisdictions have a uniform impact on crime. Therefore, the results of disaggregated regressions must, counter to PW s claim, be taken as a more authoritative assessment of the overall impact of concealed-carry laws.
Their second method of proof fails because PW seriously miscoded their new county dataset in ways that irretrievably undermine every original regression result that they present in their response. As a result, the new PW regressions must simply be disregarded. Correcting PW s empirical mistakes once again shows that the "more guns, less crime" hypothesis is without credible statistical support.
And this:
It is important to note that what we now refer to as the PW response has already been widely circulated as a draft, whose first author is John Lott. Moreover, Lott has repeatedly told the press and/or published to the Internet that Ayres and Donohue have simply misread their own results. But after seeing this Reply to the original Lott, Plassmann, and Whitley paper, Lott asked the Stanford Law Review to take his name off the work. We hope that this indicates that the arguments in our Reply have caused the primary proponent of the more guns, less crime hypothesis to at least partially amend his views. We note that to this day, legislators are still voting for the adoption of concealed-carry laws while citing Lott's work.
And this is from their conclusion:
While we emphasized the severe selection-effect problem of estimating the effects of concealed handguns by aggregating across all the adopting jurisdictions, PW simply ignore this concern. When they contend that we have misread our own results, it is because they cite the jurisdictionally aggregated regression estimates that we showed were flawed and continue to pin their more guns, less crime hypothesis on this flawed estimation approach. Indeed, if one accepts our view on this point, one has to jettison virtually their entire paper, which probably explains why they did not respond to the issue. PW present twelve figures and thirteen tables in their paper that offer estimates of the effect of concealed-carry laws on crime, and of these, every one but PW table 6 and appendix table 4 is unreliable because they rely on the discredited jurisdictional-aggregation assumption. Moreover, every new regression on PW s extended county dataset is fatally flawed by coding errors that conveniently support their thesis, so readers must be careful to disregard every regression finding that PW ran (that is, everything from PW table 3 on and PW figure 4 on).PW charge that we have misread their results, but only because they ignored our discussion of the dangers of aggregation so well documented in our AD figures 3a through 3i and our AD tables 7 and 8. The bottom line is that the best evidence suggests overall small increases in crime associated with adoption of concealed-carry laws, but there are enough factors to counsel caution in making strong conclusions. One such concern is the fact that the most consistently strong results suggest increases in property crime, even though the theoretical link between these laws and property-crime hikes is obscure.
In the wake of some of the criticisms that we have leveled against the Lott and Mustard thesis, John Lott appeared before a National Academy of Sciences panel examining the plausibility of the more guns, less crime thesis and presented them with a series of figures showing year-by-year estimates that appeared to show sharp and immediate declines in crime with adoption of concealed-carry laws. David Mustard even included these graphs in his initial comment on the Donohue paper in the Brookings book that PW refer to repeatedly in their current response. But Donohue privately showed Mustard as well as the Brookings editors that the graphs were the product of coding errors in creating the year-by-year dummies, and in the end Mustard conceded and withdrew them from his comment on Donohue. Now PW respond to our paper with an array of regressions that purport to support their thesis, but again are utterly flawed by similar coding errors. We previously made no mention of the initial National Academy of Sciences/Brookings comment error, since we know how easy it is to make mistakes in doing this work. But for the second time Lott and coauthors have put into the public domain flawed regression results that happen to support their thesis, even though their results disappear when corrected. Claiming we misread our results in the face of such obvious evidence to the contrary and repeatedly bringing erroneous data into the public debate starts suggesting a pattern of behavior that is unlikely to engender support for the Lott and Mustard hypothesis. We feel confident concluding that we have indeed shot down the "more guns, less crime" hypothesis.
I really like the understatement in this bit:
Claiming we misread our results in the face of such obvious evidence to the contrary and repeatedly bringing erroneous data into the public debate starts suggesting a pattern of behavior that is unlikely to engender support for the Lott and Mustard hypothesis.
This, of course, is the same pattern of behaviour we have seen from Lott in the scandal about his 98% brandishing statistic.
Glenn Reynolds also discusses this latest twist. Addendum: He has linked back and wonders whether the bad data was in More Guns, Less Crime or is new data. The answer is that it was new data in Confirming "More Guns, Less Crime" (the paper that Lott removed his name from) and his new book, and a second set of data that he used for his NAS panel presentation. Ayres uses "more guns, less crime" to refer to the hypothesis, not just Lott's book More Guns, Less Crime. I apologize for the lack of clarity in my writing.