Lott's cherry-picking

Still nothing from Lott on whether he concedes or denies the charge of coding errors.

In the mean time, let's examine his other claim: "Ayres and Donohue have simply misread their own results." This is a remarkable claim. Lott is saying that crime went down but somehow Ayres and Donohue read a decrease as an increase. If you look through the Lott/Plassman/Whitley (I'll just abbreviate this to Lott) paper to find the basis for this, you find they are referring to Ayres and Donohue's figures 3a to 3e. Figure 3b shows murder, with virtually no change in the first two years after the law, then a big decline for the next decade which bottoms out at 13 years after the law, and then an even bigger increase over the next four years.

Here's what Lott says about this graph:

Their state level regressions indicate that murder rates were rising in the three years prior to the law being passed and then falling over the next thirteen years. Only one state, Maine, had had the law in effect for more than 13 years. The increase during years 14, 15, 16, and 17 thus solely reflect changes in Maine s murder rate and since this is state level data each coefficient represents only one data point. ... The increase between years 13 and 14 is also more apparent than real. The real increase is actually not due to any sudden change in Maine s crime rates, but due to the fact that other states are included in calculating the crime rate for year 13, while only Maine is used for year 14.

Gee, how could Ayres and Donohue have "misread their own results" so badly as not to have noticed that decrease? Let's look at what they wrote about that graph:

To underscore the message that the observations from the adopting states at the two ends of the time span, either well before passage or well after passage, are causing mischief when estimating a single aggregated impact of the shall-issue laws, it may be helpful to illustrate this point graphically. ... Figure 3b shows the results for murder and it should immediately be apparent that the period from eight years before to three years after passage evidences relatively little movement in this crime category.... However, outside this time frame (especially in the last years), one sees large swings in the estimated effect of shall-issue laws on murder. Of course, the thought that shall-issue laws caused crime to drop by almost twenty-five percent in the thirteenth year after passage and then caused it to increase by almost twenty-one percent in the fourteenth year is obviously untenable. These wild swings are caused not by any true impact of shall-issue laws but by the selective dropping out from the individual year estimates of states that adopted the law more recently, leaving only the shrinking number of earliest adopters to identify the particular annual impact.

Oh, Ayres and Donohue did notice the decrease. They specifically pointed to the wild swing to show that it is incorrect to do an analysis that assumes that the law had the same impact every state. This graph appears in the first half of their paper---they go on to do analyses that do not rely on the incorrect assumption.

So what Lott did was take the part of their graph that showed a decrease, ignore the reason why the graph was included (except that he selectively used that reason to justify ignoring the increase), ignore the next sixty pages of their paper and claim that they "misread their own results". I think he has set some sort of world record for cherry picking.

To summarize, Lott's rebuttal of Ayres and Donohue consists of 50% cherry picking, and 50% miscoded data.

More like this

Jeff Johnson of CNSNews.com writes a very pro-Lott piece on the dispute between Lott and Ayres and Donohue. Probably the most notable feature is what is not mentioned---there is nothing about the coding errors Lott made. We can be sure that Donohue mentioned the problem to…
In The Latest Misfires in Support of the More Guns, Less Crime Hypothesis Ayres and Donohue write:In the wake of some of the criticisms that we have leveled against the Lott and Mustard thesis, John Lott appeared before a National Academy of Sciences panel examining the plausibility of…
Lott has a new posting where he has some more about the important matter of the coding errors in his data. Sandwiched between some more complaints about unfair the Stanford Law Review has been and some imaginary errors in Ayres and Donohue, we have: Of course, this is nothing new with…
So what was the one word correction that prompted Lott to remove his name from his paper? In their draft, Ayres and Donohue wrote (my emphasis): On the other hand, the temporal pattern, that states adopting shall-issue laws in the late 1980s did better while those adopting…