Deltoid

Summary: When Lott discovered that correcting his coding errors made his "More Guns, Less Crime" results go away, he refused to admit it. Instead, he tried to resurrect his results by quietly changing his model. When I emailed him, asking him to explain why he had changed his model, he made a clumsy attempt to cover his tracks.

The trouble with the econometric debate about "More Guns, Less Crime" is that it is so impenetrable. Lott has his model that shows that there was less crime, while Ayres and Donohue have their model that shows that there was more crime. You have to be an expert to figure out which model is more likely to be correct. Fortunately non-experts now have a solution---if you correct Lott's coding errors then, even with Lott's model there wasn't less crime. Instead of conceding this, Lott has changed his model and then tried to rewrite history to make it look like he never changed it.

Lott claims that even after fixing the coding errors there were "clear drops in violent crime for murder, rape, and robbery." To see what he means, we need to look at the numerical results of the regressions Lott, Plassmann, and Whitley computed.

There are lots of different regressions in their paper, but fortunately we only have to look at some representative results to figure out what is going on. I've chosen the crime trend using the Spline Model in their table 3a. These are the numbers that Lott usually gives when he describes his results. For example, in his 21 July letter he wrote:

Plassmann and Whitley, who examine three additional years worth of data and find "annual reductions in murder rates between 1.5 and 2.3 percent for each additional year that a right-to-carry law is in effect."

The first column in the table below shows the annual change in crime rates in the years after the law was passed. For example, the first number says that murder fell by 2% each year after a carry law was passed. The numbers in bold are statistically significant (less than 5% chance of getting a number this large or larger if there really was no trend). The underlined numbers are strongly statistically significant (less than 1% chance). The numbers in parentheses are standard errors. The standard error gives us an idea of how uncertain the estimate is. The conventional rule of thumb is that the true value could be up to two standard errors higher or two standard errors lower. For example, the reduction in murder could be as much as 2.0+20.9=3.8% per year or as little as 2.0-20.9%=0.2% per year. If the model is correct, we can be reasonably sure that there really is a reduction in murder, which is why the number is marked as statistically significant. If the model is not correct, then the standard errors are meaningless. A statistically significant result does not mean that the model is correct.

The second column shows the numbers that Ayres and Donohue got when they reran Lott's regression after correcting the coding errors. Notice that none of the numbers are statistically significant any more. This is why they said that correcting the errors eliminates Lott's results.

The third column shows the numbers that Lott says he got when he reran the regression after correcting the coding errors. The reductions in murder, rape and robbery are all now strongly statistically significant. In fact, while the reductions are half as big as before, they are more significant. What's going on here? Why are the results different?

If you look at the numbers you will see that the size of the reductions are not different---for example, they both indicate that there was an annual drop of 1% in murder. Where they differ is in the standard errors, which are much smaller in Lott's "corrected" version. Lott has put his "corrected" computer programs on his web page, so we can find out what he has done to "correct" the coding errors in a way that doesn't eliminate his finding. The program has been changed in one respect from what Lott, Plassmann, and Whitley used in their Stanford Law Review article. Lott has eliminated the command "CLUSTER," which they had used to adjust their standard errors by clustering at the state level. If you don't correct for clustering, then the standard errors you end up with are much too small, making you think that results are statistically significant when they are not. (Technical details on clustering are here.)

Lott, Plassmann and Whitley clearly state that they corrected for clustering:

Table 3a provides the exact results and significance levels behind these specifications, and reports the robust standard errors that adjust for clustering at the state level.

If you compare the first and second columns, you can see that the standard errors are about the same, indicating that both sets of regressions corrected for clustering. The standard errors in Lott's "correction" are dramatically smaller even though in his table he claims:

Robust standard errors are shown in parentheses and clustering is assumed by state.

So Ayres and Donohue were correct when they stated that correcting Lott's coding errors eliminated his results. Lott knows this is true because he must have run the regression after correcting the coding errors and before making the additional change of removing the clustering correction. It is dishonest of Lott to say that their claim is false. Lott can, if he wishes, make an argument that the clustering correction is unnecessary for some reason and offer his new regression as evidence that carry laws reduce violent crime, but it is not proper for him to pretend that he hasn't changed the way he has calculated his results. Nor is it proper for him to put forward, as he does in his July 21 letter above, results from Plassmann and Whitley that murders fell by 1.5%--2.3% annually when even his own corrected figures contradict this.

The only thing that might suggest that Lott's claim that his results survive the correction of his coding errors is an honest mistake is the fact that it is easy for anyone with a copy of Stata to run the regressions and see that Lott's claim is false. But remember that all along Lott has been extremely reluctant to comment on the coding errors. According to his website, he made his corrections on April 18. If he really thought that the coding errors were immaterial, why did he wait four months to say so? Why did he duck all the questions he was asked about the coding errors? Why did he stop citing the research based on the miscoded data? He only claimed that the coding errors didn't matter when he couldn't evade the question any more and the only alternative was to admit that they did.

This whole affair also provides an insight into the way that Lott conducts econometrics. He tries lots of different models and just presents the ones that support his position. There are so many different models that could be considered that it is hard to show that Lott looked at a particular one, but here we have proof that he considered this model and when he found out that it didn't support his position he switched to a different model with no correction for clustering.

Also note that Ayres and Donohue offer persuasive evidence that the model discussed here is not adequate and they go on to a more general model that shows that carry laws are associated with crime increases in most states. The point of all this discussion is that even with the model Lott chose, carry laws do not reduce crime.

I wrote the first part of this post on August 22. When the controversy over the fabricated survey erupted Lott wrote to Jim Henley:

One difference between newspapers and bloggers is that the newspapers at least contact the person and try to talk to them before they write something and put it out.

Fair enough, I thought. So I emailed Lott, asking him to explain what he had done. No answer. Maybe he is blocking emails from me, so I sent another email from another address. No answer. I emailed Lott's co-authors, Plassmann and Whitley asking them (and CC'd Lott as a courtesy). No reply from Whitley. Plassmann was kind enough to reply. He conceded that no significant results remain after correcting the coding errors and did not know why Lott had removed the clustering correction. I also posted my question on the firearmsregprof list (also CC'd to Lott because I am very courteous). No-one there knew of a reason either.

Then, last week, something happened. The file containing Lott's "corrected" Table 3a vanished from johnlott.org. Then, a few days later it was back. Only now it contained a different "corrected" Table 3a. How was it different? First, it no longer claimed to correct for clustering. More importantly, the results were computed from the data set with the coding errors. How come? Well, Lott is trying to pretend that he removed clustering before he corrected the coding errors. You see, if you use the dataset with the coding errors, you get significant results whether or not you use clustering. Removing clustering just so you can claim you got significant results is transparently dishonest, so Lott wants you to think that he first removed clustering (when it didn't change the significance of the results) and then corrected the coding errors. Too bad this version of Table 3a appeared after the other one.

But wait, there's more! I wondered when Lott had created this new version of Table 3a, so I looked at the modification date on the file. It said January 18, 2004, which is, uh, next year. Of course, the modification date doesn't have to be when the file was last modified---you can set it to any date you please, but why would Lott set it to next year? It was only when I wrote the date out in numerical form that I was able to figure it out. The modification date was set to 01/18/04. On the page where you download the file, Lott claims that it was "corrected: April 18, 2003", or, in numerical form, 04/18/03. Bingo! Lott was trying to set the modification date to April 18, 2003 so it would look like he created this version of the table first. He managed to set the day of the month to 18, but screwed up and set the year to 04 instead of setting the month to 04.

Unfortunately, Lott's behaviour with the dates exemplifies his approach to research as well. It is way, way past time that pro-gun folks cut him loose.