How much difference can one coding error make?

In his statement on the coding errors Lott tries to downplay the significance of the errors:

Minor coding errors were discovered in the data set after it was first given out. The files available for downloading on this site have the corrected results using the statistical county level tests employed in Ayres and Donohue's paper ("Shooting Down the More Guns, Less Crime Hypothesis"). The corrections involved a few thousandths of one percent of the data entries and occurred for observations after 1996. There were well over 70,000 observations and over a hundred variables available in the data set, we are dealing with a few hundred data entries that contained mistakes.

However, even one single error can make a dramatic difference to a least squares analysis. i-ee1ebd2e7e617e6791fa6e844112694f-1000pts.png To demonstrate this I generated 1000 random observations, using years between 1981 and 2000 and a random value between 1 and 100. These are the red crosses in the graph to the left. I introduced just one error by changing the year of the last observation from 2000 to 0. This is a similar error to one of the few hundred coding errors Lott made. Note that one out of 1000 is a much smaller percentage than the percentage of coding errors in Lott's miscoded data set. The green line is the fit to the data with the error. Correcting the error gives the dramatically different blue line. That one single error also changed the result from statistical insignificance to being significant at the 8% level.

More like this

This is a long post, so I'll start with two summaries. One sentence summary: It looks as if Lott might have been caught cooking his "more guns, less crime" data. One paragraph summary: Ian Ayres and John Donohue wrote a paper that found that, if anything, concealed carry laws lead…
In his 6/9/03 posting, Lott claims that Donohue has made a "large number of easily identifiable mistakes". Even if true, such mistakes pale into insignificance compared with the coding errors that Lott made but will not admit to, but let's examine Lott's claims and see how many…
Lott has posted some criticism of Chris Mooney's article. Let's see how many errors he has successfully identified: 1) Paraphrasing claim from the Chronicle of Higher Education stating that the "coding errors had not been reviewed by a third party." I was never asked by the…
Ayres and Donohue have sent a letter to the St. Louis Post-Dispatch, replying to Lott's 21 July letter. I agree with their description of Lott's behaviour as dishonest. On July 21, 2003, researcher John Lott wrote a letter to the editor in which he tried to shore up support for his now…