I've discovered another one of John Lott's attempts to rewrite history. Read on.
Lott has written a response to Kevin Drum's summary of Lott's model changing antics. Here's Drum:
1. Lott and two coauthors produced a statistical model ("Model 1") that showed significant crime decreases when states passed concealed carry gun laws.
2. Back in April, two critics discovered that there were errors in the data Lott used. When the correct data was plugged into Lott's model, his results went away.
3. After a long silence, Lott admitted the data errors and posted a table with new results. Oddly, though, his new results were similar to his old ones and continued to show significant drops in crime. So who's right, Lott or his critics?
4. Answer: his critics. It turns out that since he really had no choice but to use the corrected data, and the corrected data erased his results, he decided to invent a different model ("Model 2") for use in this new table -- but without disclosing the fact that he had switched to a new model specifically constructed to keep his results intact. Note: In less refined circles this would be called "lying."
5. When Tim discovered that Lott had surreptitiously changed his model, he emailed Lott. No response.
6. It turns out Lott was busy covering his tracks. How? By quietly removing the corrected table from his website and replacing it with a new corrected table. This one uses Model 2 but has the old, incorrect data.
7. Here's where you have to pay attention. Why would Lott do this?
Answer: this new table claims to be "corrected: April 18, 2003," and it turns out that Lott is trying to pretend that this was the original table he had posted all those months ago. That way, he could claim that he had never changed his model at all. Model 2 is the one he's been using all along!
8. Unfortunately, when Lott changed the revision date on the document to make it look like it had been created on 4/18/03, he changed it to 1/18/04 instead. What's more, Lott apparently doesn't know that you can check the create date of PDF documents anyway, and this one was created on 9/2/03. That is, it was created in September, not April.
Basically, Lott wants to pretend that Model 2 is the one he's always used. That way, when he corrects the data errors, his results still hold up. Unfortunately for Lott, his attempts to rewrite history were as clumsy as they were dishonest. His original table did use Model 1, his results do go away when the corrected data is plugged in, and he did respond to this by furtively devising a new model that would continue to give him the results he wanted.
I've numbered Lott's response to each of Drum's points
1. The paper that is being referred to here was published by Florenz Plassmann and John Whitley in the Stanford Law Review. I was not an author on their published article. I had helped with them for a while on this project, but dropped out before publication (see Plassmann's statement on the fact that John Lott helped us out on" it, their paper also has a footnote making a similar statement).
Lott was actually the primary author of the paper as can be seen from this draft at SSRN. The version published in the Stanford Law Review was essentially the same as the draft, except for a some interesting differences that we will discuss below. Lott did not drop out before the paper was finished--he removed his name in a fit of pique over a one word editorial correction.
2. Out of about 7.5 million data cells, the Plassmann and Whitley data set had accidentally left 180 cells blank. The results did not go away. Some results were weakened, but the central results that the Plassmann and Whitley article emphasized and which they said were the proper way that they thought the estimates should be run were not affected
In fact, it is Lott's data set, and he is the one who made the coding errors, not Plassmann or Whitley. Lott just will not accept responsibility for his mistakes. The "central results that the Plassmann and Whitley article emphasized" is a reference to the Poisson estimates. Actually, they are not emphasized and take less than one page of a 42 page article. When Lott has written about the paper he has presented the least squares estimates, which do go away when the coding errors are corrected. For example, see this letter he wrote.
Footnote 31 of the draft paper states "As we will discuss later, using weighted least squares is not the ideal estimation technique." However, this footnote does not appear in the published version. You can find the solution to this mystery if you look at the document properties of the draft paper. While the paper is dated December 9, 2002, it was created on October 1, 2003. Yes, this is another of Lott's attempts to rewrite history, this time by adding a footnote that wasn't there before. And point 8 above was written in September 2003, so after he got caught trying to rewrite history by the document properties information, he did it again and got caught the same way.
3. As a favor to Plassmann and Whitley, I had put up their data set on a website that I have (www.johnlott.org). When the errors in the data set were discovered, I immediately put up a corrected version. There was a note next to the data set announcing the correction and that note was updated over time.
You only have to click on the link to johnlott.org/ to see that the data isn't there as a favour to Plassmann and Whitley. It's the website for Lott's The Bias Against Guns, and the data is used in Appendix 1 of that book. Lott will not tell the truth about even little things like this. And while the data was corrected in April, the note admitting that there were coding errors in the data was not added until August.
4. The data set was corrected in April 2003 (See the statement by James Knowles who was my Research Assistant at that time), and that was months before these charges were made in September. First, Plassmann and Whitley's results, which they said were the way that they thought the regressions should be run, were not affected. I did not invent a new model. I assume that what is being referred to here is that the estimates that I provided were without "clustering," which is exactly the way that I have always done this when I provide estimates on this issue, the way that the people Plassmann and Whitley were debating ran their estimates, and the way the recent National Academy of Sciences panel ran the estimates (see here).
The way Lott ran the estimates in the published paper (model 1 above) was with clustering. From page 1345: "clustering is assumed by state". So it is not true that this is exactly the way he always did it. Mind you, the draft paper doesn't use clustering, but that's only because it was altered in October 2003.
5. Because [Tim Lambert] has been so abusive, I programmed the filter on my email program to filter out emails from Tim Lambert.
My emails to Lott have never been abusive. You can see an example here. In any case, when he did not respond I sent another email from a different address.
Jeff Koch, my web master, accidentally cut the link to the correct file and when he tried to fix this problem he reconnected it to the wrong file. The error was there for less than a day.
This is not true. I downloaded the file in June. Not only did Chris Mooney download it in August, but Lott emailed him a copy of the same file. Koch conceded that his explanation was incorrect.
7. The corrected file was indeed put up when the data set error was discovered. Please see the statement by James Knowles.
But the version that Knowles says he uploaded is the one that Mooney and I downloaded: no clustering, coding errors fixed. The version that is there now, and that Lott is pretending was there all along (except for one day), has no clustering and still contains the coding errors. In other words, Knowles' statement proves that Lott's claims are false, and he tries to pretend that it confirms them.
8. Please again see a statement from my webmaster Jeff Koch who handled this episode and James Knowles who put the original corrected files together for the website (http://johnrlott.tripod.com/other/J.html). Because the names on the files were similar and in order to make sure that the wrong one wasn't posted, I provided Jeff with another copy of the file. The 1/18/04 date, which was well in the future at that time of these other events, was due to a faulty time date on the computer that was being used.
If the date on the computer was accidentally set in the future, then the document properties data would also have been in the future. Instead, that date was in the past and the file creation date was in the future. This seems likely to be the result of someone altering the file creation date to try to backdate the changes and messing up.
OK, so Lott altered the draft paper back in October 2003 to support his position and I only just noticed. But who would have thought that he would do anything so silly? Did he really think he could get away with making it look like Plassmann and Whitley introduced clustering after Lott had dropped out of the paper? Note to anyone thinking of co-authoring with Lott: don't do it -- he'll stab you in the back.
Dang! Lott gets shadier and shadier every day. Why is anyone still paying him a salary? At this point, he ought to be radioactive in terms of "helping" RKBA.
It strikes me that the evidence here, while consistent with the conclusion, does not necessarily lead to that conclusion.
The evidence here is also consistent with the possibility that there were two errors, discovered separarely: one in the model, and one in the data. Now when the paper was first distributed and published, it is possible that this version had both sets of errors inadvertently; this may well have gone unnoticed by Lott if the results were similar to those with both errors corrected.
When questioned, Lott may have simply regenerated the results with both the errant and corrected data, but only with the corrected model (it being quite possible that the errant model code was not available).
I also note that given the tone of this blog entry, it is certainly possible (though not proven) that a similar tone might have been present in one or more of Lambert's email messages to Lott, and that this could have been interpreted by Lott as being "abusive".
In addition, a PDF creation date later than expected means only that the PDF itself was created subsequently, not the the base documents were updated subsequently. This is not at all uncommon (to save space I often delete PDF files after distributing them while retaining the original source, and then regenerating the PDF files if and when needed).
Finally, I note that this is an ad hominem criticism, which does not affect the strength of the corrected model. If such criticisms can be made, then they they should be made; if not, ad hominem is a feeble substitute, and would better be fodder for politicsblogs, rather than scienceblogs.
Well Jimmy P, let me tell you that John Lott is a serious and respected scholar who would never stoop to responding to, or indeed levelling, ad hominem criticisms.
James, to run the model with clustering you just add the cluster command to the command file. It's not something you would do accidently. And the paper states that the results were computed using clustering, so clearly he originally used clustering. Your alternative explanation is also inconsistent with Lott's own account -- he's denying that he ever used clustering at all.
My post links to an example of an email I sent him -- the tone of that email is no different from any others that I have sent him. Not that I got an answer to that or any subsequent email.
Scientific fraud is an appropriate subject for science blogs. And if you discover that a scientist is guilty of fraud, it's actually a very strong argument for not trusting any of their other results.
BTW: Where has the banning of guns led to a decrease in crime?
Firstly, Terry I hoepyo uaren't suggesting that it's alright for Lott to lie and falsify research becasue he's on the "right" side.
Secondly, the American National Academy of Sciences investigated Lott's claims that right-to-carry laws reduced crime and concluded there was no evidence that they either reduced or increased criem rates.
Thirdly, the US crime rate fell significantly in the years following the national assault rifle ban.
Fourthly, Australian homicide and aremd robbery rates have fallen since we tightened gun laws in 1995. Canada and the UK also tightened gun laws in the 1990s and subsequently saw declines in their crime rates.