Cramer’s defence of Lott

Clayton Cramer has mounted a defence of Lott on the survey question. Just as Bellesiles tried to persuade people that the attacks on his work were politically motivated, Cramer tries to persuade us that the attacks on Lott are politically motivated. Now, Jim Lindgren has been prominent in raising question about Lott’s mysterious survey, but Cramer wrote Lindgren out of the survey controversy and replaced him with John Donohue. Cramer doesn’t mention Lindgren once, instead asserting that Donohue is using the lack of evidence for a survey “to cast doubt on Dr. Lott’s integrity.” However, contrary to the impression that Cramer leaves, Donohue has hardly mentioned the survey, while Lindgren has written an extensive report and further comments on the survey. Why does Cramer replace Lindgren with Donohue? Well, he can’t paint Lindgren as partisan, while he has made a (specious) case that Donohue is partisan. I am the other critic that Cramer mentions and according to him I am only attacking Lott because of my “desire to see victims disarmed and murdered by criminals”.

Continuing with his “it’s just a partisan attack” theme, Cramer writes:

Unfortunately, Dr. Lott stumbled into a highly politicized area, and when you are confronting people like Tim Lambert or John Donohue, you need to have very persuasive evidence. Why? Because both of them are partisans on this issue (Professor Donohue’s protestations are not persuasive), and they are going to demand very high standards of proof.

What has been asked of Lott is not that he meet some super high standard of proof, but the normal standard in any scientific endeavour. If you present a finding from a study you conducted, then you are expected to have the data to support your finding. In fact, it is scientific misconduct to report research that cannot be supported because the data no longer exists.

Cramer then proceeds to downplay and ignore the evidence that points to Lott not having done a survey while exaggerating the evidence for the survey’s existence.

While Cramer mentions the singular lack of evidence for the existence of the survey, he does not mention the fact that Lott attributed the 98% figure to other surveys for over two years. Lott only said that the number came from his own survey when Otis Dudley Duncan pointed out that the other surveys did not yield a figure even remotely like 98%. He does not mention the fact that Lott claimed that the survey was conducted over 3 months in 1997, but managed to present the 98% statistic (and attribute it to other polls) on Feb 6, 1997, well before his survey was completed. Or the fact that nine published surveys give results that differ markedly from 98%. Or the fact that he repeatedly changed his story about how the survey was conducted. Or the fact that he has been caught lying about other matters rather than admit to a mistake.

Cramer presents three pieces of evidence suggesting that Lott did a survey. Each piece of evidence is far weaker than he thinks.

First, there is David Mustard’s statement that he believes it likely that Lott told him about the survey in 1997 and is certain that Lott told him about it before October 1999. However, as Lindgren explains, Mustard’s account of when he heard about the survey has kept changing into versions more and more favourable for Lott:

When I discovered that Mustard had told Frank Zimring on the telephone in the summer of 2002 that he knew “nothing” about Lott’s 1997 survey, I called Mustard and we had a series of long talks.

Mustard confirmed the substance of his conversation with Zimring, but said that his general statement of knowing nothing about Lott’s 1997 survey followed a series of specific questions from Zimring about the survey, which he couldn?t answer. Mustard said that he meant that he knew nothing specific about the survey since he was not involved in it. In Mustard’s conversations with me, he also backed off his claim that he was fairly confident that he heard in 1997 about the survey, saying that he was certain that he learned about it before his October 1999 testimony, but he couldn’t remember whether he heard about it weeks, months, or years earlier. He said that his memory of talking with Lott about follow-ups in 1996 was firm and his memory of what he knew in 1999 was firm, but between late 1996 and late 1999 he did not know when he first learned of the 1997 survey. Nonetheless, Mustard then released a public statement covering much the same ground as he had covered with me, but adding claims about both 1998 and 1997. About 1997, Mustard wrote: “I believe it likely that John informed me of the completed survey in 1997.” I have not talked with Mustard since, so I never learned the basis for his recovered belief that Lott informed him in 1997 or his statement about 1998. I can only say that Mustard did not have either of those recollections when I spoke with him at length a few weeks before.

We do know that Mustard heard about the survey in 1999, but that actually makes things worse for Lott. Even though the survey was supposedly conducted in 1997, Lott made no direct or indirect reference to it until May 1999, when Duncan pointed out to him that his claim that the 98% came from national surveys was wrong. I suspect that he invented the survey then so that he would not have to admit that the statistic was wrong. Soon after that, he likely told Mustard about the survey. I got an email from him out of the blue in June 1999, telling me that he had conducted a survey. I have compiled a list of the direct references (where he says he conducted a survey) and the indirect references (where he states that the number of defensive gun uses is about two million, the number from his survey, rather than the 2.5 million that Kleck found). Notice that the first mention is May 13, 1999. Although it is mentioned frequently after that date, there is no mention before that date, even though he made many “98% brandishing” claims before then.

Second, there is David Gross’ statement that he believes that he was surveyed by Lott. However, as Lindgren explains, his account of the survey he was in doesn’t match Lott’s survey very well at all:

When I asked him if he remembered anything about who called, he said that he “was beginning to think” that the call came from students in Chicago, perhaps at Northwestern or the University of Chicago, but he was very uncertain about whether the call came from a Chicago area source. In his public statement issued after he talked with me more than once, however, Gross’s very uncertain memory became a bit more certain, suggesting that the call probably came from the University of Chicago. That and the timing (which he was also not certain about) were the only things that pointed to him having been called by Lott as opposed to another survey organization.

As I delved into the other studies being done in the 1996-97 period, I found that Gross’s description of the questions that he was asked fit a 1996 Harvard study by Hemenway & Azrael better than Lott’s account of his study questions. First, Gross said that the person who called him was interested in a defensive gun use that happened a few years before he was surveyed, but was not interested in a defensive use that occurred many years before that. This would not fit Lott’s survey, since Lott asked only about DGUs in the prior year. It would fit the Harvard study perfectly, which asked about DGUs in the prior 5 years, but excluded events before that. Further, Gross said that he gave a narrative account of the event, which the caller was interested in. Lott’s study had asked closed-end questions, which would make the narrative superfluous, while the Harvard study was one of the first to ask for a narrative account of DGUs. Last, Gross reported that there was a question about state gun laws, which Lott did not ask, but the Harvard study did.

To be fair, Hemenway doesn’t think it likely that Gross was surveyed in the 1996 Harvard study. He writes:

He doesn’t seem to have been. No one from his state, of his gender and general age responded Yes to the self-defense gun use question, and no one in the whole survey told a story highly similar to his story…

None the less, there doesn’t seem to be any reason to think that Gross was surveyed by Lott rather than by another survey. Another important point is that Gross was one of the prime movers behind the recently passed concealed-carry law in Minnesota. It is highly unlikely that someone with this great a motive for preserving Lott’s credibility would be included in a random sample of 2,000 people.

That doesn’t mean that Gross made the story up—there is another possibility. Gross may have been contacted because there was a news story about his defensive gun use. You see, if you are designing the questions for a defensive gun use survey, you might want to test the questions out on a few people to see if they elicit the information you are after. The trouble here is that if you call people at random you will have to call hundreds of people before you get enough people who have used guns to properly test your questions. So what you can do is call some people who you know have used a gun for defence and try your questions out on them. I found a news story published in 2002 that reported Gross’s activism and his defensive gun use. Obviously that story could not have caused someone to contact him in 1998 (or 1997 or 1996) and ask questions about defensive gun use, but if there was a story in 2002 there could have been an earlier story as well.

So it’s conceivable that Lott was thinking of doing a survey in 1997 and Gross was contacted to test some possible questions. (This would also explain why the questions that Gross reported don’t match Lott’s survey.) But even this most generous interpretation of the evidence doesn’t help Lott, since even if you show he was planning to conduct a survey, there is still no evidence that he actually conducted a survey, and plenty of evidence that he didn’t.

Cramer’s third piece of evidence is the one that he mistakenly regards as the strongest. He writes:

The 2002 survey that Dr. Lott did (and it is well established to have happened), gave roughly similar numbers to the 1997 survey results. It seems unlikely that Dr. Lott could construct a survey that would give him numbers to match a previous survey, unless that previous survey was real. He could certainly just make numbers up to match—but running a real survey certain to give the right numbers would require an amazing level of knowledge of how real people would answer the questions.

The trouble with this piece of evidence is that it just isn’t true. Lott has repeatedly claimed that the 2002 survey gave similar results to the 1997 survey. But it didn’t. The 2002 survey “shows” that people fire their guns five times as frequently as Lott claims his 1997 survey found. (I put the scare quotes around “shows” because the sample size of the 2002 survey is actually far too small for it to tell you anything useful about how often people fire their guns, but let’s ignore that for now.)

So Lott didn’t have to construct a survey that gave numbers that matched a previous survey. All he had to do was construct a survey that gave different numbers, make up some numbers that matched his previous survey, and then claim that they came from his new survey. Furthermore, this fact has been explained to Cramer in online discussion before but he keeps repeating Lott’s claim that the 2002 survey gives similar results.

Cramer’s final argument is that for something as serious as fraud, it is necessary to prove the case beyond a reasonable doubt. But this isn’t a criminal trial. If you think the evidence only shows that Lott probably fabricated the survey, then you should conclude that Lott probably fabricated the survey. You don’t have to acquit him or anything.


  1. #1 ThinkTank
    October 23, 2003

    I can’t wait till the new edition of Arming America comes out, its going to shut up so many of those right wingers. The difference between John Lott and Michael Bellesiles? Lott was out to make “propaganda” at least in a limited sense for the gun industry. Seeing as his chair at U of Chicago was paid for by gun industry allies. While Bellesiles was an almost unknown academic historian working at Emory. He even admitted to being a gun owning republican before “Arming America” came out. You can see the difference in how they respond to criticism. Lott and his allies say they didn’t make any errors. Bellesiles admits that NO history text is without errors and that peer review is the best way to catch them. I BELIEVE that Bellesile’s thesis in Arming America was correct, and that the American gun culture was only invented recently. Most likely by the likes of Colt and Winchester to sell more product.