Chronicle on Lott vs Levitt

David Glenn has a stellar article in the Chronicle of Higher Education on Lott’s lawsuit. (Free access only for five days.) Glenn writes:

The passage concludes with these 60 words:

Then there is the troubling allegation that Lott actually invented some of the survey data that support his more-guns/less-crime theory. Regardless of whether the data were faked, Lott’s admittedly intriguing hypothesis doesn’t seem to be true. When other scholars have tried to replicate his results, they found that right-to-carry laws simply don’t bring down crime.

It is easy to see why Mr. Lott might find those sentences obnoxious and unfair. It is a glib, breezy passage about a complicated research strategy. The second sentence is constructed in a way that might lead a lazy reader to believe that Mr. Lott’s entire data set on guns and crime — as opposed to the much smaller telephone survey — was under suspicion of fabrication. And the authors chose not to mention a few recent scholarly papers that endorse Mr. Lott’s basic hypothesis, if not his precise statistical model.

But those are not the complaints that Mr. Lott makes in his lawsuit. Instead, he alleges that the last sentence is false and defamatory in a very specific way: “The term ‘replicate’ has an objective and factual meaning in the world of academic research and scholarship,” the lawsuit reads. “When Levitt and Dubner allege that ‘other scholars have tried to replicate his results,’ the clear and unambiguous meaning is that ‘other scholars’ have analyzed the identical data that Lott analyzed and analyzed it the way Lott did in order to determine whether they can reach the same result.”

While Glenn finds two supporters of Lott on the meaning of replicate, their assertions are trumped by the examples of it being used with the more general meaning.

Lott’s charge in the second part of his lawsuit also seems on shaky ground:

In “symposium” issues of journals, there is a tacit understanding that all, or nearly all, of the papers will be published, said Jeffrey A. Miron, a visiting professor of economics at Harvard who participated in the conference. “The rate at which the journal accepts these papers and chooses to publish them is 90 to 100 percent,” he said. “So it’s not refereed in the same way, or subject to the same standards, as if you submit something cold to The Journal of Law and Economics.” (The journal’s normal acceptance rate is less than 10 percent.)

Mr. Miron said Mr. Levitt’s characterization of the issue as “not peer-refereed” was an exaggeration but not an outrageous untruth, and Mr. Miron doubts Mr. Lott will win on this point in court.

This bit is interesting:

In any case, Mr. Moody wishes Mr. Lott the best in his battle with Mr. Levitt. If the Freakonomics passage about replication had been written about his own work, Mr. Moody said, “I’d feel as if a knife had been stuck in my chest. That’s the worst thing that you can say about an academic. It’s horrible.”

Oh, I don’t know if it’s the worst thing. How about this statement about an academic made by Moody at an AEI symposium?

The second cut is, as you say, is the data available to other researchers [inaudible], and the answer is no for Kellermann, so I think he’s lying. He’s refused repeated requests for his data. So, no, I don’t trust him and I don’t think anybody here in this room should trust him, or anywhere else, for that matter.

I pointed out to Moody that Kellermann’s data was available for download from the ICPSR. Moody had his slander of Kellermann edited out of the transcript. I suggested that he should publicly apologize to Kellermann. He initially said that he would be happy to do so, but reneged, saying that he thought Kellermann would prefer not to get an apology (false, in fact).

Comments

#1 A. Random Physicist
April 20, 2006

I just wanted to comment on this part of the article:

Mr. Lott is drawing on a particular sense of “replication” that has recently been promoted among social scientists, most prominently by Gary King, a professor of government at Harvard University. Mr. King would like the data sets and statistical models of economists and other social scientists to be routinely “rerun” by their colleagues, much as chemists and physicists confirm one another’s experiments. “The essence of science is replication — making sure that what’s on the record is on the record,” said Mr. King. Carefully reproducing complicated econometric studies like Mr. Levitt’s and Mr. Lott’s can help uncover innocent and not-so-innocent mistakes, he said.

I think it’s pretty misleading to compare what Lott and King are asking for to what working scientists do. If you ask scientists what they mean when they talk about replicating (or repeating, or reproducing) someone else’s work, you’ll get an answer that is much closer to Levitt’s usage than Lott’s. In fact, I would argue that replication in Lott’s usage is basically useless. One of the main reasons for repeating someone else’s work is to see how robust the results are to changes in data and/or methodology.
#2 Bill Hooker
April 20, 2006

I agree with ARP. When King says “the essence of science is replication”, he seems to misunderstand how results are typically confirmed in science. It’s far more usual to have a result confirmed by experiments that build on it than by having another lab repeat it exactly. A brief example of what I mean is here (from this discussion).

Also, thank you for standing up for common courtesy (publicly making the point about the apology Moody owed but did not make). Apropos this discussion, I’m glad to see someone defending a reasonable set of standards by which academics/scientists ought to treat each other.
#3 Brett Bellmore
April 20, 2006

In fact, wasn’t the allegation that Kellerman had refused to release his data true, for a good four years? (What’s more, I seem to recall that he only relented when Congress passed some sort of legislation on the subject.) I don’t think it’s so outrageous of people to have failed to notice him *eventually* releasing it, long after the research in question was no longer the subject of a live contraversy.
#4 Mike Maltz
April 20, 2006

In his court filing, Lott claims that he has “a reputation for exacting, credible, and reliable economic analysis.” Other than with Mary Rosh, with whom? In our letter to the head of the American Enterprise Institute in 2003, which Tim cites somewhere in his blog, Dudley Duncan and I laid out a half-dozen of Lott’s procedures that lack exactitude, credibility, and/or reliability.
#5 Eli Rabett
April 20, 2006

Don’t know how this got by everyone, but Glenn is building a strawman. Levitt said

“have tried to replicate his results,”

Not his data, not him method, REPLICATE THE RESULTS. RTFR
#6 Robert
April 20, 2006

Kellerman’s data were published by ICPSR in 1997. Moody made his statement in 2003.
#7 Brett Bellmore
April 21, 2006

Correct, Robert, my point simply being that Kellerman is NOT owed an apology. He did in fact “refuse repeated requests for his data”. The fact that he finally relented years later doesn’t erase the long period of demanding that people take his results on faith, a demand which rightly raised doubts about his research.

Did anyone go back and check that research when the data were finally at long last released? Or was it just ignored as no longer a live contraversy?
#8 Bill Hooker
April 21, 2006

Kellerman is NOT owed an apology

I think he is. Moody would owe no apology if he’d said Kellerman “finally relented in 1997″, but his phrasing as given by Tim above is in the present tense, falsely asserting that the data were not available in 2003. Moreover, Moody had had six years in which to check the data himself, so he should have either provided proof of his charge that Kellerman lied, or not made the charge at all.

I agree, though, that four years’ refusal to release data is also unacceptable. Moody had a perfectly good point right there, so why’d he push beyond the criticism the facts would support?
#9 z
April 21, 2006

“In fact, wasn’t the allegation that Kellerman had refused to release his data true, for a good four years? (What’s more, I seem to recall that he only relented when Congress passed some sort of legislation on the subject.)”

In other words, Kellermann was not legally required to release his, labor-intensive and laboriously and expensively collected, data until the firearm industry had their shills change the fules after the fact. This is not unusual in any way; research raw data which is the result of one’s own labors is extremely valuable as the source for publications, the fuel for an academic career, and is not legally required to be handed out to anybody while the original investigator is still generating publications from it, any more than the Coca Cola company is required to give NRA-Cola its formula just because NRA may accuse them of putting toxins in it.

This is entirely different from the situation with Lott, whose data was collected entirely from public sources financed by public money in the US and is therefore public domain. (and would therefore be at least theoretically subject to being precisely replicated from said sources independently anyway).

Of course, the bottom line on the Kellermann data after years of having been accused of fraud, cherry picking, creating data out of thin air, etc., after it was made public was that nobody could find anything wrong with it. This, of course, has not stopped the usual gunloons from continuing to accuse him of fraud, cherry picking, creating data out of thin air, etc., or for that matter, of “not releasing his data”.