What does “replicate” mean?

William Ford has two interesting posts analysing the key premise of Lott’s lawsuit: that “replicate” can only mean to analyse exactly the same data in exactly the same way.

He looks at the scientific literature on the meaning of replicate and finds that it is used in several different ways. He quotes Paul Sniderman on the different meanings:

Replication in sense 1 involves the use of the same data set, procedures of measurement, and methods of estimation to verify the accuracy of reported results. Replication in sense 2 involves the use of the same data, but not the same methods of measurement or estimation, to confirm the adequacy of the interpretation [and] reported results. Replication in sense 3 involves the use of a different data set and comparable measurement and estimation procedures, to validate the robustness of both the results initially observed and the interpretation originally given to them.

According to Lott’s lawsuit, sense 1 is the only accepted meaning.

In his second post Ford finds example of Lott using replicate in sense 2:

Since the commission’s report, which was presented to the Senate, shows exactly what regression specifications were examined, I tried different specifications to replicate the commission’s results.

and in sense 3 (incidently, it is not true that he got similar results):

The survey was also replicated and obtained similar results to the first survey and the new data has been made available since the beginning of the year.

Let me add a couple to the mix. First, Lott using “replicate” in sense 3 (from a post on his blog):

We had been unable to replicate their claimed results using fixed effects and the only way we could get something similar was without fixed effects. It really shouldn’t have been that difficult for us to confirm what they found since we were used their dates for the laws. Unfortunately, Cummings, Grossman, Rivara, and Koepsell were unwilling to give us their data when we asked for it. I asked for the data from Cummings and one other coauthor. Possibly we should have made a big deal of yet more academics who refused to share their data, but we decided that the more straightforward approach would be to simply say what we found. Alternatively, we could have simply stated that we were unable to confirm their results.

And second, the National Academy of Science panel report on Lott’s research, using sense 1 and stating that could not replicate one of Lott’s results.

Rows 2 and 3 of Table 6-1 report the results of the committee’s replication of these estimates. In row 2, we use the revised original data set and Lott’s computer programs. The committee was unable to replicate Lott’s estimate of the reduction in the murder rate, although the estimates are close and consistent with the conclusion that right-to-carry laws reduce the incidence of murder. Through communication with Lott, the committee learned that the data used to construct Table 4.1 of Lott (2000) were lost and that the data supplied to the committee are a reconstruction and not necessarily identical to the original data.

(Hat tip Josh Wright).

Comments

#1 TrekJunkie
April 18, 2006

I’ve served in dozens of review panels at NSF, NIH, etc. IMHO, when refering to replication, most of the time is in sense 3. New data, and new approaches provides a level of robustness that the previous two can’t. If congruent, it allows for consensus building. If not, then you develop new experiments. Lott has dug his own grave if the defense starts digging up panel summaries at NSF.
#2 Rufus
April 18, 2006

Have you written to Lott to ask him about which meaning he is assigning to “replicate?”

If not, why not?
#3 Mary Rosh
April 18, 2006

I’d like to do some replicating with the brilliant and sexy Prof. Lott.
#4 Paul Crowley
April 19, 2006

I take it that Levitt’s legal team are being made aware of the wonderful work being done on their behalf here?
#5 eudoxis
April 20, 2006

Let me add a couple to the mix. First, Lott using “replicate” in sense 3 …

No, he uses it in sense 1: he was unable to replicate the data as in sense 1. He uses a different data set and therefore doesn’t call it replication. Instead, they “decided that the more straightforward approach would be to simply say what we found. Alternatively, we could have simply stated that we were unable to confirm their results. ”

And second, the National Academy of Science panel report on Lott’s research, using sense 1…

No, they use it in sense 3 because they don’t use the same data set, they use a revised data set.

Lott’s use of ‘replication’ is consistent with the suit.
#6 Chris Jarrett
April 20, 2006

Here’s an example of Plassmann and Lott using the term ‘replicate’ in one of their papers:

“Because the main purpose of this exercise is to compare our data to Duggan’s (2001) data, we replicate his statistical model and regress (a) changes in logsubscriptions on lagged changes in log-murders and lagged changes in log-subscriptions and (b) changes in log-murders on lagged changes in log-murders and lagged changes in log-subscriptions in two separate analyses. To make the results comparable to his, we restrict our analysis to counties with populations of more than 100,00 persons, and use fixed effects dummies together with a very similar set of the covariates that he uses.” (Plassmann & Lott, 2004. More Readers of Gun Magazines But Not More Crimes, http://www.binghamton.edu/econ/wp02/WP0204.pdf)

Note they use the term ‘replicate’ but they don’t use exactly the same data and exactly the same model.
#7 eudoxis
April 21, 2006

“Note they use the term ‘replicate’ but they don’t use exactly the same data and exactly the same model.”

Thy don’t use replicate in reference to the study or the results. From the quote: “the main purpose of this exercise is to compare our data to Duggan’s”. Note that he uses ‘replicate’ in reference only to the model and regress which are not changed and so his use here is consistent with his restrictive use in the suit.

I don’t know enough about Lott to pass judgement on his work, but the point about Lott not using ‘replicate’ in the way he defines it in the lawsuit doesn’t stand up to scrutiny.
#8 Tim Lambert
April 21, 2006

Eudoxis: He wrote We were unable to replicate their results using fixed effects. He was using a different data set and he was trying to replicate their results with it.
#9 Anonymous
April 21, 2006

Okay, you are right if they are trying to replicate using a different data set. I was under the impression that the fixed effects would pull out the same data set.
#10 eudoxis
April 21, 2006

..
#11 Kristjan Wager
April 23, 2006

“I was under the impression that the fixed effects would pull out the same data set.”

But Lott says that not only does the data have to be the same, the methods needs to be the same as well. So even if they had used the same data set, by the definition in Lott’s lawsuit, it still wouldn’t be replication.