William Ford has two interesting posts analysing the key premise of Lott’s lawsuit: that “replicate” can only mean to analyse exactly the same data in exactly the same way.
He looks at the scientific literature on the meaning of replicate and finds that it is used in several different ways. He quotes Paul Sniderman on the different meanings:
Replication in sense 1 involves the use of the same data set, procedures of measurement, and methods of estimation to verify the accuracy of reported results. Replication in sense 2 involves the use of the same data, but not the same methods of measurement or estimation, to confirm the adequacy of the interpretation [and] reported results. Replication in sense 3 involves the use of a different data set and comparable measurement and estimation procedures, to validate the robustness of both the results initially observed and the interpretation originally given to them.
According to Lott’s lawsuit, sense 1 is the only accepted meaning.
In his second post Ford finds example of Lott using replicate in sense 2:
Since the commission’s report, which was presented to the Senate, shows exactly what regression specifications were examined, I tried different specifications to replicate the commission’s results.
and in sense 3 (incidently, it is not true that he got similar results):
The survey was also replicated and obtained similar results to the first survey and the new data has been made available since the beginning of the year.
Let me add a couple to the mix. First, Lott using “replicate” in sense 3 (from a post on his blog):
We had been unable to replicate their claimed results using fixed effects and the only way we could get something similar was without fixed effects. It really shouldn’t have been that difficult for us to confirm what they found since we were used their dates for the laws. Unfortunately, Cummings, Grossman, Rivara, and Koepsell were unwilling to give us their data when we asked for it. I asked for the data from Cummings and one other coauthor. Possibly we should have made a big deal of yet more academics who refused to share their data, but we decided that the more straightforward approach would be to simply say what we found. Alternatively, we could have simply stated that we were unable to confirm their results.
And second, the National Academy of Science panel report on Lott’s research, using sense 1 and stating that could not replicate one of Lott’s results.
Rows 2 and 3 of Table 6-1 report the results of the committee’s replication of these estimates. In row 2, we use the revised original data set and Lott’s computer programs. The committee was unable to replicate Lott’s estimate of the reduction in the murder rate, although the estimates are close and consistent with the conclusion that right-to-carry laws reduce the incidence of murder. Through communication with Lott, the committee learned that the data used to construct Table 4.1 of Lott (2000) were lost and that the data supplied to the committee are a reconstruction and not necessarily identical to the original data.
(Hat tip Josh Wright).