I buried this among a bunch of other cool links yesterday, but there was a study the other day, in the Journal of Cell Biology, that seriously calls in question the methodology used by Thompson Scientific to calculate the sacred Impact Factor, the magic number that makes and breaks lives and careers of scientists.
Apparently, it is really a magic number calculated in a mysterious way, not in the way that Thompson Scientific claims they do it. Who knows what subjective factors they include that they do not tell us about?
When we examined the data in the Thomson Scientific database, two things quickly became evident: first, there were numerous incorrect article-type designations. Many articles that we consider "front matter" were included in the denominator. This was true for all the journals we examined. Second, the numbers did not add up. The total number of citations for each journal was substantially fewer than the number published on the Thomson Scientific, Journal Citation Reports (JCR) website (http://portal.isiknowledge.com, subscription required). The difference in citation numbers was as high as 19% for a given journal, and the impact factor rankings of several journals were affected when the calculation was done using the purchased data (data not shown due to restrictions of the license agreement with Thomson Scientific).
Your database or mine?
When queried about the discrepancy, Thomson Scientific explained that they have two separate databases--one for their "Research Group" and one used for the published impact factors (the JCR). We had been sold the database from the "Research Group", which has fewer citations in it because the data have been vetted for erroneous records. "The JCR staff matches citations to journal titles, whereas the Research Services Group matches citations to individual articles", explained a Thomson Scientific representative. "Because some cited references are in error in terms of volume or page number, name of first author, and other data, these are missed by the Research Services Group."
When we requested the database used to calculate the published impact factors (i.e., including the erroneous records), Thomson Scientific sent us a second database. But these data still did not match the published impact factor data. This database appeared to have been assembled in an ad hoc manner to create a facsimile of the published data that might appease us. It did not.
Opaque data
It became clear that Thomson Scientific could not or (for some as yet unexplained reason) would not sell us the data used to calculate their published impact factor. If an author is unable to produce original data to verify a figure in one of our papers, we revoke the acceptance of the paper. We hope this account will convince some scientists and funding organizations to revoke their acceptance of impact factors as an accurate representation of the quality--or impact--of a paper published in a given journal.
Just as scientists would not accept the findings in a scientific paper without seeing the primary data, so should they not rely on Thomson Scientific's impact factor, which is based on hidden data. As more publication and citation data become available to the public through services like PubMed, PubMed Central, and Google Scholar®, we hope that people will begin to develop their own metrics for assessing scientific quality rather than rely on an ill-defined and manifestly unscientific number.
Of course, this was written in a polite language of science. But on a blog, I can say that this is at least very fishy and suspect. And several other bloggers seem to agree, including Bjoern Brembs, The Krafty Librarian, Eric Schnell, Peter Suber and Stevan Harnad who each dissect the paper in more detail than I do, so go and read their reactions.
- Log in to post comments
I think the Brembs post has it best. If not a nail in the coffin of IF, this is probably the best critique to launch serious debate that I've seen. Of course, if Nature's admission of the high degree of skew in their own distribution didn't in and of itself fix the problem....well one's hope must be tempered.
Impact factor always struck me as sort of a quarterback passer rating for journals. Whenever the announcers speak of a QB's impressive passer rating, they describe the value as the result of some complex equation with mysterious factors that the general viewing public is just too dense to comprehend. So take their word for it, the QB's great; you just wouldn't understand.
yajeev:
Imagine, however, if part of the QBs rating were how many officially licensed jerseys sold with his name and number. And if there was a bump for average TV audience. That would be more relevant.
and the referenced article would be taken as discovering that the Pats were able to convince the QB raters that they shouldn't count some fraction of Brady's interceptions whereas for some down-market backup everything was counted because the team didn't bother to argue about it.
Thomson Scientific Corrects Inaccuracies In Editorial
Article Titled "Show me the Data", Journal of Cell Biology, Vol. 179, No. 6, 1091-1092, 17 December 2007 (doi: 10.1083/jcb.200711140) is Misleading and Inaccurate
http://www.scientific.thomson.com/citationimpactforum/
-cross commenting-
interesting defense there. Mostly it is arguing about the tone taken by the JCB piece, fair enough. The point about refusing to negotiate with editors about the categorization of content gives us an alternative hypothesis. Namely that journals look at the way ISI decides on �front material� and makes sure to adapt their journal practices to produce the outcome desired. arguable.
I still think they are totally bogus on the mean vs median thing though. Isn�t it basic stat inference that when you have a skewed distribution the median is considered more representative and therefore �correct� as a description? the point is not whether it would increase or decrease the IF number, but whether it would be more accurate!