NSF Workshop on Scholarly Evaluation Metrics – Morning 2

Continuing my stream of consciousness notes from this meeting in DC, Wednesday, December 16, 2009.

Jevin D West (U Washington, Eigenfactor) - biology and bibliometrics. biology has a lot of problems that are studied looking at networks. From ecosystems to genomes. They want to take these huge networks and be able to tell stories. The citation network is a model for information flow that they can then use in biology. WoS 8k journals, 15 years, 60M citations. Goals of eigenfactor: develop tools to comprehend large networks in all areas of science - employ these tools to understand scholarly communication in science.

Eigenfactor - based on Bonaicich (1972). Not just that you have friends, but how important your friends are.  He showed us an example of it being calculated using a voting model.  Citations from highly cited journals are worth more. Citations from non-review journals are worth more (if something has a lot of links out, dilutes voting). Article influence (?). Economics R=0.83 - correlated IF and article influence.

Mapping over time see Rosvall & Bergstrom, 2009 - alluvial map.

audience q: combine your map and Johan's (a: they're working on it)

audience q: time period (IF is 2) - but can change and depends by field. Can find citation peak for each field (fields found using clustering algorithms), and then use those as the parameter

Jorge Hirsch (UCSD) - H index fame - thought it up in 2003, then a couple years later posted a paper to arxiv, interviewed by science then articles in... so eventually pnas paper. This really took off. 2007 paper, predictive power of h index.

How to take into account multiple authors? - 3 modifications suggested, all dividing. this doesn't take author role/position into account and it discourages collaboration.

His proposal hbar index - takes into account if the article is in the h core of any of the co-authors. A single authored paper counts, a paper authored with a more junior colleague counts, one that is co-authored with a senior colleague (if their h index is 60, the paper has 56, then it is in their core) it doesn't count. See arXiv: 0911.3144v1 This would discourage authoring with a more senior author, but there are lots of reasons to do that, so not an issue. This would discourage honorary authorships by the lab head.

A metric is good if it identifies good scientists doing good science, helps decision making, and results in better science being done.