NSF Workshop on Scholarly Evaluation Metrics – Morning 2

Continuing my stream of consciousness notes from this meeting in DC, Wednesday, December 16, 2009.

Jevin D West (U Washington, Eigenfactor) - biology and bibliometrics. biology has a lot of problems that are studied looking at networks. From ecosystems to genomes. They want to take these huge networks and be able to tell stories. The citation network is a model for information flow that they can then use in biology. WoS 8k journals, 15 years, 60M citations. Goals of eigenfactor: develop tools to comprehend large networks in all areas of science - employ these tools to understand scholarly communication in science.

Eigenfactor - based on Bonaicich (1972). Not just that you have friends, but how important your friends are.  He showed us an example of it being calculated using a voting model.  Citations from highly cited journals are worth more. Citations from non-review journals are worth more (if something has a lot of links out, dilutes voting). Article influence (?). Economics R=0.83 - correlated IF and article influence.

Mapping over time see Rosvall & Bergstrom, 2009 - alluvial map.

audience q: combine your map and Johan's (a: they're working on it)

audience q: time period (IF is 2) - but can change and depends by field. Can find citation peak for each field (fields found using clustering algorithms), and then use those as the parameter

 

Jorge Hirsch (UCSD) - H index fame - thought it up in 2003, then a couple years later posted a paper to arxiv, interviewed by science then articles in... so eventually pnas paper. This really took off. 2007 paper, predictive power of h index.

How to take into account multiple authors? - 3 modifications suggested, all dividing. this doesn't take author role/position into account and it discourages collaboration.

His proposal hbar index - takes into account if the article is in the h core of any of the co-authors. A single authored paper counts, a paper authored with a more junior colleague counts, one that is co-authored with a senior colleague (if their h index is 60, the paper has 56, then it is in their core) it doesn't count. See arXiv: 0911.3144v1 This would discourage authoring with a more senior author, but there are lots of reasons to do that, so not an issue. This would discourage honorary authorships by the lab head.

A metric is good if it identifies good scientists doing good science, helps decision making, and results in better science being done.

More like this

In scientific publishing, one of the important things is what is known as the "impact factor" which is the the average number of citations a journal receives over a 2 year period.
A while ago, I posted about eigenFACTOR, a bioinformatics tool that can be used to calculate the relative impact of scientific journals.
The nice folks at Seed named me a revolutionary mind this month. It's an honor to be in the company of the minds behind GISAID, Eigenfactor, Voice of Young Science, and CubeSat.