NSF Workshop on Scholarly Evaluation Metrics – Morning 2

Continuing my stream of consciousness notes from this meeting in DC, Wednesday, December 16, 2009.

Jevin D West (U Washington, Eigenfactor) - biology and bibliometrics. biology has a lot of problems that are studied looking at networks. From ecosystems to genomes. They want to take these huge networks and be able to tell stories. The citation network is a model for information flow that they can then use in biology. WoS 8k journals, 15 years, 60M citations. Goals of eigenfactor: develop tools to comprehend large networks in all areas of science - employ these tools to understand scholarly communication in science.

Eigenfactor - based on Bonaicich (1972). Not just that you have friends, but how important your friends are.  He showed us an example of it being calculated using a voting model.  Citations from highly cited journals are worth more. Citations from non-review journals are worth more (if something has a lot of links out, dilutes voting). Article influence (?). Economics R=0.83 - correlated IF and article influence.

Mapping over time see Rosvall & Bergstrom, 2009 - alluvial map.

audience q: combine your map and Johan's (a: they're working on it)

audience q: time period (IF is 2) - but can change and depends by field. Can find citation peak for each field (fields found using clustering algorithms), and then use those as the parameter

 

Jorge Hirsch (UCSD) - H index fame - thought it up in 2003, then a couple years later posted a paper to arxiv, interviewed by science then articles in... so eventually pnas paper. This really took off. 2007 paper, predictive power of h index.

How to take into account multiple authors? - 3 modifications suggested, all dividing. this doesn't take author role/position into account and it discourages collaboration.

His proposal hbar index - takes into account if the article is in the h core of any of the co-authors. A single authored paper counts, a paper authored with a more junior colleague counts, one that is co-authored with a senior colleague (if their h index is 60, the paper has 56, then it is in their core) it doesn't count. See arXiv: 0911.3144v1 This would discourage authoring with a more senior author, but there are lots of reasons to do that, so not an issue. This would discourage honorary authorships by the lab head.

A metric is good if it identifies good scientists doing good science, helps decision making, and results in better science being done.

More like this

I attended this one-day workshop in DC on Wednesday, December 16, 2009. These are stream of consciousness notes. Herbert Van de Sompel (LANL) - intro - Lots of metrics: some accepted in some areas and not others, some widely available on platforms in the information industry and others not. How are…
Interesting conversation at lunch today: topic was academic performance metrics and of course the dreaded citation index came up, with all its variants, flaws and systematics. However, my attention was drawn to a citation metric which, on brief analysis, and testing, seems to be annoyingly reliable…
In scientific publishing, one of the important things is what is known as the "impact factor" which is the the average number of citations a journal receives over a 2 year period. The impact factor is often used by librarians and researchers to determine which journals to purchase and where to…
The gold standard for measuring the impact of a scientific paper is counting the number of other papers that cite that paper. However, due to the drawn-out nature of the scientific publication process, there is a lag of at least a year or so after a paper is published before citations to it even…