NSF Workshop on Scholarly Evaluation Metrics – Afternoon 2

Continuing stream of consciousness notes from this workshop held in DC, Wednesday December 16, 2009

Alexis-Michel Mugabushaka. (European Research Council) - intertwined research funding structures at national and European level.

At the national level two main funding modes - institutional (block research funding of higher ed institutions), and competitive.

Orgs structured at European level - like CERN or EMBL. Joint research funding ESF or bi or multilateral. Research Frameworks. ERC

ERC Scientific Council - 22 eminent scientists. Executive Agency (where he works). 2 programs: starting grants, advanced grants.

Evaluation activities of national funding agencies. ESF - 80 orgs, 20 countries. Venue to support members sharing info on evaluation methods. It's not whether or not to fund basic science - that's not on the table for them - they're interested in how to best fund it, what schemes, and what types of research in which areas.

challenge 1 - high quality data: get data without over burdening researchers - allow long term (post project) monitoring of effects

challenge 2 - ?

recent trends - metrics, broader contextual questions, other evaluation questions (societal impact)

wish list - things mentioned earlier, awards and recognition (best paper prizes), better metrics for patents (measuring patents influences universities to patent, but that costs a lot of money, pulling from other areas, and licenses don't always cover). How to reward communicating to a non-academic audience (over value publications over other contributions to the community - tools, software, teaching, public communication, etc).

audience q: also need to add "micro attribution" - how much data is being put in public repositories, how is that data reused.

 

Jan Velterop- Concept Web Alliance - semantic web - forging quality out of quantity. Why publish: record, transfer, credit.

"scientific egosystem" "acknowledgement economy" - all of the standard metrics are popularity contests - not really about quality at all. What does quality look like? An edifice with an ivory tower and a foundation of hard packed rubble (no citations, but support the entire edifice). What if your article has no citations but is heavily used in teaching? It's better quality if it's got more formulas? It's better if you can't understand it?

What about brownie points for commenters on plos1 or f1000.

Nano-publications - triples with attributes like author. what must a publication be? citable, valuable, convey knowledge, full length- no. One number for each concept (in any language). Link to concept wiki for attributes/definition - need browser. So maybe we can have metrics for first use and times reused. Brownie points for annotation. Whole datasets can be triplized. Can be cited. Benefit authors and publishers. see http://bit.ly/6B1XqK (reminds me of a project from Catherine Blake where she's doing text mining for knowledge claims to map them from article to article - but hers seems more sophisticated. Also, Jan doesn't mention if they are linking these concept wiki pages to community accepted naming schemes for genes, chemicals, stars, etc)

More like this

One of the open problems in article level metrics is how to automate, quantify, and describe the exposure an article has had in popular science pieces in newspapers and general science magazines. Peter Binfield (PLoS) and Alexis-Michel Mugabushaka. (European Research Council) both brought this up…
this continues my stream of consciousness notes from the workshop held in DC, December 16, 2009. Peter Binfield (PLOS) - article level metrics. Not talking about OA, not talking about journal level.  Journal is just packaging, and shouldn't necessarily judge articles by the packaging. PLoS ONE has…
I attended this one-day workshop in DC on Wednesday, December 16, 2009. These are stream of consciousness notes. Herbert Van de Sompel (LANL) - intro - Lots of metrics: some accepted in some areas and not others, some widely available on platforms in the information industry and others not. How are…
If you are a regular reader of this blog, you are certainly aware that PLoS has started making article-level metrics available for all articles. Today, we added one of the most important sets of such metrics - the number of times the article was downloaded. Each article now has a new tab on the top…

"Also, Jan doesn't mention if they are linking these concept wiki pages to community accepted naming schemes for genes, chemicals, stars, etc)"

Yep. We do. Extensively and we try to be comprehensive in that regard. Anyway, you can add accepted names, should you miss any.

Jan

By Jan Velterop (not verified) on 02 Jan 2010 #permalink