Measuring Research Impact on the Web
-
Upload
charleston-conference -
Category
Technology
-
view
283 -
download
4
description
Transcript of Measuring Research Impact on the Web
MEASURING RESEARCH IMPACT ON THE WEB
Charleston Conference, November 2013
Iain HrynaszkiewiczOutreach Director, Faculty of 1000
[email protected]://f1000.com
@iainh_z
The Seer of Science PublishingScience 4 October 2013: Vol. 342 no. 6154 pp. 66-67 DOI: 10.1126/science.342.6154.66 http://www.sciencemag.org/content/342/6154/66.full.pdf
http://blog.f1000.com/2013/10/04/vitek-tracz-science-interview/
1. Individual papers and scientists judged on journal-based metrics
2. Closed, pre-publication peer review3. Lack of access to original research – and data4. Lack of credit for many of scientists’ contributions
SOME PROBLEMS WITH SCHOLARLY COMMUNICATION
SOME PROBLEMS WITH SCHOLARLY COMMUNICATION #1
“[Journal Impact Factor is] a poor indicator of citations to specific
papers or of the future performance of individual researchers”
Nature Materials 12, 89 (2013)
“I am sick of impact factors and so is science.”
Prof Stephen Curry, Imperial College, Aug 2012
http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/
“The widely held notion that high-impact publications determine who gets academic
jobs, grants and tenure is wrong.”
Dr Michael Eisen, UC Berkeley, Feb 2012http://www.michaeleisen.org/blog/?p=911
“Citations are heavily gamed and are painfully slow to accumulate, and overlook increasingly important societal and clinical
impacts.”
Priem et al., PLoS ONE 7(11): e48753
http://am.ascb.org/dora/
Themes of DORA recommendations:
1.Eliminate the use of journal-based metrics in funding, appointment, and promotion considerations2.Assess research on its own merits not the journal3.Capitalize on the opportunities provided by online publication such as exploring new indicators of significance and impact
DORA has been signed by >9000 individuals and nearly 400 organisations. Original signatories included American Association for the Advancement of Science (AAAS), American Society for Cell Biology, Howard Hughes Medical Institute, Faculty of 1000, Public Library of Science
IMPACT FACTORS AND CITATION METRICS
Advantages Disadvantages
Reproducible Slow – delay of up to 2 years
Transparent calculation (kind of) Data not publicly available
Curated – some human filtering Can be manipulated
Predictor of journal quality Poor predictor of paper and researcher quality
I, Cawi 2001 [CC-BY-SA-2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commonshttp://commons.wikimedia.org/wiki/File:Internet-Sign.jpg
“ALTERNATIVE” METRICS
Source: Wikipedia, http://en.wikipedia.org/wiki/Alternative
ALTMETRICS – WHY ARE THEY IMPORTANT?
“For all new grant applications from 14 January, the US National Science Foundation (NSF) asks a principal investigator to list his or her research
“products” rather than “publications” in the biographical sketch section. This means that, according to the NSF, a scientist's worth is not dependent
solely on publications. Data sets, software and other non-traditional research products will count too.”
Heather Piwowar, UBC, in Nature Jan 2013http://www.nature.com/nature/journal/v493/n7431/full/493159a.html
ALTMETRICS – WHY ARE THEY IMPORTANT?
“Metrics --- such as the number of article citations, your h-index or others (such as those available at ImpactStory.org) --- can be useful in making the
case that the publication or scholarly work was significant. The Faculty Promotions Committee discourages the use of journal-based metrics (such
as journal impact factors), since it is the quality and importance of the research contribution itself that is the key.”
Excerpt from UC Denver promotions handbook guide to dossier preparation
TOOLS FOR SHARING RESEARCH PRODUCTS...
AND TOOLS TO INTERACT WITH THEM
Source: http://impactstory.org/collection/v7b25c
Source: http://www.altmetric.com/details.php?citation_id=938211
(ALT)METRICS – ADVANTAGES AND DISADVANTAGES
Advantages Disadvantages
Fast – data available immediately Heterogeneity across tools
Transparent (mostly) Can lack context/meaning
Lots of open data and tools available Can be manipulated
All research products tracked Not always reproducible (ephemeral)
Much broader picture of impact Emergence and disappearance of tools
WHAT’S OFTEN MISSING?
By Taco Hoekwater; Wolfgang Schuster; Xan; svg from Lumu [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commonshttp://upload.wikimedia.org/wikipedia/commons/0/0d/ConTeXt_Unofficial_Logo.svg
HOW F1000PRIME WORKS
• Faculty includes over 5000 peer-nominated scientists and clinical researchers and ~5000 Associates
• Faculty Members select, rate and comment on the most interesting and important research articles (2-3% of the life science literature) from ~3,700 journals
• Assigns one of three positive ratings: Exceptional (3 stars), Very Good (2 stars) or Good (1 star)
• Text also serves as a short, expert, plain English recommendation written for a global readership
• Adds relevant classifications (e.g. changes clinical practice)
• Publishes about 1500 recommendations per month (>143,000 published to date)
EXAMPLE RECOMMENDATION
F1000, ALTMETRICS AND RESEARCH ASSESSMENT/IMPACT
The Wellcome Trust published in PLOS ONE an analysis of F1000Prime, finding that expert review highlights important papers that bibliometric indicators alone would miss (http://bit.ly/F1000Wellcome).
A 2009 study by the MRC concluded that F1000Prime recommendations are an indicator of future citation impact (http://bit.ly/F1000future).
Collection of studies of F1000Prime:
http://www.mendeley.com/groups/3748101/studies-involving-faculty-of-1000-s-f1000prime/
Also, studies have shown that tweets and number of readers on Mendeley predict citations.
How much does all this matter?
SOME PERSPECTIVE. WHAT IS IMPACT?
IS THIS IMPACT?
Source: http://www.jmedicalcasereports.com/mostviewed/alltime
THIS IS IMPACT
Source: http://www.ncbi.nlm.nih.gov/pubmed/15555666
SUMMING UP
• Impact factors are not good for assessing individual papers and individuals
• The internet has given us a wealth of new data and tools to give a broader picture of the impact of research
• We are still trying to understand the meaning of many new metrics – we need data with context
• Some “alternative” metrics are beginning to influence research assessment
• All metrics – citation and non-citation metrics – have advantages and disadvantages