Gather evidence to demonstrate the impact of your researchMaximize Your Impact Workshop Series

Heather Coates, MLS, MSDigital Scholarship & Data Management LibrarianIUPUI University Library Center for Digital Scholarship

OverviewProductsLevels of Metrics/EvidenceTypes of Metrics/Evidence

Examples

Data Sources & Tools

Overview

IUPUI Value

s

Civic Engagement

Collaboration

Diversity, Equity,

Inclusion

Economic Development

(Indiana)

Interdisciplinary Work &

Publication

Open Access

Public Scholars/

Scholarship

R&CA in Urban Environment

RISE

Translational Research

https://academicaffairs.iupui.edu/PromotionTenure/IUPUI-Guidelines

Why do we need metrics?• P&T review committees can’t read everything• P&T review committee members are not experts in your

field• Models for producing scholarship differ by discipline and

research methods used• Some metrics are decent indicators of scholarly use

(e.g., citation)• Some metrics are decent indicators of public discussion

(e.g., Twitter, Facebook, news media attention)• Some metrics are decent indicators of reuse (e.g., data

citations)

Metrics are only one kind of evidence

How do we measure…• Relationships developed with practitioners and

patients• Affecting change in how stakeholders see an issue• The long-term benefits of community partnerships to

document and preserve local history• Increases in the civic literacy of adults across central

Indiana• Increased understanding of an issue by the state

legislature resulting from ongoing policy discussions

Consider

Leiden Manifesto for research metrics: http://www.leidenmanifesto.org/

San Francisco Declaration on Research Assessment (DORA): http://www.ascb.org/dora/

“Do not use journal-based metrics, such as Journal Impact Factors (JIFs), as surrogate measures of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.”

Citation-based Metrics

Journal Level Metrics

Output/Article Level

Metrics

Author Level

Metrics

Citation Metrics: Levels of Evidence

• Article/Item-level: – citation counts, Field Weighted Citation Impact

(Scopus), Eigenfactor Article Influence Score, Relative Citation Ratio (NIH - https://icite.od.nih.gov/)

• Journal-level: – Journal Impact Factor, 5-year JIF, Eigenfactor

• Author-level: – h-index, i-10 index (Google Scholar)

Web Metrics

View Use

Social Media Metrics

Share

Recommend

Discuss

Altmetrics

Altmetrics• Supplement traditional metrics• Scope extends beyond the formal published

scholarly record (journals & books)• Timeframe: immediate to short-term impact• Sources: focus is online attention, interaction,

usage• DOES capture and link to qualitative data (ex: blog

snippets, tweet content)

Altmetrics Sources• Publishers

• Aggregation services– Impact Story:

http://impactstory.org/– Altmetric.com:

http://www.altmetric.com/– Plum Analytics:

https://plu.mx/g/samples

• Subject repositories– PubMedCentral (PMC)– arXiv– SSRN

• Institutional repositories– IUPUI ScholarWorks:

http://scholarworks.iupui.edu – IUPUI DataWorks:

http://dataworks.iupui.edu

Aggregator Sources• News media• Social media platforms• Blogs• Twitter• Reference managers (e.g., Mendeley)• other online conversations captured include

YouTube, Wikipedia, Reddit, F1000, Q&A, Policy Docs

https://www.altmetric.com/about-our-data/our-sources/

Using altmetrics to demonstrate impact

• Complementary/Supplemental– More immediate (before citations accrue)– Indications of impact outside your discipline and academia – Reflect engagement with and impact upon a broader audience,

beyond academia and community of practice

• Flavors of impact (Priem et al, 2012)– “popular hit” – highly tweeted and shared on non-academic social

media sites– “expert pick” – good F1000 ratings and subsequent citations, few

shares or social media mentions

https://becker.wustl.edu/impact-assessment

Impact IndicatorsResearch Output & Activities

Biological materials, collaborations, data, databases, repositories, techniques & procedures, grey literature, invention disclosures, mobile apps, patents, trainees, etc.

Advancement of Knowledge

Books/chapters (inclusion in bibliographies, library ownership, textbook use), change in understanding (paradigm shift, lead to new approach), citations (first & second generation citations, countries and institutions represented), conference themes, new centers/institutes

Clinical Implementation (or TRIP)

Biological materials, study cited in clinical decision aid, clinical/practice guidelines, diagnostic application, instruments, quality measure guidelines (gov’t or NPO), reporting requirements

Legislation & Policy

Committee participation, study cited in guidelines, study cited in policy, study cited in enactment of standards

Economic Benefit

Findings cited in reduced costs for delivery of healthcare services, findings result in enhancement of existing resources and expertise, license agreements for use of IP, spinoff or startup company

Community Benefit

Public awareness of risk factors, patient decision materials, cited in public/private insurance benefit plan

Examples

Citation Metrics Altmetrics

Product Title Journal/ SourceYear of Publication

JIF [Quartile Rank] Citations

Field-Weighted Citation Impact

Citation Benchmarking

Mendeley reads Twitter Blogs

Dietary inferences from dental occlusal microwear at mission San Luis de Apalachee

American Journal of Physical Anthropology 2005 2.1 [Q3] 22 1.51 55th percentile 47 0 0

A comparison of calcium to zoledronic acid for improvement of cortical bone in an animal model of CKD

Journal of Bone and Mineral Research 2014 6.8 [Q1] 16 3.9 86th percentile 16 1 0

Reference-point indentation correlates with bone toughness assessed using whole-bone traditional mechanical testing Bone 2013 4.5 [Q1] 48 6.28 97th percentile 53 2 0

In vivo reference point indentation reveals positive effects of raloxifene on mechanical properties following 6 months of treatment in skeletally mature beagle dogs Bone 2013 4.5 [Q1] 27 3.47 94th percentile 19 0 1

Structure and function of platyrrhine caudal vertebrae Anatomical Record 2010 1.4 [Q3] 23 2.39 84th percentile 31 0 1

Functional correlates of fiber architecture of the lateral caudal musculature in prehensile and nonprehensile tails of the platyrrhini (primates) and procyonidae (carnivora) Anatomical Record 2009 1.5 [Q3] 23 2.04 80th percentile 103 0 1

Contours of the hominoid lateral tibial condyle with implications for Australopithecus

Journal of Human Evolution 2006 3.3 [Q2] 24 1.16 60th percentile 40 0 0

Product TitleItem

Views

Item Downloa

ds

Physician Workforce Collection 735 1363

Mental Health Workforce 819 1004

Nursing Workforce 662 1258

Oral Health 577 849

Primary Care Workforce 284 512

Pharmacy 447 432

Full Collection 3545 6019

How is this being disseminated to local and statewide communities of practice?

Qualitative feedback on these reports

Data literacy instructional materials (shared in 2014) Views

Downloads Favorites

Email Shares

Data Management Lab: Session 1 Slides 700 4 2 0Data Management Lab: Data management plan instructions 376 2 0 0Data Management Lab: Data mapping exercise instructions 807 4 0 0Data Management Lab: Data mapping exercise example 5407 5 0 1Data Management Lab: Session 2 slides 432 2 0 0Data Management Lab: Session 2 - Documentation Instructions 327 2 0 0Data Management Lab: Session 3 Slides 645 2 0 0Data Management Lab: Session 3 Data Review Checklist 402 3 0 0Data Management Lab: Session 3 Data Entry Best Practices 384 2 0 0Data Management Lab: Session 3 Data Coding Best Practices 270 2 0 0Data Management Lab: Session 4 Slides 590 6 0 0Data Management Lab: Session 4 Review Outline 538 3 0 0Data Topics Series: Ensuring data quality 550 11 0 1Data Topics Series: Preventing data loss 461 2 0 0Data Topics Series: Practical Data Management Plans 400 2 0 0

http://opensyllabusproject.org/

Chronicling Hoosier

https://katiemartinpapers.wordpress.com/2016/08/07/abstract-analysis-chronicling-hoosier/

Top 5 Non-Article OutputsTitle Impact EvidenceHuntington Study Group website Community

Receives more than 2,000 unique visitors per month.

HSG participant database Community4300 registered potential participants

Research sites Community  63 research sites in 11 countries

Physician's Guide to the Management of Huntington Disease

Clinical Implementation/TRIP Downloaded more than 2,400 times

UHDRSClinical Implementation/TRIP

 Downloaded more than 13,000 times

Data Sources & Tools

Sources of Data• Web of Science• Journal Citation Reports (JCR)• Scopus• Google Scholar Citations• ImpactStory• Altmetric Bookmarklet• Publisher 

Google Scholar

Gathering Data• Gather manually– Free– Time-consuming–Messy & redundant– Tailored to your

scholarship & argument– Include unusual or field-

specific sources • Use an aggregator

–Minimal cost (individual)– Less time-consuming– Generic, broad

presentation– Heavy emphasis on major

social media channels

https://impactstory.org/u/0000-0002-4356-3837

The problem is not a lack of data, but how to

evaluate, select, and make sense of it.

Using metrics responsibly• What is your case? Your story?–What statements do you need to support?

• Present the data in context

• Provide a range of evidence – metrics + qualitative evidence

Resources• Sarli, C. C., Dubinsky, E. K., & Holmes, K. L. (2010). Beyond citation analysis: a

model for assessment of research impact. Journal of the American Medical Library Association.

• Piwowar, H. & Priem, J. (2013). The power of altmetrics on a CV. Bulletin of the Association for Information Science & Technology. https://www.asis.org/Bulletin/Apr-13/AprMay13_Piwowar_Priem.html

• Roemer, R. C. & Borchardt, R. (2012). From bibliometrics to altmetrics: A changing scholarly landscape. College and Research Libraries News, 73(10), 596-600. http://crln.acrl.org/content/73/10/596.full

• Dinsmore, A., Allen, L., & Dolby, K. (2014). Alternative perspectives on impact: The potential of ALMs and altmetrics to inform funders about research impact. PLoS Biology, 12(11), e1002003. doi: 10.1371/journal.pbio.1002003

• Steven Roberts website & P&T dossier, both incorporating altmetrics: http://faculty.washington.edu/sr320/?p=2806

• Altmetric webinar: Applied Altmetrics: http://godigitalscience.com/view/mail?iID=Y9PXAVDJMH5JVAUPJU79

Images• Publish: https://thenounproject.com/search/?q=article&i=74762• Evaluate: https://thenounproject.com/search/?q=peer+review&i=210141 • Article: https://thenounproject.com/search/?q=article&i=74762 • Book: https://thenounproject.com/search/?q=book&i=547809• Binary code: https://thenounproject.com/search/?q=binary+code&i=177042 • Data stream: https://thenounproject.com/search/?q=data+stream&i=160443 • Q&A:

https://www.readytalk.com/blog/readytalk-admin/question-how-do-i-manage-the-qanda-portion-of-my-webinar

Heather CoatesDigital Scholarship & Data Management LibrarianUniversity Library Center for Digital Scholarship

hcoates@iupui.edu https://ulib.iupui.edu/digitalscholarship/impact