Evaluating the impact of health research:

Revisiting the Canadian Institutes of Health Research Impact Assessment Framework

Nicola Lauzon, Marc Turcotte & Laura McAuley

Impact Assessments Unit

Canadian Institutes of Health Research

Ottawa, Ontario, Canada

nicola.lauzon@cihr.gc.ca

Presentation Overview

• CIHR impact assessment framework history

• Study purpose & methods

• Findings• Uses of CIHR framework• Focus group feedback from framework users

• Application of Lessons learned• Changes to the framework categories• Development of a simplified conceptual model

• Next steps

• Questions

2

CIHR Impact Assessment Framework 2006

Category Description

Advancing Knowledge

• discoveries, breakthroughs, and contributions to the scientific literature

Building Capacity

• development and enhancement of research skills in individuals and teams

Informing Decision-Making

• impacts in the areas of science, public, clinical, and managerial decision-making, practice and policy

Health & Health Systems Impacts

• advances in prevention, diagnosis, treatment and palliation, functioning of the health system

Economic Impacts

• commercialization of research discoveries and human capital gains

3

Study Purpose & Methods

Purpose• To document lessons learned and inform framework improvement

Methods• Document review of impact reports & protocols

• How has it been used, results of use, lessons learned

• Focus groups• Feedback and input from users and potential users of framework

• Six groups; total 35 participants

4

Results: Document Review

Uses of CIHR framework

• Topic-based assessment• Focus on specific disease condition

• Investigate impacts across multiple categories

• Institute-based assessment• Focus on research mandate areas of CIHR Institutes

• Investigate impacts across multiple categories

• Category-based assessment• Focus exclusively on one impact category

5

Topic-based impact assessments

Topic• Cardiovascular disease• Obesity• SARS

Main Lessons learned• Validation of CIHR impact categories• Impacts continue after 20+ years• Attribution difficult• Topic dictates types of impacts seen

6

Institute-based impact assessments

CIHR International Review (2010 – 2011)

• Impacts of broad mandate areas of all 13 CIHR Institutes between 2000 – 2010

Lessons learned• Limited availability of indicators in latter impact categories• Need to revisit the categorization and utility of certain indicators• Potential to improve bibliometric methodology

7

Results: Focus Groups

Main theme of discussions:• Attribution versus contribution

• Direct attribution desired

• Struggle to find value of latter categories and associated indicators

Lessons Learned• Need to clarify to stakeholders:

• Our conceptualization of the research impact process

• How CIHR fits into the research impact process

• How the CIHR framework fits into the research impact process

8

Simplified conceptual model of research impact

9

Research Enterprise Research Users

Society

KnowledgeDecision-Making

Health

Knowledge Translation

Research Findings EffectUseFunding

Impacts on…

Capacity

Health System

Economy

Research Enterprise Research Users

Society

KnowledgeDecision-Making

Health

Knowledge Translation

Research Findings EffectUseFunding

Impacts on…

Capacity

Health System

Economy

Not-so-simplified version

10

Research Enterprise

Research Users

Society

KnowledgeDecision-Making

Health

Knowledge Translation

Research Findings EffectUseFunding

Impacts on…

Capacity

Health System

Economy

Not-so-simplified version

11

Government priorities

Public prioriti

es

Funder priorities

Gov

ernm

ent b

udge

t

Partnerships &

leveraged funds

Pe

er re

view

Researcher

priorities

Research

Institution

priorities

Infr

astr

uctu

re

Re

sea

rch

te

am

co

mp

osi

tion

Publicatio

n bias

Pe

er re

view

Re

sea

rch

er

cha

ract

eris

tics

Tim

e la

g t

o p

ub

lica

tion

Continued relevance

User

characteristics

Governm

ent

prioritiesU

ser a

ccep

tan

ce

Acce

ssibility

Sys

tem

cap

acity

Eco

nom

ic o

utlo

ok

Per

sona

l ch

arac

teris

tics

Dem

and

Po

litical clim

ate

Syste

m ca

pa

city

Simplified conceptual model of research impact

12

Research Enterprise Research Users

Society

KnowledgeDecision-Making

Health

Knowledge Translation

Research Findings EffectUseFunding

Impacts on…

Capacity

Health System

Economy

Simplified conceptual model of research impact

13

Research Enterprise Research Users

Society

KnowledgeDecision-Making

Health

Knowledge Translation

Research Findings EffectUseFunding

Impacts on…

Capacity

Health System

Economy

Simplified conceptual model of research impact

14

Research Enterprise Research Users

Society

KnowledgeDecision-Making

Health

Knowledge Translation

Research Findings EffectUseFunding

Impacts on…

Capacity

Health System

Economy

Simplified conceptual model of research impact

15

Impacts on…Knowledge

Capacity

Decision-Making

Health

Health System

Economy

Research Enterprise Research Users

Society

Knowledge Translation

Research Findings EffectUseFunding

16

CIHR Impact Assessment Framework

Category Lower order indicators Higher order indicators

Knowledge • # Publications from grants in specific research area

• # Canadian publications in specific research area

Capacity • # CIHR-supported PhD trainee awards

• # Canadian PhD grads in health

Decision-Making

• Citation in research • Citation in health policy

Health • Outcome measures of condition X for clinical trial patients

• Prevalence of condition X in Canada

Health Systems

• Health efficiencies from specific grants

• Wait times

Economy • License or royalty agreement counts

• # jobs created from spin-off companies

Next steps & Conclusions

• Continue testing model, framework categories and indicators

• Feedback & validation from stakeholders

• Alignment of model and framework with literature

17

Questions???

18