Assessing the impact of research and KTE activities
description
Transcript of Assessing the impact of research and KTE activities
Assessing the impact of research and KTE activities
Sandra Nutley
Agenda for this session
1. Key approaches to assessing the use and impact of research and knowledge mobilisation activities
2. Common challenges and methodological issues
3. Summary of findings from existing assessment activities
Why assess research impact?
• Addressing accountability• Assuring value for money• Setting priorities• Assisting learning• Improving outcomes
Summative or formative purposes?
Impact of research
Impact of initiatives to increase research use
Research use in ‘user’ communities
Systematic reviews
Centres promoting research use
1. Key approaches to assessing research impact
Target KTE interventions
Organisational research use
Single studies
Research programmes
Policymakers
PractitionersMedia
Forward tracking
Tracking back
Evaluation of initiatives
Forward tracking from research projects and programmes to use and impact
Two common methods:
• Describe and quantify impacts of research - E.g. Payback model 5 categories of possible impact:– Knowledge production– Research capacity building– Policy or product development– Sector benefits– Wider societal benefits
• Describe and map networks and flows of knowledge and the effects of any interactions
Tracking back from decisions or behaviours to research influences on these
• Bibliometric studies of policy documents and practice guidelines
• User panels – surveys and workshops• Ethnographic studies of communities of
practice• Economic impact evaluations – estimate
economic impact of a policy and estimate extent of research influence on that policy
Figure 5: Approaches and methodologies
0
10
20
30
40
50
60To
tal n
um
ber
of
pap
ers
Discussed
Used
What methods are used in practice?Systematic review by Boaz et al 2008
Evaluations of initiatives to increase research impact
• Standard programme evaluation strategies: – Qualitative investigation of processes and
perceived effects– Action research– Experimental and quasi-experimental methods
• Importance of strong theories of change & models of KTE to design and test interventions
• Need shared taxonomy of KTE strategies and interventions
KTE models
Ward et al 2009:• 63 different theories or models of KTE• Distilled down to 5 components of the KTE
process• Connection between them is seen as
interactive and multi-directional• Aim of framework is to provide a foundation
for gathering evidence about knowledge into action processes
Knowledge/Research
Context Barriers/Supports
Problem
Interventions Utilisation
Attributes of knowledge (relative advantage, complexity and compatibility)
How it is synthesised and adapted
Recognition of a need for action
Organisational, individual, environmental and structural
Distribution-type interventions Linkage-type interventions
Process of using intervention: cycles of selection, tailoring, implementation & evaluation
Conceptual, instrumental, political, procedural
Taxonomies of KTE strategiesE.g. Mitton et al 2007 – 8 key strategies• Face-to face exchange between researchers and
‘users’• Education sessions for users• Networks and communities of practice• Facilitated meetings between researchers and users• Interactive, multidisciplinary workshops• Capacity building in user organisations• Web-based information, electronic communications• Steering committees for research
projects/programmes
RURU taxonomy
• Intervention types and underlying mechanisms, with focus on the latter
• Five key mechanisms:– Dissemination– Interaction– Social influence– Facilitation– Incentives and reinforcement
2. Common challenges and methodological issues
• What types of use/impact are of interest?• When to assess impact?• Importance of context – assessing actual or
potential impacts?• Dealing with attribution and additionality –
constructing a convincing impact narrative• Getting away from linear models research
use/impact
What uses/impacts are of interest?
• Instrumental, conceptual or symbolic research use?
• Reach/awareness, impact on behaviour & service delivery, outcomes for service users?
• Positive and dysfunctional consequences?
When to assess impact?
Impact often occurs far down the line, but impact trails grow fainter over time
Tools for analysing context
• Accounting for the difference between actual and potential impacts
• E.g. Conducive policy environments - research more likely to be used when information, ideology and interests coincide and institutions can support developments (Weiss)
Constructing a convincing impact narrative
• Dealing with attribution – the potential of contribution analysis
Slide source: Wimbush 2010
Slide source: Wimbush 2010
Slide source: Wimbush 2010
Conclusions on methods• Case study methods to take account of differing
types of research and contexts for impact• Combination of quantitative and qualitative
methods and indicators within case study approach
• Need for research impact theory/model to guide data collection and analysis
• Limited benefits from studying the impacts of individual research projects
• Dangers of generalising from case sampling – because distribution of research impact is likely to be uneven
3. Summary of findings from the UK ESRC’s impact assessment activities
• The most important drivers of impact are:– Established relationships and networks with user
communities– Involving users at all stages with research– Well-planned user-engagement and KTE strategies– Portfolios of research activity that build reputations with
research users– Good infrastructure and management support– The involvement of intermediaries and knowledge brokers
as translators, amplifiers, network providers
RURU’s conclusions on generic features of effective KTE practices
Research must be translated - adaptation of findings to specific policy and practice contexts
Enthusiasm- of key individuals - personal contact is most effective
Contextual analysis - understanding and targeting specific barriers to, and enablers of, change
Credibility - strong evidence from trusted source, inc. endorsement from opinion leaders
Leadership - within research impact settings Support - ongoing financial, technical & emotional
support Integration - of new activities with existing systems
and activities
Implications for future research on research use and impact
• Much that we still need to know• We need to move away from:
– Poor documentation and under-evaluated KTE activities– Studies that focus only on the instrumental use of research– An assumption that research is used and applied mainly by
individual practitioners– Studies that result only in a now familiar listing of barriers
and enablers, especially where these are the barriers/enablers experienced by individual practitioners
Key messages• Need to understand more about how research-based
knowledge enters and flows through diverse policy, organisational and practice settings
• Need realistic assumptions about the nature and processes of research use and impact– these are many and complex
• No single model of research use is likely to be sufficient for all situations when impact is to be assessed
• Need to make choices about where and how to look for use and impact based on the purpose of the assessment
ReferencesMain reference• Davies HTO and Nutley SM (2008) ‘Learning More about How Research-
Based Knowledge Gets Used Guidance in the Development of New Empirical Research’, Working Paper for the WT Grant Foundation, New York
Other useful references• Boaz et al (2008) ‘Assessing the impact of research on policy: A review of
the literature’, Kings College London/ PSI• ESRC (2009) Taking Stock: A summary of ESRC’s work to evaluate the
impact of research on policy and practice, http://www.esrc.ac.uk/_images/Taking%20Stock_tcm8-4545.pdf
• ESRC (2010) Branching out: New direction in impact evaluation from the ESRC’s Evaluation Committee http://www.esrc.ac.uk/_images/Branching%20Out_tcm8-14881.pdf
• Meagher L, Lyall C and Nutley S (2008) ‘Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research’ Research Evaluation 17(3): 163-173
• Ward et al (2009) ‘Developing a framework for transferring knowledge into action: a thematic analysis of the literature’ Journal of Health Services Research and Policy 14(3): 156-164
For further information
The following three slides summarise the main research questions identified by Davies and Nutley (2008) when they were asked to define the emerging research agenda on research use and impact for the WT Grant Foundation. These questions are categorised under the three headings of:
• Knowledge source, presentation and integration
• Context and connections
• Strategies and process
Knowledge source, presentation and integration
• What models of research supply and synthesis might better support knowledge integration by potential users?
• How do different kinds of messaging and messengers affect the use of research knowledge?
• What is the role of the web in providing access to existing research?
• To what extent are policymakers and service managers conducting their own in-house research using administrative and local data?
• How is new knowledge integrated into current ways of thinking and models of practice?
Context and connections• What communities are involved in the field of interest?
How are these connected?• Are policy and practice communities networked and
does research-based knowledge flow across these networks?
• How, where and under what circumstances do practitioners source new knowledge?
• What models of research brokerage or intermediary activities have the best potential for fostering research use?
• How can lay people, service users, and others contribute more fully to evidence-informed discussions?
• How are education and CPD connected to and supportive of knowledge accumulation and integration?
Strategies and processes• Do different models of research use co-exist in
different practice settings?• What knowledge management strategies are used
across the sector?• What models of push, pull and linkage-exchange are
in place? How are these evolving and connecting?• What strategies for increasing research use and
impact have the best evidence in support of them, and how might they be tested to consolidate our knowledge base in this area?
• How can we get new knowledge on the research-action process to influence the future actions of researchers, funders, intermediaries, policy/decision-makers, and practitioners/end users?