Towards KT Measurement:Towards KT Measurement: What Constitutes KT Outcomes What Constitutes KT Outcomes

and Outcomes Monitoring in and Outcomes Monitoring in the NIDRR Contextthe NIDRR Context

(An Evaluation Perspective)(An Evaluation Perspective)

Presentation to National Center of Dissemination of Disability Research

(NCDDR) Workshop: December 3, 2007

Margaret L. Campbell, Ph.D Senior Scientist for Planning and Evaluation

National Institute on Disability and Rehabilitation Research

Focus is on KT as a set of strategies for accelerating the uptake of R&D findings to improve systems and benefit end users – i.e., the desired result of KT is “getting to outcomes.”

R&D outcomes do not occur without some type of KT, including publication.

Initial questions posed by NCDDR for this segment of the workshop are in many ways ahead of NIDRR’s development of KT and integration into planning and evaluation. Because of this, it is necessary to reformulate the objectives of this presentation.

Working Assumptions and Working Assumptions and Perspective of PresentationPerspective of Presentation

Illustrate how NIDRR’s existing strategic framework can be used to clarify what constitutes an outcome and to suggest possible theoretical KT outcomes.

Describe components of NIDRR’s existing evaluation framework, explain current strategies used to monitor and assess the quality of program outcomes, and present quantitative results from recent Annual Performance Assessment Expert Review (APAER) process.

Use examples of “noteworthy” accomplishments from 2007 APAER for Participation and Community Living to illustrate the kinds of outputs and outcomes grantees are reporting and whether they correspond to suggested theoretical KT outcomes.

Presentation ObjectivesPresentation Objectives

NIDRR’s Long-Range Plan and Logic Model help clarify and identify possible theoretically KT outcomes to be tracked and evaluated for quality? The Long-Range Plan identifies strategic goals and

objectives and provides a roadmap to the kinds of program outcomes NIDRR is seeking;

The Logic Model identifies outcome arenas that correspond to strategic goals, types of R&D outcomes, and intermediary and intended beneficiaries.

Part 1:Part 1: Illustrate Utility of Illustrate Utility of NIDRR’s Strategic FrameworkNIDRR’s Strategic Framework

To advance knowledge through capacity-building: By increasing capacity to conduct and use high quality and relevant

disability and rehabilitation research and related activities designed to guide decision making, change practice and enhance the lives of individuals with disabilities.

To advance knowledge through research and related activities: By generating science-based knowledge, technologies and applications

in order to contribute to policy; change practice; and enhance the lives of individuals with disabilities.

To advance knowledge through translation by utilizing science-based knowledge, technologies and applications: By promoting the effective use of science-based knowledge,

technologies and applications to contribute to disability and rehabilitation policy; improve practice; and enhance the lives of individuals with disabilities.

NIDRR’s Strategic Goals, NIDRR’s Strategic Goals, Long-Long-Range Plan – 2005-2009Range Plan – 2005-2009

What is a logic Model? Graphic blueprint of how a program is supposed to work to achieve benefits for participants. Ideally, a logic model depicts both the “theory of the problem” and the “theory of program.”

Utility of logic modeling? In additional to serving as a valuable communication tool, logic modeling ideally helps researchers, program managers and evaluators: Chart the pathways by which program activities and

outputs will have their desired effects or outcomes; Identify “indicators of success” or milestones along the

outcomes pathway that occur with the “span of accountability” of the program.

Brief Background on Logic Brief Background on Logic Models Models (Making sure we’re on the same (Making sure we’re on the same page)page)

Situation or “theory of the problem” to be addressed Outcome Arenas -- correspond to strategic goals of the LRP

for CB, R&D and KT and point to where program results occur. Types of Outcomes:

Short-term – advances in understanding, knowledge, skills and learning knowledge that occur through discoveries, theories measures and methods, and interventions, products, devices, etc.

Intermediate – adoption and use of short-term outcomes to change/improve policy, practice & system capacity

Long-term – changes/improvements in overall conditions of the system or population (e.g., increased access and improved rates of employment)

Components of NIDRR’s Logic Components of NIDRR’s Logic Model for Research Outcomes Model for Research Outcomes (See Handout)(See Handout)

Together, these outcomes:

• Reflect a hierarchy or sequence of anticipated results in order of program implementation;

• Depict the program’s change strategy; and

• Help to identify areas of possible implementation failure.

Beneficiaries Intermediate Beneficiaries or Initial Users Intended Beneficiaries or End-Users

Components of NIDRR’s Logic Components of NIDRR’s Logic Model for Research Outcomes Model for Research Outcomes (Continued)(Continued)

Still a work in Progress. Some of the gaps include: No specific KT & CB outcomes are identified; No depiction of the pathways by which outputs of

NIDRR’s investments in R&D become outcomes and what is the role of KT is this process;

Similarly, no depiction of the technical and infrastructural developments or “milestones” necessary to support advances in knowledge and changes in policy, practice and systems capacity. CHIR is addressing this challenge by identifying “indicators of success” that approximate and anticipate KT outcomes;

Stated another way, no acknowledgement that outcomes evolve over time and do not emerge fully developed and to scale across a target population of beneficiaries or system.

Lessons Learned in Lessons Learned in Implementing NIDRR’s Logic Implementing NIDRR’s Logic ModelModel

Ideas for Possible Future Development: Elaborate CB and KT outcome arenas; Chart pathways by which KT activities and outputs lead to

KT outcomes and facilitate R&D outcomes; and Identify “indicators of success” that can be used to mark

progress towards KT outcomes.

Caveat: For the above developments to occur, we need more input

from stakeholders and more data on the performance of the NIDRR’s new KT centers.

So what can we do in the meantime?

Possible Future Development of Possible Future Development of NIDRR Logic ModelNIDRR Logic Model

We can assume that: KT outcomes do not differ theoretically from R&D

outcomes in their type or sequence or in the intended target populations of users and beneficiaries.

The only differences are in the specific knowledge areas being advanced and to some degree in the methods used to achieve outcomes.

We can use the tools of logic modeling, along with the ongoing refinements in KT NIDRR is making under Pimjai Sudsawad’s leadership, and the experience of other federal and non-federal agencies to suggest possible theoretical KT outcomes that could be tracked through NIDRR’s existing evaluation system to determine their frequency and appropriateness.

Proposed Interim StrategyProposed Interim Strategy

CIHR’s initial definition and operationalization of KT and results of recent initiatives described early in this workshop.

On the U.S. side, results of the T.R.I.P. initiative (Translating Research into Practice) sponsored by the Agency for Healthcare Research and Quality (AHRQ) that suggest to accelerate research into policy and practice research users need three types of KT outcomes: Research results about what works and what doesn’t, Knowledge about how others have implemented findings

within their organizations, and Knowledge about how the culture and values of an

organization affect the uptake and continues use of the findings.

Building Blocks for Identifying Building Blocks for Identifying Possible KT OutcomesPossible KT Outcomes

Possible Short-Term Outcomes: Increased understanding and knowledge of the KT

concept and methods, as reflected in results of KT training and use and/or adoption of KT outputs;

Rigorous research results about which KT strategies and methods work and which do not and in which settings and for which target populations. This would be reflected in the increased availability of “best practices.”

Increased knowledge about how the culture and values of research users, including consumers, and organizations affect the uptake of R&D findings.

Possible KT Outcomes within Possible KT Outcomes within the NIDRR Context – the NIDRR Context – For For Discussion OnlyDiscussion Only

Possible Intermediate Outcomes: Change in R&D policy as reflected in increased funding of

KT initiatives by federal and non-federal organizations; Change in R&D practice as reflected in increased

utilization of KT strategies and methods by practitioners; Increased capacity to support the exchange, synthesis and

ethically sound application of R&D findings to change policy, practice, as reflected in infrastructural developments at a systems level.

Possible KT Outcomes within Possible KT Outcomes within the NIDRR Context – the NIDRR Context – For For Discussion Only (Continued)Discussion Only (Continued)

Possible Long-term Outcome: Reduced time intervals between the generation of R&D

findings and the ethically sound application of findings to eliminate disparities between people with disabilities and the general population in employment, health & function, and participation and community living.

Possible KT Outcomes within Possible KT Outcomes within the NIDRR Context – the NIDRR Context – For For Discussion Only (Continued)Discussion Only (Continued)

Current Status: More emphasis on outcomes monitoring versus outcomes measurement and evaluation Outcomes monitoring constitutes an ongoing system of

reporting and tracking program operations and results typically through the use of administrative information systems and performance indicators, and is used to answer questions that ask for a description of “what’s happening.”

Outcome evaluation extends beyond monitoring to an examination of the extent to which outcomes are caused by the program. Outcomes evaluation requires stronger evidence and seeks to answer questions like “What difference does the program make and what about the program is responsible for the difference”?

Part 2:Part 2: Describe NIDRR’s Describe NIDRR’s Evaluation FrameworkEvaluation Framework

Annual Performance Reporting (APR) form -- primary data warehouse for NIDRR’s outcomes monitoring and evaluation system is the.

Performance Measures – out of 7 programmatic GPRA measures 2 pertain directly to KT.

Annual Performance Assessment Expert Review (APAER) process Phase 1 – external quality review of individual grantee

accomplishments documented in APR and related to one of major life domain of NIDRR Logic Model; and

Phase 2 – expert panel review of aggregate accomplishments to assess progress towards NIDRR long-term goals.

Components NIDRR’s Components NIDRR’s Performance Monitoring & Performance Monitoring & Evaluation SystemEvaluation System

To assess the effective use of NIDRR-funded scientific-based knowledge, technologies, and applications, the agency’s collects data from the APR to determine: The average number of publications per award based on

NIDRR-funded research and development activities in refereed journals.  • Results: For calendar years 2004 to 2006, the average

number of peer-reviewed publication per award for the five eligible program categories is 2.71, 3.81, and 2.71, respectively.  

The number of new or improved NIDRR-funded assistive and universally designed technologies, products, and devices transferred to industry for potential commercialization.  • Results for FY 2006 & 2007 available 12-31-07.

How NIDRR Currently How NIDRR Currently Measures KT Performance: Measures KT Performance: GPRA MeasuresGPRA Measures

Purpose: To satisfy GPRA R&D measure #2.1, which calls for "the number of accomplishments developed and/or tested with NIDRR funding that have been judged by expert panelists to be of high-quality and to advance the field," the agency contracts with an outside organization to provide external reviews of individual grantee accomplishments associated a major life domain under the Long-Range Plan.

Focus of 2007 APAER: Participation and Community Living (P/CL) Domain

Data: 84 individual accomplishments (76 outputs and 8 outcomes) documented in the APR from 37 awards were considered eligible for external review based on completeness of information.

How NIDRRHow NIDRR Assesses Quality of Assesses Quality of Program Outputs & Outcomes -- Program Outputs & Outcomes -- Phase 1 of APAERPhase 1 of APAER

Methods: Reviewers were asked to rate the quality and contribution

to the field of each accomplishment using a 5-point scale for outputs and a 3-point scale for outcomes, with each scale ranging from no contribution to outstanding contribution.

Ratings were collected from 12 reviews. To establish a basis for estimating rater effects, 10 accomplishments were rated by all reviewers with the remaining 76 accomplishments overlapping across reviewers and rated by three reviewers. A total of 272 ratings were available for outputs and 34 for outcomes.

How NIDRR Assesses Quality of How NIDRR Assesses Quality of Program Outputs and Outcomes: Program Outputs and Outcomes: Phase 1 of APAER Phase 1 of APAER (Continued)(Continued)

Data Analysis: Used Many-faceted Rasch Measurement (MFRM) statistical technique to control for rater effects in arriving at a composite rating for each accomplishment.

Using this approach, it is possible: To determine if there are significant differences in the

ratings across accomplishments and reviewers and then to identify which accomplishments are more highly and which are more lowly rated and which reviewers are more severe and which are more lenient in their ratings;

Adjust the ratings based on reviewer severity and produce a composite rating that is reported in terms of logits (or log-odds probabilities), which represent equal-interval scales that that allow for analysis using parametric statistical techniques.

How NIDRR Assesses Quality of How NIDRR Assesses Quality of Program Outputs and Outcomes: Program Outputs and Outcomes: Phase 1 of APAER Phase 1 of APAER (Continued)(Continued)

Quantitative Results: Of the 84 accomplishments rated: 49 (or 58%) were judged to be of high-quality and advancing

the field based on adjusted composite rating, which is equivalent to a rating of substantial or higher on the observed scale.

Of these, only 6 were determined to be outcomes, which may be due primarily to the way performance data were collected in the APR with an emphasis on publications, tools and measures; technology products, and informational products.

Significance: First time NIDRR has had reliable data to satisfy GPRA performance measure 2.1, and to use to establish a baseline against which we can demonstrate progress towards long-term goals, consistent with PART requirements.

How NIDRR Assesses Quality of How NIDRR Assesses Quality of Program Outputs and Outcomes: Program Outputs and Outcomes: Phase 1 of APAER Phase 1 of APAER (Continued)(Continued)

Preliminary Impressions of Qualitative Data: Of the 43 outputs and 6 outcomes judged to be advancing the field: The vast majority were the result of R&D activities, with

few direct links to KT activities. Again, this may be due to the way performance data are collected in the APR;

None of the 6 outcomes and only 1 of the 43 outputs appears to correspond to the types of suggested KT outcomes presented above;

However, more than half of the outputs describe findings that appeared to be “ready” for the application of KT strategies to accelerate use or uptake based on strength of presented;

And, several of the accomplishments described utilizing KT strategies in the generation of the output – e.g., use of networks or partnerships among researchers, developers, and industry representatives.

Part 3:Part 3: Examples “Noteworthy” Examples “Noteworthy” Accomplishments: Accomplishments: Phase 1 Phase 1 APAERAPAER

Program Mechanism: RRTC NIDRR LRP Domain: Employment Accomplishment Type: Output, Peer-reviewed Lit Review Objective and Key Findings:

Summarizes research knowledge from fields of disability, economics, health care, and labor force research, to describe the nature of barriers to paid work and economic security for people with mental disorders; and

Describes how findings can be applied to systems change activities that can be pursued at local, state, and federal levels.

Also summarizes NIDRR-funded research that was used by the President’s New Freedom Commission on Mental Health.

Part 3:Part 3: Examples “Noteworthy” Examples “Noteworthy” Accomplishments – Ready for Accomplishments – Ready for Application of KT?Application of KT? # 1# 1

Program Mechanism: RRTC NIDRR LRP Domain: Participation and Community Living Accomplishment Type: Output, Peer-reviewed Pub Objective and Key Findings:

Uses most recent available data (2002) to present three per participant expenditure comparisons between Medicaid HCBS waivers (which require that participants have an institutional level of care need) and institutional care: (1) program expenditures; (2) total Medicaid expenditures, and (3) estimated total public expenditures.

Analysis estimates that when compared with Medicaid institutional care in 2002, HCBS waivers produced a national average public expenditure saving of $43,947 per participant.

Part 3:Part 3: Examples “Noteworthy” Examples “Noteworthy” Accomplishments – Accomplishments – Ready for Ready for Application of KT?Application of KT? #2 #2

Program Mechanism: RRTC NIDRR LRP Domain: Participation and Community Living Accomplishment Type: Output, Peer-reviewed Pub Potential Policy Implications:

As long-term care policy makers struggle with competing challenges including state budget deficits and pressures to expend home and community based services (HCBS), there is a pressing need for information on the comparative cost of Medicaid HCBS and institutional care.

Part 3:Part 3: Examples “Noteworthy” Examples “Noteworthy” Accomplishments – Accomplishments – Ready for Ready for Application of KT?Application of KT? #2 #2 (Continued)(Continued)

Program Mechanism: RRTC NIDRR LRP Domain: Participation and Community Living Accomplishment Type: Output, Community of Practice Objective & Desired Outcomes:

Developed Early Childhood Family Support CoP to foster wisdom-based action, including integration of best available research, for the ultimate benefit of families. Outcomes envisioned:

• For families and practitioners that they become informed and inspired to implement wise solutions.

• For researchers that they will be guided by families and practitioners as they design future research.

Part 3:Part 3: Examples “Noteworthy” Examples “Noteworthy” Accomplishments – Already Accomplishments – Already Applying KT StrategiesApplying KT Strategies #3 #3

Program Mechanism: RRTC NIDRR LRP Domain: Participation and Community Living Accomplishment Type: Output, Community of Practice Objective & Desired Outcomes (continued):

• For policy leaders that they work in with families, professionals, and researchers to translate best available research and family wisdom into meaningful policy.

KT Strategies: Received RUSH grant to: Assess the effectiveness of CoP implementation; Measure gains in knowledge utilization based on random

sample of CoP members; and Utilize mixed-method design to document changes in

behavior of stakeholder groups related to using CoP.

Part 3:Part 3: Examples “Noteworthy” Examples “Noteworthy” Accomplishments – Already Accomplishments – Already Applying KT StrategiesApplying KT Strategies #3 #3

NIDRR has only just embarked on the road to measurement of KT outcomes.

Much work needs to be done to get agreement on KT outcomes; chart pathways of KT outcomes; formulate KT outcome goals, and identify indicators of success.

However, significant process has been made in: Improving performance reporting and data collection

systems; Establishing a performance monitoring system to track KT

accomplishments; and Launching a performance assessment expert review

process to rate individual grant accomplishments for contribution to advancing the field and to assess the overall progress towards NIDRR’s long-term goals.

Conclusions & Next Steps:Conclusions & Next Steps:

Thank you