Knowledge Transfer Discussion Paper - University of...

1

TWO TOOLS FOR WORKING TOGETHER EFFECTIVELY: A KNOWLEDGE TRANSFER DISCUSSION PAPER:

By

Lynne MacLean, Anita Kothari, Nancy Edwards

Acknowledgement: The work reported here was supported by the Ontario Ministry of Health and Long Term Care through a System-Linked Research Grant to the Community Health

Research Unit. The opinions expressed here are those of the authors. Publication does not imply any endorsement of these views by either of the participating partners of the CHRU, nor by the

Ontario Ministry of Health and Long Term Care.

2

OVERVIEW

TWO TOOLS FOR WORKING TOGETHER EFFECTIVELY: A KNOWLEDGE TRANSFER DISCUSSION PAPER

The Community Health Research Unit, in partnership with the Research Unit, Corporate Policy Branch, Ontario Ministry of Health and Long-Term Care, has developed two tools to help researchers and decision-makers work together effectively in research partnerships. These are: Tool 1: Research Transfer: An Evaluation Workbook. The first tool: “Research Transfer: An Evaluation Workbook” (Kothari, MacLean, & Edwards, 2003) is a tool to be used by decision-maker partners to evaluate key areas of the partnership. The sections of the discussion paper devoted to this tool are composed of a technical report giving the development of the Workbook over three successive studies, and the Workbook itself. Tool 2: Formal Relationship Guidelines For Research Transfer Partnerships. The second tool: “Formal Relationship Guidelines For Research Transfer Partnerships” (MacLean & Edwards, 2005) provides some key concepts about such formal research transfer partnership relationships. These are followed by a brief set of guidelines for researchers and decision-makers considering entering into such partnerships with funding from the Ontario Ministry of Health and Long-Term Care (MOHLTC), but which should have implications for such relationships between researchers and decision-makers from other bodies. The impetus for this tool came from the need to more clearly specify some of the linkage activities developed in the Workbook tool. The two tools and the projects they developed from are first described separately. The implications for their implementation are found in a common conclusion section.

PROJECT 1: TOOL 1, THE RESEARCH TRANSFER EVALUATION WORKBOOK

Purpose of Project This project was a partial requirement of the 2002/03-contribution agreement between the Research Unit, Corporate Policy Branch of the Ministry of Health and Long Term Care (Manager: Catia Creatura-Amelio) and the Community Health Research Unit at the University of Ottawa (Director: Nancy Edwards). The purpose of the project was to develop a tool to assist the Research Unit in their evaluation of the research transfer activities between Ministry-sponsored external research projects and associated Ministry partners. As the project progressed, it was suggested that the tool be flexible enough to assess research transfer associated with an entire research project (such as a collaborative research process), or, to a lesser extent, to assess specific research transfer activities (such as a workshop). The tool took the form of an Evaluation Workbook. An Executive Summary and a Technical Report were added to capture recommendations, background information and methodological details.

3

Background In 2000-2001, a study was conducted to examine research receptor capacity and research utilization needs within the Ontario Ministry of Health and Long Term Care (MOHLTC). The research receptor capacity study focused on the abilities of Ministry staff to find, understand, and use evidence-based research in policy development processes. The study was conducted using the blended methods of a cross-sectional survey followed by qualitative interviews. This study resulted in specific, detailed recommendations to improve access to research information, enhance use of the information once accessed, and promote an organizational culture supportive of research utilization. The study was a joint initiative of the Community Health Research Unit (CHRU), University of Ottawa and the Research Unit, Corporate Policy Branch, Integrated Policy and Planning Division of the MOHLTC. Methods As per study recommendations, the next phase involved developing indicators that reflected possible linkage mechanisms between Health System Linked Research Units and MOHLTC partners. A tentative set of indicators was developed based on the research transfer literature. The set was expanded and refined using results from telephone interviews with key informants associated with Health System Linked Research Units and the MOHLTC. A two-stage validation process ensued. First, the evaluation workbook (with indicators) was presented for discussion at a focus group; participants were invited from the MOHLTC and Health System Linked Research Units. Results from the focus group were reported in a summary document sent to the Research Unit in January, 2003, and again, on February 26, 2003. Format and content suggestions were incorporated into the revised eva luation workbook. Procedural suggestions from the focus group are incorporated into the Implementation Recommendations reported here. The revised evaluation workbook was then piloted with six projects associated with the 2001 MOHLTC Making Research Relevant competition. Pilot Results: Suggestions for change were based on the researchers’ experience in using the tool, as well as feedback from pilot participants. Modifications to the tool based on the pilot were minor. These included the addition of a space to put in estimated frequencies of relevant indicators on each page of the workbook, and correction of formatting and typographical errors. The key findings from the application of the evaluation workbook were in the areas of procedures for use, such as consultation to get more than one perspective, data which would be accessible and useful, and when to collect this data. These procedural suggestions are made in the Implementation Recommendations section at the end of the discussion paper. Although there was a lack of information in the project grant files, it was felt that obtaining all necessary supporting data from participants would be too onerous for the participants. Rather, participants were asked to comment on their projects in relation to the evaluation questions, as well as describe whether data to support their views was obtainable, and how accessible such data would be. Further, the RTA was asked to provide information on each project to supplement what was in the file. Sometimes, combining these information sources made it possible to make conclusions with sufficient basis for using the rating scale, sometimes it

4

did not. In view of this, evaluation data used in the evaluation report did not include use of the ratings. In the pilot process, each file took approximately an hour to read and rate, with follow-up interviews consuming approximately one hour more. If the suggested procedures are used, the time to fill in the tool will likely be longer: there will be more documentation in the file, and there will still be the need to discuss the more subjective indicators with at least one partner. This may be balanced somewhat by the knowledge of the project held by the RTA, which the researchers here would not have. It may be useful to apply the tool to a file from another project where there is more detailed documentation, particularly of communication.

Description of the Evaluation Workbook and the Technical Report The Technical Report reviews the research transfer literature that supported the development of the indicators. Methodological details about the interviews with key informants, the focus group and the pilot testing are included as well. The Evaluation Workbook consists of instructions, evaluation questions (related to research transfer), and indicators to use in assessing the evaluation questions. Also, an overall rating of each evaluation question, and any other comments, can be recorded. It is expected that the evaluator (i.e., the Research Unit), will select some appropriate indicators in accordance with the activity or project under review.

PROJECT 2: TOOL 2

FORMAL RELATIONSHIPS GUIDELINES FOR RESEARCH TRANSFER PARTNERSHIPS

Purpose of Project This project was a partial requirement of the 2004/05-contribution agreement between the Research Unit, Corporate Policy Branch of the Ministry of Health and Long Term Care and the Community Health Research Unit at the University of Ottawa. Researchers at The Community Health Research Unit (CHRU) at the University of Ottawa in collaboration with the Ontario Ministry of Health and Long-Term Care (MOH-LTC), Corporate Policy Branch conducted research to examine what makes successful research transfer partnership relationships in health care most effective; to understand the benefits of these relationships to the research itself and to the partners; and to provide guidelines to decision makers and researchers on factors leading to a positive and effective partnership experience. A research transfer partnership in health care is defined as a partnership between academic researchers and decision-makers in health care agencies, government, professional associations or non-governmental organizations. Methods The research team interviewed 8 members of successful ministry and university-based research partnerships. The MOH-LTC, Corporate Policy Branch provided this sample of researchers and decision maker partners with a history of successful partnerships.

5

The results of the interviews were used, along with the literature, to develop a guideline documents for Ministry personnel or other decision makers and researchers on ways to set up and operate a successful research transfer relationship and thus help build and disseminate practical knowledge about successful research transfer partnerships. Description of the Technical Report and the Formal Relationship Guidelines The Technical Report supplies the background and methods used in the development of the Formal Relationships Guidelines, followed by a presentation of the Guidelines themselves.

CONCLUSIONS: TOOL 1 & TOOL 2 A final set of implications for the tools ends the discussion paper.

6

Part 1: Tool 1

RESEARCH TRANSFER: AN EVALUATION WORKBOOK

A. Technical Report B. The Evaluation Workbook

7

Project 1:

RESEARCH TRANSFER EVALUATION WORKBOOK

TECHNICAL REPORT PURPOSE OF REPORT: To provide further information relevant to the Research Transfer Evaluation Workbook in the areas of:

• Background • Literature • Methods • Detailed Evaluation Framework • References

INTRODUCTION Research Transfer refers to the dissemination and uptake of research findings. Traditionally, researchers assumed the role of disseminator while practitioners or policymakers were expected to automatically incorporate research findings into their professional activities. There is, however, a recent interest in identifying more active strategies to enhance the dissemination and uptake process. In particular, increased interactions (“linkages and exchanges”) between researchers and policymakers during the research process are seen as a promising strategy. Both parties have valuable knowledge to bring to the research. Researchers can ensure that the research question is oriented appropriately vis a vis previous research in the area, that the process is methodologically rigorous, and that appropriate conclusions are made given the findings. Policymakers, on the other hand, can bring policy-relevant research questions to the table, and can help interpret the conclusions into recommendations that are contextually relevant and feasible. The short-term outcomes resulting from a research transfer strategy are that the research findings are read, understood and perhaps considered in a policy discussion by decision-makers. Longer-term outcomes refer to the actual utilization of research findings in health policies and programs. Other outcomes also occur as a result of partnerships, such as increased awareness of the other partner’s time lines or constraints. Determining how well research transfer strategies achieve such outcomes requires systematic assessment. The intent of the Research Transfer Evaluation Workbook is to determine the effectiveness of research transfer between Ministry-funded research projects and government partners. The interest lies in identifying the range of research transfer activities in use, and assessing their effectiveness. This technical report provides information on the development of the workbook, and the evaluation framework underpinning it.

8

BACKGROUND The development of the Research Transfer Evaluation Workbook was a partial requirement of the 2002/03-contribution agreement between the Research Unit, Corporate Policy Branch of the Ministry of Health and Long Term Care (Manager: Catia Creatura-Amelio) and the Community Health Research Unit at the University of Ottawa (Director: Nancy Edwards). The purpose of the project was to develop a tool to assist the Research Unit in their evaluation of the research transfer activities between Ministry-sponsored external research projects and associated Ministry partners. As the project progressed, it was suggested that the tool be flexible enough to assess research transfer associated with an entire research project (such as a collaborative research process), or, to a lesser extent, to assess specific research transfer activities (such as a workshop). The tool took the form of an Evaluation Workbook. The need for such a tool was determined as a result of a series of research activities. In 2000-2001, a study was conducted to examine research receptor capacity and research utilization needs within the Ontario Ministry of Health and Long Term Care (MOHLTC) (Tiefenbach & Edwards, 2001). The research receptor capacity study focused on the abilities of Ministry staff to find, understand, and use evidence-based research in policy development processes. The study was conducted using the blended methods of a cross-sectional survey followed by qualitative interviews. This study resulted in specific, detailed recommendations to improve access to research information, enhance use of the information once accessed, and promote an organizational culture supportive of research utilization. The study was a joint initiative of the Community Health Research Unit (CHRU), University of Ottawa and the Research Unit, Corporate Policy Branch, Integrated Policy and Planning Division of the MOHLTC. This study was accompanied by a literature review on research receptor capacity and research utilization (Griffin & Edwards, 2001). The purpose of the literature review was to identify and describe (i.e. “best strategies”) to build research capacity among health policy decision makers and to facilitate the utilization of research in policy decision-making. RELEVANT LITERATURE (literature cited in this section from review by Griffin & Edwards, 2001): As a result of the literature review it became evident that positive linkage between researchers and Ministry decision makers was a key component in enhancing transfer of research findings into government decision making. For example, in a key study of Canadian professionals and managers (Landry et al, 2000), variables found to be significantly and positively related to utilization were: “(i) the acquisition efforts of users (personal efforts to establish relationships with university researchers and sufficient resources to access university research work); (ii) the adaptation of research products by researchers (ease of comprehension, specific conclusions and recommendations, focus on interventions under user control, credible source, relevance, plausibility, ability to evaluate quality of research, exclusive results, and appeal of reports); (iii) linkage mechanisms (meetings with colleagues, conferences/seminars with researchers, electronic mail and the Internet, and reference library); and

9

(iv) the organizational context of the users (pertinence and timeliness of research findings). Based on these results, the investigators suggested that in order to facilitate research utilization by policy decision makers, incentives should be created to: (i) foster linkage mechanisms between decision makers and researchers, (ii) promote the adaptation of research products for decision makers by researchers, (iii) make the decision makers’ context more obvious to researchers, (iv) support the acquisition efforts of decision makers, and (v) develop measures targeting the receptive capacity of decision makers. Paluck et al (2001) again found an important role for linkage activities in increasing research transfer. In a study looking at use of public health research in Regional Health Authorities across Canada, some key strategies to increase use of research were identified that highlighted the role of linkage: Allocating resources to create linkages with academic centres, improving communication with the authorities by establishing on-going relationships, identifying their research needs, learning about the decision-making process, and involving policy decision makers in research projects to improve their relevance and accessibility. Adapting the research findings by tailoring the content and improving readability of reports was suggested, as was using multiple strategy dissemination activities such as workshops, symposia, conferences, fact sheets, newsletter articles, project reports and executive summaries. Similar findings occurred in a study examining use of research in policy decisions (Lavis et al., 2000). Overall, investigators using various methods and participant groups found that the use of health services research and other types of information in the policy making process were most influenced by the interaction between policy decision makers and researchers as well as the existence of a designated “receptor” for research within government (Landry et al., 1998; Landry et al., 2000; Paluck et al., 2001; Lavis et al., 2000). For policy decision makers, this suggests that to increase the use of research evidence, researchers should be more directly involved in the policy development process. It also points to the facilitation role tha t might be played by “receptors” whose function in this study was performed either by an ‘expert’ working group with researcher members or by the establishment of a government position to improve knowledge of specific health issues and community concerns. Barriers to research utilization common to both researchers and decision makers which prohibit successful communication include (Caplan, 1979; Rich, 1979; Frenk, 1992; Webber, 1987; Oh and Rich, 1996): time constraints, timelines, identification of appropriate partners and processes, cultural differences between research and ministry in degree of openness in release of research findings, awareness of each others’ needs/interests, and differing expectations and evaluations of the use for research (Griffin & Edwards, 2001). Clearly, communication between researchers and decision makers is important for optimal linkage and research transfer. Determination of appropriate and feasible indicators of linkage and communication was an area with little development. This was a necessary gap to fill in order to develop a more complete evaluation framework for research transfer. METHODS

10

Literature Review: As per study recommendations, the next phase involved developing indicators that reflected possible linkage mechanisms between Health System Linked Research Units and MOHLTC partners. Combined with the information on research receptor capacity and research utilization needs from the literature (Griffin and Edwards, 2001), the findings of the previous study (Edwards et al., 2001), Health System-Linked Research Unit 10 year outcome reports, and other documents, a tentative set of indicators were developed. The Health System-Linked Research Unit program was selected for these initial steps in indicator development. This program is one in which participants often had much experience working with Ministry partners developing research directly intended for transfer into ministry decision making. Health System-Linked Research Units are 8 research units funded by the MOHLTC, linked with various partners from different parts of the health system. Their goals involve enhancing health services through provision of research to enhance evidence-based practice and policy. Interviews: Qualitative, semi-structured telephone interviews with HSLRU directors, or their designates, were conducted. Participants were asked their views on measuring the linkage process in general, including key outcomes and strategies in linking with and transferring research to the Ministry. The MOHLTC representatives most involved in the work of the HSLRUs were also interviewed on these topics. In all, 16 participants (8 Ministry and 8 HSLRU) from 8 partnerships provided information. Framework: The indicator information from the interviews was then combined with the indicator information from the other sources above. Indicators for the framework were not solely focused on linkage, but included others appearing to be key to research transfer (e.g., deliverable formats) as found in the literature (Caplan, 1979; Frenk, 1992; Griffin & Edwards, 2001; Landry et al., 1998; Landry et al., 2000; Lavis et al., 2000; Oh and Rich, 1996; Paluck et al., 2001; Rich, 1979; Tiefenbach & Edwards, 2001; Webber, 1987.) The indicator set was expanded and refined to result in an evaluation framework. The framework served as the basis for a quick evaluation tool that would allow ministry co-investigators to collect information across researchers and research programs about effective research transfer programming. Format of the tool was based on several iterations of feedback with the Ministry members of the development team in order to meet user needs.

Framework and Workbook Validation: A two-stage validation process then ensued. 1.Focus Group: First, the evaluation framework and evaluation tool (now called the workbook) were presented for discussion at a focus group; participants were invited from the MOHLTC and Health System Linked Research Units. Six participants were from HSLRUs and one was a Ministry partner. Format and content suggestions were incorporated into the revised evaluation workbook. Procedural suggestions from the focus group are incorporated into the Implementation Recommendations reported in the Executive Summary document. The revised evaluation workbook was then piloted with six projects associated with the 2001 MOHLTC Making Research Relevant competition. A research program other than that of the HSLRUs was selected to establish generalizability to different types of research contracts. 2. Pilot: As the evaluation workbook was intended for use by the Research Transfer Advisor (RTA), the pilot procedure was meant to mimic such use as closely as possible. The information in the

11

project grant files of the six selected MRR projects was applied to the workbook. Then, the RTA was asked to provide any further information he might have about each file that was not in the documentation. As a final step, the key researchers involved were also contacted and asked to comment on the indicators, and on the availability and accessibility of documentation that would support their impressions. While Ministry partners had been invited to take part in this process as well, no signed consent forms were received in time for such participation. Based on the pilot, the evalua tion workbook was further refined. THE EVALUATION FRAMEWORK While the completed Research Transfer Evaluation Workbook is attached separately, it does not include the complete evaluation framework. The Evaluation Framework follows here, and indicates (by *) which indicators were developed as a result of data gathered from participants solely as a result of this project. It also makes more detailed suggestions as to what kind of data may be needed to answer key research transfer evaluation questions. Indicators that seemed to apply to other evaluation activities conducted by the Ministry were not used as a part of the framework or evaluation tool. For example, identification of the relevance of the research question, or of the degree to which the project met its agreed upon objectives were considered to be part of the standard evaluation of the specific contract deliverables. What is unique about the evaluation framework is that it recognizes that research transfer is dependent on the maturity of the researcher/decision maker partnership. “Early” partnerships, or younger relationships will demonstrate different research transfer outcomes than will “Mature”, or older, more developed partnerships. The Core evaluation questions apply to all partnerships, regardless of stage of development. A further unique contribution of this framework is the degree of detail regarding linkage indicators. These should help evaluators to specify these processes which are often difficult to pin down.

12

THE EVALUATION FRAMEWORK Table 1. Core Evaluation Questions (note: * = indicators derived solely from project data)

A. CORE RESEARCH TRANSFER QUESTIONS

EXAMPLES OF INDICATORS OF SUCCESS EXAMPLES

OF DATA REQUIRED

POTENTIAL DATA

SOURCES

A.1. Is the communication process between partners effective (i.e., among the research triangle: researchers, Ministry partners and primary community stakeholder)

1. Communication is clear. • Clarification is on-going. • Communication involves face-to-face meetings as well as telephone, mail, email, and fax methods. • The same contact people continue over the life of the project. • A common language/lexicon is used by both parties.

2. Communication is relevant. • *Roles, expectations, and criteria for deliverables are explicit.

3. Communication is timely. • Communication is frequent.

4. *Communication is respectful. • *Partners value each other’s contributions. • *Partners are acknowledged in project documents.

Frequency, type, and purpose of contact. Information on who has been involved, how, when, and in what capacity. Degree of satisfaction with communication.

Agenda and minutes of meetings. Project documents.

A.2. Is the research developed collaboratively?

1. Joint meetings occur at most stages of research. • Joint identification of research questions. • Each partner’s needs and constraints expressed. • Joint designing of research. • Joint data collection. • Joint data analysis. • Joint discussion of findings and implications.

2. Joint meetings occur to discuss research dissemination and utilization plans.

• *Feedback about research report is provided before final draft. • *Response to feedback is prompt. • *Only a few rounds of revisions before deliverable is acceptable to all.

3. Feedback is given after the final deliverable is received.

Frequency, type, and purpose of meetings. Tracking of feedback and draft versions of research report.

Written dissemination and utilization plan.

Agenda and minutes of meetings.

Project documents.

A.3. Were research findings disseminated effectively to Ministry partners and the primary community stakeholder?

1. Multiple formats of written and/or other forms of presentation (e.g., newsletter, website summary, interim report, oral presentation

• Stakeholders and Ministry partners received relevant documents.

2. Presentation formats in layman’s terms 3. Presentation formats include recommendations for action 4. Where appropriate, presentation formats are concise (e.g., less than two pages). 5. *Stakeholders contacted researcher or government partner to discuss the research findings.

13

Table 2. Early Evaluation Questions (note: * = indicators derived solely from project data) B. EARLY

RESEARCH TRANSFER QUESTIONS

EXAMPLES OF INDICATORS OF SUCCESS

EXAMPLES OF DATA REQUIRED

POTENTIAL DATA

SOURCES

B.1. Do partners negotiate clear project parameters (i.e., use of principles of project management)?

1. Negotiation occurs at various stages of the research process.

• Roles and responsibilities are documented. • Written terms of reference for research project (or similar document).

2.Negotiated items are clearly understood by all.

• Requirements for deliverables and time lines are documented. • *Partners make their needs explicit (i.e., in terms of accountabilities, priorities, and long term interest). • Partners document the above needs.

Documentation of meeting content.

Partner perceptions of conflict resolution process and outcomes.

Documentation of timelines and deliverables.

Documentation of priorities.

See the Research Grants Administration Guidelines and other documentation for data requirement and source ideas.

Agenda and meeting minutes.

Project documents.

B. 2. Are efforts made to enhance the partnership (among the research triangle)?

1. Clear leadership with respect to partnership management.

• Key players and senior management are visibly involved.

2. Development of team mentality. • Discussion of potential long-term plans/structure to ensure continuity of relationship.

3. *Early engagement of people. • *Staff with previous linkages with each other are incorporated into partnership

4. Exposure to both team structures. 5. *Partners have a previous, positive relationship, when possible.

Written terms of reference for research project (or similar document).

Partner perceptions of resources allocated to partnership.

Agenda and meeting minutes.

B.3 Are research findings considered during the development of Ministry partner’s programs and policies?

1. *Research findings are discussed in policy deliberations.

• Research findings are presented in policy-related format and language • Implications of findings are understood by all. • Documentation of feedback to researchers.

Government-reported instances of consideration.

Good feedback to researchers.

Government partner survey.

Unofficial communications.

14

Table 3. Mature Evaluation Questions (note: * = indicators derived solely from project data) C. MATURE RESEARCH TRANSFER QUESTIONS

EXAMPLES OF INDICATORS OF SUCCESS EXAMPLES OF DATA REQUIRED

POTENTIAL DATA

SOURCES

C.1. Are each partner’s research/policy information needs being met?

1. *Partners are flexible about meeting partner’s changing needs, and revising research plans and time lines.

• Project timelines and changes have been tracked through documentation. • Roles and responsibilities have been defined up front. • Research purpose and objectives have been defined, documented, and referred to in an on-going fashion as the research progresses.

2. *Partners understand the limits of each other’s flexibility. 3. Partners understand research findings, their limits, and their implications for M inistry work.

Perceptions of the extent to which needs are being compromised.

Documentation of timelines, roles, responsibilities, purposes, and objectives.

Evidence of partners referencing the purpose and objectives, and expressing ongoing satisfaction with them or their changes.

Meeting minutes.

Other project documents.

C.2. Is there increasing rapport between partners?

1.Conflict is dealt with openly, informally, and promptly. • *More informal communication occurs, though formal meetings and communication continue.

2. *Trust has increased between partners. • *Appreciation is shown of each other’s efforts.

3. *Comfort has increased between partners. • *Partners support each other publicly.

4. *Openness has increased between partners. • *Partners provide advance notice of surprising or dismaying aspects of research findings or government decisions.

5. *Partners begin speaking a common language regarding research. 6. *Partners facilitate removal of barriers for each other’s work. Partners understand:

• -*how things are communicated within the partner organization; • -*how senior level people work and what their concerns are; • -agendas, priorities, expectations and limits; • -dissemination opportunities within the partner organization; • -opportunities for research use and impact within the partner organization; • -costs of monitoring, influencing, and incorporating research into decision-making.

7. *Linkage with partner enhances partner linkage with community/other stakeholders.

• *Linkage with partner does not detract from previously established linkages with other partners.

Frequency, type, and purpose of contact. Information on who has been involved, how, when, and in what capacity. Degree of satisfaction with rapport between partners.

Agenda and minutes of meetings. Project documents.

15

C.3. Do partners demonstrate increasing commitment to the research project?

1. *There is joint commitment to the research project. • The partners contribute more resources, material and otherwise, to the research project. • *Partners willingly provide “extras”, such as extra time or staff, to the project.

2. *There is an increase in joint activity around the project. 3. *There is an increase in involvement as co-investigators, with ministry partners becoming more involved in all aspects of the research.

• More people from each partner organization have a formal specific knowledge of the partner organization and its mandate.

Records of resources spent such as staff time, budget allotment, and space. Partner perceptions of commitment. Identification of authorship of project-related reports and articles.

Agendas and meeting minutes. Project documents.

C.4. Do partners demonstrate increasing commitment to the partnership?

1. There is a commitment to pursuing future joint projects.

• *Partners take on new roles with each other.

2. *On-going dialogue moves a research program forward over a series of projects. 3. *Partners are perceived as experts in the research/policy area and are referred to as such to others.

• *Partners introduce each other to new networks. • Partners think of each other in relation to projects, committees, etc., outside of the research project relationship.

4. *An informal or formal infrastructure exists for linking and transferring research between partners.

• The partnership’s work becomes integrated with work associated with other stakeholders.

Documentation of planning for future projects. Records of partners being referred for their expertise or resources to other stakeholders. Records of partners serving on new committees, etc.

Agenda and minute meetings. Project documents.

C.5. Are research findings discussed and/or incorporated into government partner’s programs or policies?

1. Research findings are discussed or are reflected in government meetings/research documents. Relevant discussion or

citation

Unofficial communica-tions or minutes

16

COMMUNITY HEALTH RESEARCH UNIT


A. Kothari, Ph.D. L. MacLean, Ph.D.

N. Edwards, R.N., Ph.D.

Please do not photocopy or distribute without the prior written approval of

Nancy Edwards, Professor, School of Nursing, University of Ottawa

Director, Community Health Research Unit [email protected]

Acknowledgement: The work reported here was supported by the Ontario Ministry of Health and Long Term Care through a System-Linked Research Grant to the Community Health

Research Unit. The opinions expressed here are those of the authors. Publication does not imply any endorsement of these views by either of the participating partners of the CHRU, nor by the

Ontario Ministry of Health and Long Term Care.

17


Contents

1. Introduction 2. Purpose 3. Roles and Responsibilities 4. Information Tracking Summary 5. Workbook Instructions 6. Workbook

…..2 …..2 …..3 …..4 …..6 …..8

18

Introduction

Research Transfer refers to the dissemination and uptake of research findings. Traditionally, researchers assumed the role of disseminator while decision-makers were expected to incorporate research findings into their professional activities. There is a recent interest in identifying more active strategies to enhance research transfer. In particular, increased interactions (“linkages and exchanges”) between researchers and decision-makers during the research process are seen as a promising strategy. Both parties have valuable knowledge to bring to the process of research development and utilization. Some intermediate outcomes related to research transfer are that the research findings are read, understood and perhaps considered in a policy discussion by decision-makers. Other intermediate outcomes also occur as a result of a successful research transfer process, such as increased awareness of each other’s needs from the research. The indicators in this Workbook emphasize these intermediate outcomes. Longer-term outcomes refer to the actual utilization of research findings in developing health policies and designing programs. These outcomes are difficult to measure, and are often determined informally. This Workbook is based on the idea that linkages and exchanges between researchers and decision-makers is a necessary component of program and policy development.

Purpose of the Evaluation Workbook

1. Facilitate research transfer between Ministry-funded academic researchers and their Ministry partners. 2. Identify the range of research transfer activities currently in use. 3. Determine the effectiveness of research transfer activities.

The Workbook can be used to evaluate either a particular research transfer activity (e.g., a workshop, literature review, newsletter) or the set of research transfer activities associated with a research project (the conduct of a piece of research which may include a variety of research transfer activities) or a research program (a series of thematically-related research projects over a longer period of time). The Workbook is sensitive to differences in research transfer efforts and outcomes related to the age of the partnership. Older partnerships, which have had time to mature and establish procedures, will demonstrate different research transfer outcomes than newer ones (e.g., relationships that are less than a year old).

19

Roles and Responsibilities The Research Unit, of the Corporate Policy Branch, Ministry of Health and Long-Term Care, will assume prime responsibility for conducting the evaluation. The Unit will collect data for the evaluation from their grant files, as well as from the Ministry-funded academic researchers and associated Ministry partners. Information about the primary community stakeholder group will generally be gathered indirectly, through the researcher. It is recommended that the Research Unit provide and discuss the evaluation tool in advance with the researchers and Ministry partners. That way both parties can devise systems to collect the required information, and contribute to the final assessment of the research transfer activity. A background technical document briefly describing the research transfer literature, the methodological development of the indicators and the history of this document are also available from the Ministry. This can be obtained from the Research Unit, Corporate Policy Branch, Integrated Policy and Planning Division, Ontario Ministry of Health and Long Term Care. The next section describes the overall process of using the Evaluation Workbook.

20

Information Tracking Summary The following are examples of information elements that might be useful to collect and maintain during the project for subsequent evaluative purposes.

Examples of Potential Data Required Potential Data Sources

• Frequency, type, and purpose of meetings and other contact (e.g., phone calls,

email).

• Information on who has been involved, how, when, and in what capacity.

• Documentation of meeting content.

• Documentation of timelines and deliverables, roles, responsibilities, purposes,

and objectives.

• Documentation of priorities.

• Written terms of reference for research project (or similar document). • Records of resources spent such as staff time, budget allotment, and space.

• Degree of satisfaction with communication, collaboration, resources, etc.

• Partner perceptions of conflict resolution process and outcomes.

• Partner perceptions of resources allocated to partnership.

• Partner perceptions of the extent to which needs are being compromised.

• Evidence of partners referencing the purpose and objectives, and expressing

ongoing satisfaction with them or their changes.

• Degree of satisfaction with rapport between partners.

• Partner perceptions of commitment.

• Tracking of feedback and draft versions of research report.

• Written dissemination and utilization plan.

• Copies of presentation formats.

• Documentation of feedback to researchers.

• Identification of stakeholders who could and could not recall seeing the research

report.

• Tracking of stakeholder inquiries about research findings.

• Government-reported instances of consideration of research findings.

• Relevant discussion or citation of research findings by government.

• Identification of authorship of project-related reports and articles.

• Documentation of planning for future projects.

• Records of partners being referred for their expertise or resources to other

stakeholders.

• Records of partners serving on new committees for each other outside of the

project.

• Agendas and minutes of meetings.

• Project documents (e.g., work plans).

• Unofficial government communications and minutes

• Community stakeholder reports

• Partner interviews

• Dissemination Tools (e.g., web sites, policy documents)

21

Workbook Instructions Protocol For Application of Evaluation Framework Fill in Workbook:

• When a project ends; • After each 12-month period that a program/contract is active; or • When there is a need to evaluate a project at midpoint to check progress, clarify problems, or provide new ideas.

Step 1: Obtain the Required Data from Data Sources Different kinds of data or information are required to determine the degree to which the indicators have been achieved. See the Information Tracking Summary on pg. 4 of the document for data source suggestions (e.g., project records, meeting agendas). Step 2: Select workbook tables, based on the maturity of the partnership There are three sections: Core, Early, and Mature.

• All programs should fill in the sections for “Core” indicators. (You will find that these indicators and sub- indicators are broader than for the Early and Mature sections.) • Programs/Contracts of a year or less should fill in the Early sections as well as the Core sections. • Programs/Contracts that have existed for more than one year should fill in the Mature sections, as well as the Core sections.

Relevant Workbook Tables Duration of Partnership/Contract Core Early Mature 1 year or less X X Over 1 year X X

Step 3: Select Relevant Indicators Select relevant indicators for the project based on

• The maturity of the partnership; and • The availability of complete and accurate data.

There is a menu of several indicators for each evaluation question. Select as many indicators as are useful for this specific evaluation. Check off the indicators selected for each evaluation question, for each of the tables (core plus early or mature).

Step 4: Evaluate the Indicators

• Rate the indicator, based on conclusions reached in synthesizing the data. Determine to what extent each indicator was achieved.

22

• Rate the success of the project in answering each evaluation question, over all indicators. This is done at the end of each indicator question section. This is not meant to be an average of the other ratings, but a more global assessment based on evaluator judgment. • Document the data sources used to assess the indicator, including any estimations of frequency, if applicable. • Add any additional comments about the indicator in the Comments Section.

Step 5: Summarize Key Findings

At the end of the workbook there is a place to summarize key findings from the evaluation. These could include:

• Summary remarks about the program, • Highlights and/or key concerns • Comments about future areas for improvement and steps to get there,

There is a space for each of the evaluation areas. Record the overall rating assigned to each of these areas, as well as any written comments. Please note there is an example of what a completed page might look like, immediately following.

23

Example of Completed Page “C2” C2. Core Research Transfer Question (for early and mature relationships): Is the research developed collaboratively?

Examples of Indicators Degree to which indicator was achieved (indicate number):

1) Not achieved 2) Sometimes 3) Sufficient 4) Most of the ti me 5) Always

Ø 1.0 Joint meetings occur at most stages of research. Ø 1.1 Joint identification of research questions. Ø 1.2 Each partner’s needs and constraints expressed. Ø 1.3 Joint designing of research protocol.

q 1.4 If relevant, joint data collection. q 1.5 If relevant, joint data analysis.

Ø 1.6 Joint ongoing evaluation of relevance of research (e.g. current project, new findings, new partner needs etc.) Ø 1.7 Joint discussion of findings and implications.

Ø 2.0 Joint meetings occur to discuss research dissemination and utilization plans.

q 2.1 Feedback about research report is provided before final draft.

Ø 2.2 Response to feedback is prompt.

q 2.3 Only a few rounds of revisions before deliverable is acceptable to all.

Ø 2.4 Feedback is given after the final deliverable is received.

3. We usually have meetings at each stage 5. Good negotiations here 4. People pretty open about being clear about this 3. Some take on more of a lead than others. 5. Very well done. 5. All parties seem eager and interested 1.Dissemination not discussed 2. Needs improvement 1. Ministry partner needs to do this more.

Overall collaboration effectiveness:

q no collaboration q ineffective q acceptable p good q excellent [The above rating to be transferred over to page 18, section C.2] Estimate of frequency, if applicable: Frequency of contact varied: at start of project , 1 teleconference, 1 face to face with teams. Throughout data collection and analysis phases, teleconference every 2 months. Email updates monthly and as needed. Data sources: Minutes of meetings, letters of agreement, contract, written research plans with goals and objectives, correspondence tracking (emails). Comments: In general, all partners were excellent at collaborative planning. Need to do more development of collaboration around dissemination and utilization, especially if another phase to the research to be contracted. [comment may be transferred to Highlights section of pg.19]. Next steps: contact partners to organize teleconference to discuss feedback, dissemination, and utilization. May want to organize a face to face, if future work on this planned [comment may be transferred to Next Steps section of

24

pg.19].

Research Transfer Workbook Cover Sheet

Project Title: Covering dates (from/to): to MOHLTC Program Area Lead:

Telephone: Email:

Researcher Lead: Research Organization:

Telephone: Email:

Date Evaluation Completed by Research Transfer Office:

Ministry of Health and Long Term Care Corporate Policy Branch, Research Unit

9th Floor, 101 Bloor Street West Toronto, ON M5S 2Z7

Indicate who is completing this worksheet.

25

C1. Core Research Transfer Question (for early and mature relationships): Is the communication process between partners effective (i.e., among the research triangle: researchers, Ministry partners and, where relevant, primary community stakeho lder)?

Examples of Indicators

Degree to which indicator was achieved (indicate number):

1) Not achieved 2) Sometimes 3) Sufficient 4) Most of the time 5) Always

q 1.0 Communication is clear. q 1.1 Communication is on-going.

q 1.2 Communication involves face-to-face meetings as well as telephone, mail, email, and fax methods. q 1.3 The same contact people continue over the life of the project. q 1.4 A common language/lexicon is used by both parties.

q 2.0 Communication is relevant.

q 2.1 Roles, expectations, and criteria for deliverables are explicit.

q 3.0 Communication is timely. q 3.1 Communication is frequent.

q 4.0 Communication is respectful.

q 4.1 Partners value each other’s contributions. q 4.2 Partners are acknowledged in project documents.

Overall communication effectiveness: q poor q below average q acceptable q good q excellent Estimate of frequency, if applicable: Data sources: Comments:

26

C2. Core Research Transfer Question (for early and mature relationships): Is the research developed collaboratively?



q 1.0 Joint meetings occur at most stages of research.

q 1.1 Joint identification of research questions. q 1.2 Each partner’s needs and constraints expressed. q 1.3 Joint designing of research protocol. q 1.4 If relevant, joint data collection. q 1.5 If relevant, joint data analysis. q 1.6 Joint ongoing evaluation of relevance of research (e.g. current project, new findings, new partner needs etc.) q 1.7 Joint discussion of findings and implications.

q 2.0 Joint meetings occur to discuss research dissemination and utilization plans. q 2.1 Feedback about research report is provided before final draft. q 2.2 Response to feedback is prompt. q 2.3 Only a few rounds of revisions before deliverable is acceptable to all. q 2.4 Feedback is given after the final deliverable is received.

Overall collaboration effectiveness:

q no collaboration q ineffective q acceptable q good q excellent Estimate of frequency, if applicable: Data sources: Comments:

27

C3. Core Research Transfer Question (for early and mature relationships): Were research findings disseminated effectively to Ministry partners and, where relevant, the primary community stakeholder(s)?



q 1.0 Multiple formats of written and/or other forms of presentation (e.g., newsletter, website summary, interim report, oral presentation q 1.1 Stakeholders and Ministry partners received relevant documents.

q 2.0 Presentation formats in layman’s terms

q 3.0 Presentation formats include recommendations for action

q 4.0 Where appropriate, presentation formats are concise (e.g., less than two pages).

q 5.0 Stakeholders contacted researcher or government partner to discuss the research findings.

Overall dissemination effectiveness: q no dissemination q ineffective q acceptable q good q excellent Estimate of frequency, if applicable: Data sources: Comments:

28

C4. Core Research Transfer Question: Are research findings considered during the development of Ministry partner’s programs and policies?



q 1.0 Research findings are discussed in policy deliberations.

q 1.1 Research findings are presented in policy-related format and language q 1.2 Implications of findings are understood by all. q 1.3 Documentation of feedback to researchers. q 1.4 Ministry senior staff are aware of research findings. q 1.5 Research findings are discussed or are reflected in government meetings/research documents.

Overall consideration of research findings (where relevant): q no consideration q minimal consideration q acceptable q good q excellent Estimate of frequency, if applicable: Data sources: Comments:

29

E1. Early Research Transfer Question: Do partners negotiate clear project parameters (i.e., use of project management principles)?



q 1.0 Negotiation occurs at various stages of the research process.

q 1.1 Roles and responsibilities are documented. q 1.2 Written terms of reference for research project (or similar document).

q 2.0 Negotiated items are clearly understood by all. q 2.1 Requirements for deliverables and time lines are documented. q 2.2 Partners make their needs explicit (i.e., in terms of accountabilities, priorities, and long term interest).

q 2.3 Partners document the above needs.

Overall negotiation effectiveness: q no negotiation q ineffective q acceptable q good q excellent Estimate of frequency, if applicable: Data sources: Comments:

30

E2. Early Research Transfer Question: Are efforts made to enhance the partnership (among the research triangle)?



q 1.0 Clear leadership with respect to partnership management. q 1.1 Key players and senior management, where relevant, are visibly involved/supportive.

q 2.0 Development of team mentality.

q 2.1 Discussion of potential long-term plans/structure to ensure continuity of relationship.

q 3.0 Early engagement of people.

q 3.1 Staff with previous linkages with each other are incorporated into partnership

q 4.0 Exposure to team/organization structures of research partners.

Overall partnership enhancement: q no efforts q ineffective q acceptable q good q excellent Estimate of frequency, if applicable: Data sources: Comments:

31

M1. Mature Research Transfer Question: Is each partner’s research/policy information needs being met?



q 1.0 Partners are flexible about meeting partner’s changing needs, and revising research plans and time lines. q 1.1 Project timelines and changes have been tracked through documentation. q 1.2 Roles and responsibilities have been defined up front. q 1.3 Research purpose and objectives have been defined, documented, and referred to in an on-going fashion as the research progresses.

q 2.0 Partners understand the limits of each other’s flexibility.

q 3.0 Partners understand research findings, their limits, and their implications for Ministry work.

Overall effectiveness in meeting information needs: q no efforts q ineffective q acceptable q good q excellent Estimate of frequency, if applicable: Data sources: Comments:

32

M2. Mature Research Transfer Question: Is there increasing rapport between partners?

Examples of Indicators Degree to which indicator was achieved

(indicate number): 1) Not achieved 2) Sometimes 3) Sufficient 4) Most of the time 5) Always

q 1.0 Conflict is dealt with openly, informally, and promptly. q 1.1 More informal communication occurs, though formal meetings and communication continue.

q 2.0 Trust has increased between partners. q 2.1 Appreciation is shown of each other’s efforts.

q 3.0 Comfort has increased between partners. q 3.1 Partners support each other publicly.

q 4.0 Openness has increased between partners.

q 4.1 Partners provide advance notice of surprising or potentially contentious research findings or government decisions.

q 5.0 Partners begin speaking a common language regarding research.

q 6.0 Partners facilitate removal of barriers for each other’s work. q 6.1 Partners understand:

- how things are communicated within the partner organization; - how senior level people work and what their concerns are; - agendas, priorities, expectations and limits; - dissemination opportunities within the partner organization; - opportunities for research use and impact within the partner organization;

- costs of monitoring, influencing, and incorporating research into decision-making.

q 7.0 Linkage with partner enhances partner linkage with community/other stakeholders.

q 7.1 Linkage with partner does not detract from previously established linkages with other partners.

Overall increase in level of rapport: q no rapport q ineffective rapport q acceptable q good q excellent Estimate of frequency, if applicable: Data sources: Comments:

33

M3. Mature Research Transfer Question: Do partners demonstrate increasing commitment to the research project/partnership?



q 1.0 There is joint commitment to the research project.

q 1.1 The partners contribute more resources, material and otherwise, to the research project. q 1.2 Partners willingly provide “extras”, such as extra time or staff, to the project.

q 2.0 There is an increase in joint activity around the project. q 2.1 Partners take on new roles with each other q 2.2 On-going dialogue moves a research program forward over a series of projects.

q 3.0 Partners are perceived as experts in the research/ policy area and are referred to as such to others.

q 3.1 Partners introduce each other to new networks. q 3.2 Partners think of each other in relation to projects, committees, etc., outside of the research project relationship.

q 4.0 An informal or formal infrastructure exists for linking and transferring research between partners. q 4.1 The partnership’s work becomes integrated with work associated with other stakeholders.

Overall commitment to the project/partnership: q no commitment q ineffective growth in commitment q acceptable q good

q excellent Estimate of frequency, if applicable: Data sources: Comments:

34

EVALUATION SUMMARY

Section C: Core Research Transfer Elements:

C.1. Overall communication effectiveness: q poor q below average q acceptable q good q excellent C.2. Overall collaboration effectiveness: q no collaboration q ineffective q acceptable q good q excellent C.3. Overall dissemination effectiveness: q no dissemination q ineffective q acceptable q good q excellent C.4. Overall consideration of research findings (where relevant). q no consideration q minimal consideration q acceptable q good q excellent

Section E: Early Research Transfer Elements:

E.1. Overall negotiation effectiveness: q no negotiation q ineffective q acceptable q good q excellent

E.2. Overall partnership enhancement: q no efforts q ineffective q acceptable q good q excellent

Section M: Mature Research Transfer Elements:

M.1. Overall effectiveness in meeting information needs: q no efforts q ineffective q acceptable q good q excellent

M.2. Overall increase in level of rapport: q no rapport q ineffective rapport q acceptable q good q excellent M.3. Overall commitment to the project/partnership: q no commitment q ineffective growth in commitment q acceptable q good q excellent

Comments:

• Highlights/ Key Concerns:

• Next Steps:

• Other:

35

Project 2: Tool 2

FORMAL RELATIONSHIP GUIDELINES FOR RESEARCH TRANSFER PARTNERSHIPS.

A. Technical Report B. The Formal Relationship Guidelines

As mentioned in the abstract, the impetus for this tool came from the need to more clearly specify some of the linkage activities developed in the Workbook tool. As well, there was a need to provide a brief set of guidelines to researchers and decision-makers which would allow them to decide if these types of partnerships would be of interest to them before they entered one: thus an entry tool more so than an evaluation one. Following find the Technical Report outlining the methods used to develop the tool, and then the tool itself: the Formal Relationships Guidelines.

36

Project 2: FORMAL RELATIONSHIPS GUIDELINES PROJECT

TECHNICAL REPORT

PURPOSE The purpose of this research was:

• To examine what makes successful research transfer partnership relationships most effective

• To examine the benefits of these relationships to the research itself and to the partners • To provide guidelines to decision makers and researchers on factors leading to a positive

and effective partnership experience INTRODUCTION How can researchers and decision makers be certain that resources directed towards research transfer partnerships are well invested? As health care expenses increase, decision makers are obliged to invest all available dollars with vigilance. Researchers must balance their initiatives with the realities of health services delivery while considering their understanding of what constitutes successful research. What criteria will ensure that the right partners have been targeted? What elements should be addressed in a partnership agreement? What roles and responsibilities are essential within a partnership? How can partners build the trust and commitment essential to continuing success? What methods of communication will help drive partnership momentum? Clearly, both researchers and decision makers need to approach the concept of a research transfer partnership armed with sufficient knowledge and understanding to move forward with optimism and confidence. Moreover, Canadians need the “real world” evidence-based results that are likely to emerge from research transfer partnerships to encourage the judicious use of funds and help diminish the occurrence of disease and the upward spiraling costs of treatment. BACKGROUND Over the last decade, there has been a cultural shift towards the integration of evidence-based research into health care services. Canadian researchers and decision makers have begun to recognize the potential benefits that accrue through the combination of their various knowledge and expertise to form research transfer partnerships. These partnerships originate from a merging of action research, participatory research, program evaluation and knowledge utilization research (Denis & Lomas, 2003b). Throughout the 1990s a similar trend towards partnership development has occurred within business management, and research and development in other disciplines ((Lewin & Johnston, 1997; Black, Akintoye, & Fitzgerald, 2000; Hall, Bockett, Taylor, Sivamohan, & Clark, 2001; Hall et al., 2001) Walter, Nutley, and Davies (2003) offer important lessons built on the understanding of partnerships drawn from the literature of social care, criminal justice services, and education (Walter, Nutley, & Davies, 2003). Literature drawn from these various disciplines suggest that the common components of successful partnerships include selecting the right set/combination of partner, putting in place an

37

agreement (formal or informal) based on trust and commitment, ensuring frequent personal interaction (communication) of partners at many levels, developing a clear understanding of roles and responsibilities, identifying a process to solve problems (conflict resolution), investing resources by both partners, and ongoing evaluation of the partnership (Paun, 1997; Ross, Lavis, Rodriguez, Woodside, & Denis, 2003; Kozak & Cohen, 1997; Black et al., 2000; Canadian Health Services Research Foundation, 2004; Canadian Health Services Research Foundation, 2002; MacLean, Kothari, and Edwards, 2003 ; Kozak et al., 1997; Ross et al., 2003). The benefits that are reaped from partnerships include a holistic effect as shared knowledge builds and each partner develops a more comprehensive understanding of the research process (Ross et al., 2003; Hall et al., 2001). Partnerships may support the work of partners during economic downtimes or during cutbacks and service re-organization (Denis et al., 2003a; Kozak et al., 1997). Partners gain knowledge of their partner’s culture, environment and roles (Kozak et al., 1997; Ross et al., 2003; Walter et al., 2003; Black et al., 2000). Researchers can screen literature and diminish the amount of time that decision makers spend on reading(Golden-Biddle et al., 2003). Decision makers can help researchers to access data sources(Ross et al., 2003). Both researchers and decision makers/practitioners gain knowledge and skills through partnership activities (Walter et al., 2003; Antil et al., 2003; Canadian Health Services Research Foundation, 200; Carayannis et al., 1999). Partners can share early results to prompt meaningful discussions that can help focus and adjust methodology (Golden-Biddle et al., 2003). A shortage of time to dedicate to the partnership can be a persistent problem for both partners as they attempt to balance their professional responsibilities and those of the partnership (Walter et al., 2003; Walter et al., 2003; Denis et al., 2003a; Denis et al., 2003b) Academic departments tend not to recognize partnership activities as career enhancing for researchers (Canadian Health Services Research Foundation, 2002; Ross et al., 2003). Researchers may believe that working with decision makers increases costs and may lead to a loss of control and biased results(Bartunek et al., 2003; Ross et al., 2003). Finally, when researchers are partnering with government agents, others may think that the researcher is an agent of the government (Goering et al., 2004). Hall et al. (2001) commented that innovation takes place at the interface of organizations and the various actors in research and design. In the case of the research transfer partnership, this concept may hold truth. Further investigation will help elucidate the essential steps to success to link research development with the practical world of health services. Potential partners who approach each other armed with knowledge gained from the exploration of successful partnerships have much to gain in helping to improve health services and the overall health of Canadians. Previous research conducted by the CHRU on research transfer resulted in a large set of research partnership success indicators (MacLean, Kothari, & Edwards, 2003, Kuhn, MacLean, & Leonard, 2004). While there was a subset of indicators for relationship success consistent with the literature reviewed above, critical factors or indicators were not identified. Furthermore, interviewees were not asked to comment directly on whether or not their relationships were successful, and what made them so, nor on whether these partnerships were a beneficial way to conduct research and transfer findings in this context. This current study allowed us to ask these questions directly, rather than infer, and to explore these indicators in more depth for

38

confirmation and elaboration. The extreme cases design will allow some assessment of which of the indicators may be most important in a researcher-decision maker context. METHODS Eight qualitative, semi-structured, 30-60 minute interviews were conducted over the telephone. Interviews were audio taped and transcribed. All identifiers were removed from transcripts (names, university affiliations). The interviews focussed on researchers’ and decision-makers’ perspectives, experiences and advice regarding conducting successful partnerships with public health research in relation to policy research directed at government. Participants Participants were prime members1 from within established research transfer partnerships identified by the MOH-LTC, Research Policy Branch. The inclusion criteria for an established partnership were that a partnership must:

• be formalized by a written agreement • have existed as a formalized partnership for a minimum of one year • have clear research objectives and an operational plan • demonstrate evidence of research output based on MOHLTC partnership evaluation data.

Two prime members from each research transfer partnership were interviewed—one from the research partnership group and one from the decision maker partnership group. Analysis Transcribed interviews were content analyzed. Qualitative data were read first for simple content summaries, and secondly for new and emerging themes. In a third step, some data was reviewed, in light of new information arising from the analysis. Confirming and disconfirming examples were identified. The investigators reviewed the results of the analysis of the semi-structured interviews along with a synthesis of the literature search to develop the guideline document. RESULTS The qualitative analysis of the interviews was used in collaboration with evidence from the literature and previous findings to create two four-page guideline documents. The documents included one general, orientation page on research transfer partnerships followed by three pages each for Ministry personnel/decision makers and for researchers on ways to set up a research transfer relationship so that it will be successful: what to do, assumptions to check out, important factors to consider, and potential benefits of such relationships, outlining factors in success and problems to avoid, along with questions to consider before committing to the relationship. While the actual guideline documents for the Ministry were as described above, we present them here in a tabular form for the purposes of ease of comparison, in the modified tool now following. 1 A prime member is a principal researcher or designate for a research transfer partnership and a senior decision maker or designate who is responsible for the research transfer partnership activities within a facility

39

PROJECT 2:

B. THE TOOL

FORMAL RELATIONSHIP GUIDELINES FOR RESEARCH TRANSFER PARTNERSHIPS

At their best, research transfer partnerships provide decision-makers and stakeholders with highly useful research findings and give researchers the opportunity to do work that makes a difference. The most successful partnerships result in high quality deliverables and committed, engaged partners. As one member of a highly successful partnership put it: “it's that unique combination of accountability within a collegial relationship.” Below are some key concepts about such formal research transfer partnership relationships. These are followed by a brief set of guidelines for researchers and decision-makers considering entering into such partnerships with funding from the Ontario Ministry of Health and Long-Term Care (MOHLTC). These are based on findings from the literature (Ralphs-Thibodeau, MacLean, & Edwards, 2004) and interviews with researchers and decision-makers from highly successful MOHLTC partnerships. Definition of partnership: There are three key types of policy maker involvement in research transfer partnerships: the formal supporter; the responsive audience, and the integral partner (Ross et al, 2003). These partnerships vary along a continuum of degree of involvement by the decision-maker partner in the research process. However, all three involve decision-makers in the role of helping develop the research questions and methods, in funding, in receiving deliverables, in negotiating changes in the product and relationship, and in providing legitimacy and access to resources within the Ministry. The guidelines described in this document should fit most types of partnership, but particularly the responsive audience and integral partner models where both partners hope to develop an ongoing and committed research transfer relationship. Benefits of research transfer partnerships :

• Research questions and findings become more applicable and policy-relevant; • The research activity has a greater impact in practice and policy; • Partners support each other and champion the research during difficult periods, including economic downturns, cut-backs, reorganizations, policy changes; • Decision makers and researchers may gain increased access to data sources; • Researchers may provide information relevant to policy beyond the scope of the specific research program; • Through working together, researchers and decision-makers share useful ideas, emerging issues, and networks beyond the scope of individual projects, enhancing research, policy, and practice opportunities in general;

Key Challenges:

• Walking the tightrope between engagement of both partners and partner independence.

40

• Understanding the roles, priorities, needs, and responsibilities of both partners; • Working with employee turnover among key people involved in a partnership. • Giving the relationship the time and attention it requires, and getting support from your organization to do so.

41

Formal Relationship Guidelines for Research Transfer Partnerships: Table Version

The following table presents the major areas for consideration in developing a successful research transfer partnerships. As well as areas important for both Decision-Makers and Researchers, the table shows information of specific importance to each.

Recommendations for Both Specific Decision-Maker Recommendations

Specific Researcher Recommendations

1. Before the Partnership: Doing your homework. No common recommendations for this section.

1.Before the Partnership: Doing your homework. Research transfer partnerships require a considerable investment in time for supporting the relationship. Time is also required for research activities. If the research resulting from this process will directly aid your policy and programming needs, if research activities fit in well with your regular work activities with little extra time required, and/or if you bring expertise to the research process or content area, it may be worth your while to consider a greater involvement in developing research ideas, data collection, interpreting results, and moving results into policy and practice. Before starting a research transfer partnership, ask yourself the following questions:

• Why do you want to see this particular research done? What information are you missing? • What is the current state of knowledge in this area? • Why is a research partnership the best way to answer your questions? • What kind of research partnership do you want?

o Working with a particular researcher or group? This could be due to special expertise and/or previous positive working history. If so, you may want to

1. Before the Partnership: Doing your homework

• Why are you considering doing this type of research? Where do you hope to see the impact of your work? • Do you understand in a basic way how government is structured, how it is run, and how it is different in priorities and timelines than the research world? • Will you enjoy working with policy people and policy spheres? • Do you work in an organization that will support the time required of you to nurture ongoing Ministry relationships? • Will your career advancement be dependent upon peer-reviewed publications? If so, be aware that such Ministry-related work often does not lead to such publications. You may wish to wait until further along in your academic career to pursue this type of research. • What decision-maker(s)

42



consider flexible funding arrangements, and joint development with researchers on the proposal and deliverables. o A formal competition with a Request For Proposals process? This will bring in a variety of researchers to choose from, with a transparent selection process. Sometimes, certain amounts of funding dollars, or access to specific initiatives are tied to an RFP process. o Are you considering bringing in community stakeholders? If so, how will that affect the choice of research partner, and the structure of the consortium?

• Will it require a few years to get the answers you need? If so, longitudinal funding and a longer-term relationship are in order.

would you like to work with? At what level of seniority? Decision-makers at a senior level can effectively champion your research and findings, but may have little time to commit personally to the work.

43



2. The Formal Agreement • Formal agreements generally specify deliverables, timelines, roles and responsibilities, termination rules, and intellectual property rights agreements. Both partners usually find the clarity in mutual accountabilities that comes from a structured agreement to be very helpful. • Often there is a structured template required by the Ministry for formal agreements such as contracts and other funding arrangements that has little room for variability. • Researchers and their institutions may have different perspectives than the Ministry on items within the standard agreements, particularly in areas such as intellectual property rights. Allow time for negotiation to occur between the Ministry and the research institution to work out agreeable specifics where possible.

2. The Formal Agreement • Remember that accountability is a two-way street, and decision-makers are accountable to researchers in supporting the work and communicating needs and changes in a clear and timely fashion.

2. The Formal Agreement • Do contact your institution’s Office of Research Services early in the process since it has the responsibility for negotiating and signing the agreement. • Researchers and their institutions may have different perspectives than the Ministry on items within the standard agreements, particularly in areas such as intellectual property rights. Allow time for negotiation to occur between the Ministry and the research institution to work out agreeable specifics where possible. • As a researcher, be fully aware of what is required for each item in the agreement, and what you must do prior to publishing in terms of formal reporting to the Ministry.

3. The Relationship: The successful relationship involves engagement, trust, and commitment between partners. These are built through frequent and effective communication, championing the needs of the research and the partner within your organization, and developing an understanding of your partner over time. Maintaining this relationship requires a great investment of time from both researchers and decision-makers.

3. The Relationship • Turnover: Turnover in MOHLTC positions is frequent. Please prepare researchers for this possibility and build in safeguards allowing your replacements to come up to speed as quickly as possible, including provisions for efficient transfer of files.

3. The Relationship • Turnover: Researchers need to be aware that if they or a member of their team leaves, their organization is still responsible to the Ministry for deliverables. This possibility needs to be prepared for by research organizations

44



• Communication: Frequent, early, and effective communication between partners about clarifying the research, problems with the research or deliverables, changes in the research or deliverables, changes in the external environment, and warnings when research results will be surprising are important to the quality of the final product and the quality of the relationship. Do:

o Meet face to face when possible; o Communicate at regular intervals, with updates, in a protected time arrangement; o Identify contact people from both partners who will have responsibility for ongoing communication; o Communicate any concerns about the project early in the process.

• Roles: Even in the most collegial of relationships, it is important to remember who is funding the research. The Ministry can offer insightful input on methods and approaches. The Ministry does not tell researchers what the outcomes of the research should be. Its role as a funder is to say what research questions are important to it, and what kind and format of information is needed to move policy forward. The researcher is responsible for

especially in cases requiring specialized research expertise. • Multiple contacts: Over time, develop a set of contacts in different policy units. This should enhance the possibility of pursuing different aspects of your research interest, and increase the breadth of impact of your work.

45



ethics clearance and compliance with ethical requirements.

• The Engagement Tightrope: Ongoing engagement is a key element in success, yet both partners need to be seen as independent from each other. Otherwise the integrity, objectivity, and rigor of the research can be questioned by the government, stakeholders, and fellow researchers. Decision-makers can be seen as playing favourites. Developing a collegial yet transparent relationship with input and engagement from both partners is the desired end.

o Transparency suggests a statement in clear language of how decisions are reached, open presentation of findings to the public, and public access to findings. The higher the level of expected risk and controversy, the greater the degree of transparency needed. o Choice of researcher(s) may need to be done through an RFP or peer-reviewed process if level of controversy is high (as stated above, this is not the only reason for using an RFP process). o Areas of conflict of interest, or of perceived conflict

46



of interest, between researchers and decision-makers must be considered in determining to what degree the decision-maker will be involved in various aspects of the research.

• Flexibility: Both partners need to be flexible with each other as policy environments change, research disruptions occur, and contact people move on.

4. Stakeholders: Consider involvement of community stakeholders. Getting their buy-in and input on models, tools, and research findings can be very helpful.

• If the project calls for involvement of community stakeholders, it will be very important to work with the researchers in an open and transparent fashion. • Everyone shares long-range goals of better health and outcomes for people. When things get stressful among all three groups (researchers, decision-makers, stakeholders), keep in mind that all involved are aiming for these long-term goals. • Researchers and decision-makers can show support for each other at conferences or workshops involving stakeholders. Allow whichever one of the two is most appropriate to take the lead in presenting, perhaps having an external facilitator, while the other

4. Stakeholders: 4. Stakeholders: Researchers should be aware that decision-makers have more constraints on them about what they can say to the community regarding ideas and opinions.

47



supports the process in some visible fashion.

5. At the Beginning: Things to get you off on the right foot at the start of the partnership.

• Both partners need to put all the information “on the table” at the start for both parties to see where and how this project fits their respective agendas. • Discussion of differences in organizational culture and procedures should occur at this stage, laying a foundation of understanding and trust. • Set clear expectations to monitor the process and ensure compliance with deliverables, timelines, etc. • Develop a joint work plan. • Meet more frequently at the beginning, particularly face to face if possible. As the relationship develops, meeting by telephone and email can become more frequent. • Appoint specific contact people in each group. Communication on a day-to-day basis will flow through them and simplify contact.



• As a researcher, be clear about what you want the research to accomplish, and be realistic about what can be done in terms of rigour, longitudinal possibilities etc. • The researcher is responsible for obtaining ethics clearance when the project includes research with human participants. Be sure to allow time in the work plan for this process.

48



6. Mistaken Assumptions:

6. Mistaken Assumptions: • Do not assume that researchers have an “ivory tower” mentality. Many have partnered with decision-makers before, or have work experience in practice, management, or government, and so have a strong awareness of practical needs and implications. • Do not assume that researchers are not interested in the impact of their deliverables, or in understanding the bigger picture that their findings fit into for policy decisions.

6. Mistaken Assumptions: • Do not assume decision-makers are not sophisticated in terms of research content areas, questions, designs, and analysis. They may have strong research backgrounds. • Do not assume that your research findings are the only inputs that decision-makers have to consider when making or changing policy. The findings are just one of many information sources and priorities. It does not mean the findings are not taken seriously if not used as recommended in the research.

The researchers and decision-makers we spoke to as part of the development of these guidelines were enthusiastic about the benefits of research transfer partnerships. They were also enthusiastic about their partners. Hopefully the guidelines will help you to develop a partnership with the communication, respect, accountability and commitment that will lead to enhanced research and even more enthusiastic research transfer.

49

TWO TOOLS FOR WORKING TOGETHER EFFECTIVELY: CONCLUSIONS OVER BOTH PROJECTS

Implementation Recommendations: The Evaluation Workbook: The Evaluation Workbook and Technical Report represent the main result of Project 1. There are other issues, related to the implementation of the Workbook, which arose over the duration of this project. These issues are highlighted here for those considering the use of a tool such as the Evaluation Workbook.

a) Collaborative Selection of Indicators at the Outset: The rigorous evaluation of research transfer activities will rely on valid and comprehensive data. It would be beneficial for those charged with the overview of Ministry-researcher partnerships (Ministry Research Unit, or “the Research Unit”) to involve researchers and their Ministry partners at the start of the evaluation, for the purposes of identifying appropriate indicators and data sources. This preliminary step will ensure that relevant data are being collected for the evaluation.

b) Suggestions for Data Sources for Grant Files: The pilot study revealed that Grant Files ought to be supplemented with additional data for completion of the Workbook. The Research Unit could ask, at the outset, that research and Ministry partners keep copies of the following data elements: meeting agendas and minutes/action plans, written correspondence (including emails), mailing lists used to disseminate materials, workshop/conference evaluations, and project website addresses and associated statistics (e.g., number of hits for relevant pdf file). These elements are relatively easy to access if partners understand from the start that they need to include them in their project files. Some elements are difficult to access in retrospect, as partners may only track them if there are problems occurring with the project.

c) Awareness of Subjective Indicators: Some indicators in the Workbook are objective in nature, while others are subjective. Assessment of objective indicators is facilitated by documented information (most of the time). Subjective indicators, on the other hand, are often assessed using perceptions or opinions. The Research Unit can have more confidence in the rating of subjective indicators if some effort is taken to validate perceptions (e.g., interviewing both researchers and Ministry partners about “rapport”, as one example, and comparing perceptions around extent of rapport).

d) Consultation On Final Assessments: The Workbook contains sections where final assessments of progress on each evaluation question are recorded. Researchers and Ministry partners indicated that they would like the opportunity to review and comment on the ratings. Even better would be a collaborative process to identify benchmarks of “success” at the beginning of the evaluation.

e) Stakeholders Extend Beyond the Triangle Partnership: The Evaluation Workbook is primarily based on a triangle partnership model. That is, sometimes a partnership between the researcher, Ministry partners and a primary community

50

stakeholder may exist. In some of these cases, however, there are additional significant stakeholders involved in the transfer of research. Furthermore, transfer to the community might have more impact on health than in the policy and program arena. The Research Unit might want to consider if and how to incorporate the effectiveness of these research transfer activities in their assessment.

f) Determining Usefulness of Understanding Research Transfer Capacity Building: Study participants mentioned that they were often involved in the training of students. While this does not represent a research transfer activity per se, it might be important to note that triangle partnership members might be involved in building capacity around research transfer. The Research Unit might want to consider if this information is useful internally, and if so, how to characterize capacity building.

g) One Tool for a Variety of Purposes? One of the strengths of the Evaluation Workbook is that it explicitly recognizes the development of partnerships over time. This growth ought to lead to different research utilization outcomes, as identified in the Workbook. “One-off” funded projects, like those funded under a strategic grant, will not reflect a dramatic growth in partnership development. Consequently, the latter third of the Workbook – indicators for mature partnerships – is less relevant for single project funded research. The Research Unit might want to reconsider their motivation for using a generic evaluation workbook for assessing different types of projects and organizations.

h) Ministry Partners Might Provide More Feedback: A common theme, throughout the study, was that researchers were unaware of the usefulness of the final report for their Ministry partners. Indeed, most researchers were unaware if their report had been used at all. The Research Unit might strongly encourage their colleagues to debrief with researchers after a final report has been submitted.

Formal Relationship Guidelines: These are presented second after the Evaluation Workbook since that was the order of development, and the Formal Relationship Guidelines emerged as an idea following the Workbook. However, they are really meant to come first in the process of a research transfer partnership. Rather than an evaluation tool, they are meant to be looked at before a partnership would begin, and at key stages throughout the partnership. They are the sober second thoughts of this process. As such they are quite straightforward, and little complexity is required in their implementation. CONCLUSION Both tools reflect the importance of partnership variables in allowing optimal flow of research findings into policy and policy back into research again. They recognize that both partnerships and the skills needed to maintain partnerships shift over time. The tools also speak to future roles that researchers might play in this process that could include helping decision makers to anticipate future research questions that they will need findings to, based on their current partnership project results. If decision makers could be assisted to anticipate their research questions, then findings might be provided in a more timely fashion. However the future unfolds for research transfer partnerships, it will be

51

important to plan and evaluate them, as the results of optimal partnership process lead to improved evidence use in policy, and improved policy-awareness and applicability in research. Tools such as these are aimed at helping partners plan, track, and improve their work together to achieve their goals.

52

REFERENCES

Antil, T., Desrochers, M., Joubert, P., & Bouchard, C. (2003). Implementation of an innovative grant programme to build partnerships between researchers, decision –makers and practitioners: The experience of the Quebec Social Research Council. Journal of Health Services Research, 8, S2:35-S2:43.

Bartunek, J., Trullen, J., Bonet, E., & Sauguet, A. (2003). Sharing and expanding academic and practitioner knowledge in health care. Journal of Health Services Research and Policy, 8, S2:62-S2:68.

Black, C., Akintoye, A., & Fitzgerald, E. (2000). An analysis of success factors and benefits of partnering in construction. International Journal of Project Management, 18, 423-434.

Canadian Health Services Research Foundation (2002). Productive Partnerships; report on the 2002 CHSRF Annual Invitational Workshop Health Services Research Foundation.

Canadian Health Services Research Foundation (2004). How to be a good research partner: a guide for health-system managers and policy makers Canadian Health Services Research Foundation.

Caplan, N. (1979). The two communities theory and knowledge utilization, American Behavioral Scientist, 22: 459-470.

Carayannis, E. G. & Alexander, J. (1999). Winning by co-opeting in strategic government-university-

industry R & D Partnerships: the power of complex dynamic knowledge networks. Journal of Technology Transfer, 24, 197-210.

Denis, J. L., Lehoux, P., Hivon, M., & Champagne, F. (2003a). Creating a new articulation between research and practice through policy? The views and experiences or researchers and practitioners. Journal of Health Services Research and Policy, 8, S2.44-S2.6.

Denis, J. L. & Lomas, J. (2003b). Convergent evolution: the academic and policy roots of collaborative research. Journal of Health Services Research and Policy, 8, S2:1-S2:6.

Frenk, J. (1992). Balancing relevance and excellence: Organizational response to link research with decision-making, Social Science and Medicine, 35(11): 1397-1404.

Goering, P., Butterill, D., Jacobson, N., & Sturtevant, D. (2004). Linkage and exchange at the

organizational level: a model of collaboration between research and policy. Journal of Health Services Research and Policy, 8, S2:14-S2:19.

Golden-Biddle, K., Reay, T., Petz, S., Witt, C., Casebeer, A., Pablo, A. et al. (2003). Toward a communicative perspective of collaborating in research: the case of the researcher—decision-maker partnership. Journal of Health Services Research and Policy, 8, S20-S25.

53

Griffin, P. & Edwards, N. (December, 2001). Literature Review: Research Receptor Capacity and Research Utilization. Paper prepared for Ontario Ministry of Health and Long Term Care, Research Unit, Corporate Policy Branch.

Hall, A., Bockett, G., Taylor, S., Sivamohan, M. V. K., & Clark, N. (2001). Why research partnerships really matter: innovation, theory, institutional arrangements and implications for developing new technology for the poor. World Development, 29, 783-797.

Kozak, R. A. & Cohen, D. H. (1997). Distributer-supplier Partnering Relationships: a case in trust. Journal of Business Research, 39, 33-38.

Landry, R., Lamari, M., Amara, N. (2000). Extent and determinants of utilization of university research in public administration, Paper prepared for presentation at the Canadian Political Science Association Meeting, Quebec, July 29-Aug 1.

Landry, R., Amara, N. and Lamary, M. (1998). Utilization of social science research knowledge in Canada, Université Laval.

Lavis, J.N., Ross, S.E., Hohenadel, J., Hurley, J., Stoddart, G.L., Woodward, C., Abelson,

J. and Giacomini, M. (2000). The Role of Health Services Research in Canadian Provincial Policy -making (Rep. No. 1997-021).

Lewin, J. & Johnston, W. (1997). Relationship marketing theory in practice: a case study. Journal of

Business Research, 39, 23-31.

MacLean, L.M., Kothari, A., & Edwards, N. (2003). Research Transfer Evaluation Workbook : Technical Report. Ontario Ministry of Health and Long-Term Care, Research Unit, Corporate Policy Branch, Integrated Policy and Planning Division.

MacLean, L. M., Plotnikof, R. C., & Moyer, A. (2000). Transdisciplinary work with psychology from a

population health perspective: an illustration. Journal of Health Psychology, 5.

Oh, C.H. and Rich, R.F. (1996). Explaining use of information in public policy-making. Knowledge and Policy, 9:1, 3-35.

Paluck, E.C., Williamson, D.L., Milligan, C.D., Frankish, C.J. (2001) The use of

population health and health promotion research by health regions in Canada, Canadian Journal of Public Health, 92(1):19-23 .

Paun, D. A. (1997). A study of "best" versus "average" buyer-sellor relationships. Journal of Business

Research, 39, 13-21.

Ralphs-Thibodeau, S., MacLean L.M., Edwards, N. (2004). Formal Linkages Guidelines Project Literature Search: Executive Summary and Annotated Bibliography. Ontario Ministry of Health and Long-Term Care, Research Unit, Corporate Policy Branch, Integrated Policy and Planning Division.

Rich, R.F. (1979). The pursuit of knowledge. Knowledge: Creation, Diffusion,

54

Utilization. 1, 6-30. Ross, S., Lavis, J., Rodriguez, C., Woodside, J., & Denis, J. L. (2003). Partnership experiences;

involving decision-makers in the research process. Journal of Health Services Research and Policy, 8, S2:26-S2:34.

Tiefenbach, H. & Edwards, N. (November, 2001). An evaluation of research transfer and utilization needs in selected branches of the Ontario Ministry of Health and Long-Term Care. Report prepared for Ontario Ministry of Health and Long Term Care, Research Unit, Corporate Policy Branch.

Walter, I., Nutley, S., & Davies, H. (2003). Increasing research impact through partnerships: evidence

from outside health care. Journal of Health Services Research and Policy, 8(S2), 58-61. Webber, D.J. (1987). Legislators` use of policy information. American Behavioral

Scientist, 30, 612-31.

Knowledge Transfer Discussion Paper - University of...

Documents

Transcript of Knowledge Transfer Discussion Paper - University of...