Protocol for evaluating stakeholder engagement in the ... · 5/25/2020 · engagement in the...

Development and evaluation of learning resources

Protocol for evaluating stakeholder engagement in the design, evaluation, and dissemination of the Informed Health Choices resources for helping secondary school students learn to think critically about health claims and choices

IHC CHOICES team

Working paper, 25. May 2020

www.informedhealthchoices.org

http://www.informedhealthchoices.org/

IHC CHOICE Stakeholder Evaluation: Protocol 2

Colophon

Title Protocol for evaluating stakeholder engagement in the design, evaluation, and dissemination of the Informed Health Choices resources for helping secondary school students learn to think critically about health claims and choices

Authors Allen Nsangi,1,2 Andrew David Oxman,3 Matt Oxman,3 Sarah E Rosenbaum,3 Daniel Semakula,1,2 Ronald Ssenyonga,1,2 Michael Mugisha,2,4 Faith Chelagat,2,5 Margaret Kaseje,5 Leaticia Nyirazinyoye,4 Iain Chalmers, 6 Nelson Kaulukusi Sewankambo1 1. Makerere University, College of Health Sciences Kampala, Uganda

(A Nsangi MSc; [email protected], D Semakula MD; [email protected], R Ssenyonga MSc; [email protected], N K Sewankambo MD; [email protected])

2. University of Oslo, Oslo, Norway (A Nsangi, D Semakula, R Ssenyonga, M Mugisha MSc; [email protected], F Chelagat MSc; [email protected])

3. Centre for Informed Health Choices, Norwegian Institute of Public Health, Oslo, Norway (A D Oxman MD; [email protected], M Oxman BJ; [email protected], S E Rosenbaum PhD; [email protected])

4. University of Rwanda, Rwanda (Micheal Mugisha; Laetitia Nyirazinyoye; [email protected])

5. Tropical Institute of Community Health and Development (TICH), Kisumu, Kenya (F Chelagat; M Kaseje PhD; [email protected])

6. James Lind Initiative, Oxford, United Kingdom; (I Chalmers; [email protected])

Corresponding author

Andy Oxman [email protected] Centre for Informed Health Choices Norwegian Institute of Public Health Postboks 222 Skøyen 0213 Oslo, Norway

Keywords Stakeholder engagement, involvement, participation, critical thinking, education,

Citation [Citation text]

Article category ☐ About Informed Health Choices

☐ Key concepts and glossary

☐ Learning resources

☐ Systematic reviews Development and evaluation of learning resources ☐ Contextualising learning resources

☐ Claim evaluation tools

☐ Editorials and commentaries

☐ Grant applications

Date May 2020

mailto:[email protected]


Abstract

Background: We will be developing and evaluating resources for helping

secondary school students learn to think critically about health claims and choices.

We will bring together key stakeholders; such as secondary school teachers and

students, our main target for the IHC secondary school resources, school

administrators, policy makers, curriculum development specialists and parents, to

enable us gain insight about the context.

Objectives:

1. To ensure that stakeholders are effectively and appropriately engaged in the design, evaluation and dissemination of the learning resources.

2. To evaluate the extent to which stakeholders were successfully engaged.

Methods: Using a multi-stage stratified sampling method, we will identify a

representative sample of secondary schools with varied characteristics that might

modify the effects of the learning resources such as, the school location (rural, semi-

urban or urban), ownership (private, public) and ICT facilities (under resourced,

highly resourced). A sample of schools will be randomly selected from the schools in

each stratum. We will aim to recruit a diverse sample of students and secondary

school teachers from those schools.

Other stakeholders will be purposively selected to ensure a diverse range of

experience and expertise.

Results: Together with the teacher and student networks and the advisory panels,

we will establish measurable success criteria that reflect the objectives of engaging

stakeholders at the start of the project and evaluate the extent to which those criteria

were met at the end of the project.

Conclusion: We aim for an increase in research uptake, improve quality and

appropriateness of research results, accountability and social justice.

Key words:

Stakeholder engagement, involvement, participation, critical thinking, education


Background

As part of the Informed Health Choices (IHC) project, we have previously

developed and evaluated resources for helping primary school children learn to

think critically about health claims and choices.1 In that project, we brought

together various stakeholders and end users, such as children, teachers and

policymakers, into the different phases of our work.2 This was particularly

important since none of the researchers belonged to the end user-groups for

which we were developing resources. Teachers were included as collaborators

through brainstorming3 and prototyping workshops; policymakers were

periodically informed of progress and consulted; and children’s feedback was

sought early on in the project through workshops and school visits. This

enabled the research team to gain insight about the context and stakeholders,

formulate ideas, sketches and prototypes.

Building on that work, we are now developing and evaluating resources for

secondary school students. We will engage teachers, students, policymakers and

other stakeholders throughout this work. The purpose of this protocol is to

describe how we will ensure that stakeholders are effectively and appropriately

engaged in the design, evaluation, and dissemination of the learning resources;

and how we will evaluate the extent to which stakeholders are successfully

engaged.

As with the previous project, stakeholders were not involved in the project grant

application, project design or in the development of this or other protocols.

In this protocol, we will use the following definition of ‘Stakeholder engagement’ and

‘stakeholders’:4

Stakeholder engagement is “an iterative process of actively soliciting the

knowledge, experience, judgment and values of individuals selected to

represent a broad range of direct interests in a particular issue, for the dual

purposes of:

- creating a shared understanding

- making relevant, transparent and effective decisions”


Stakeholders are “individuals, organisations or communities that have a vested

interest in the process and outcomes of the project, research or policy

endeavour.”

Whilst there has been growing interest and demand for stakeholder

engagement in health research4-12, there has also been growing interest in

stakeholder or user engagement in educational research.13 The terms

‘stakeholder engagement’ (commonly used in North America) and ‘patient and

public involvement’ (PPI) in research (commonly used in the UK) are sometimes

used interchangeably.14 However, there are important differences in the

terminology that is used. For example, INVOLVE (in the UK) defines ‘public

involvement’ in research as research being carried out ‘with’ or ‘by’ members of

the public rather than ‘to’, ‘about’ or ‘for’ them15, and uses the following

definitions:

• Involvement – where members of the public are actively involved in

research projects and in research organisations

• Participation – where people take part in a research study

• Engagement – where information and knowledge about research is

provided and disseminated

The International Association for Public Participation (IAP2)16 seeks to promote

and improve the practice of public participation or community and stakeholder

engagement more broadly – not specifically in research. In contrast to the

definitions used by INVOLVE, it uses the following definitions to describe

different degrees of participation in decision making:

• Inform – To provide the public with balanced and objective information

to assist them in understanding the problem, alternatives, opportunities

and/or solutions

• Consult – To obtain public feedback on analysis, alternatives and/or

decisions

• Involve – To work directly with the public throughout the process to

ensure that public concerns and aspirations are consistently understood

and considered

• Collaborate – To partner with the public in each aspect of the decision

including the development of alternatives and the identification of the

preferred solution

• Empower – To place final decision making in the hands of the public.

In this protocol, we will use terminology derived from the International

Association for Public Participation (IAP2) and Bruns’ extended ladder of

participation.17 18


Our aim is to increase the level of public impact by informing, consulting,

involving and collaborating with others in the decision making process while

retaining some authority in the final decisions. In Brun’s extended ladder of

participation17, this process is deliberately arranged horizontally (as seen in

Table 1) to suggest a range of options rather than a hierarchy.

Stakeholders can be engaged to various degrees in designing, evaluating, and

disseminating learning resources. The objectives of different levels of stakeholder

engagement, what stakeholders can expect at each level, and ways of achieving

different levels of engagement are summarised in (Table 1).

Table 1. Levels of stakeholder engagement in designing, evaluating, and disseminating learning resources*

Information Consultation Involvement Collaboration Delegation

Objectives of

stakeholder

engagement

To provide

stakeholders with

information in order

to help them

understand the need

for the resources

To obtain specific

types of input from

stakeholders, such

as feedback on the

resources (including

their usefulness and

value) or input into

the design,

evaluation, or

dissemination of the

resources

To work directly with

stakeholders in

designing the

resources, to ensure

that their views are

understood and

considered, or to

engage them in

deliberations about

problems and

proposed solutions

To partner with

stakeholders

throughout the

process of designing,

evaluating, and

disseminating the

resources

To give control over

aspects of the

design, evaluation,

and dissemination of

the resources

What stakeholders

can expect

To be kept informed To be listened to,

and provided with

information about

how their input has

influenced

subsequent

decisions

To work together in

designing,

evaluating, and

disseminating the

resources

To jointly work

through problems

and proposed

solutions that will be

incorporated as far

as possible.

To deliberate on final

decisions and

participate in

consensus

processes.

Ways of achieving

the objective

One-way information

dissemination such

as:

- A website

- Tailored information

(simplified and

translated in various

languages if

required)

- Presentations

Two-way

communication

which involves

seeking input,

listening, and the

exchange of views.

This may take the

form of:

- Written comments

- Interviews

- Focus groups

- Surveys

Interactive

discussion and

dialogue:

- Workshops

- Working groups

Stakeholder

representatives “at

the table”, and active

as team members in

the design,

evaluation, and


resources.

Stakeholders are not

involved in final

decisions but are

involved in:

- Advisory groups

- Consensus

processes

Decisions by a group

or organisation with

specific

authorisation:

- Delegation of

authority to

participate in the

decision-making

processes about the

design, evaluation, or


resources

* Adapted from IAP2 guidance and Bruns extended ladder of participation16 17 19

Potential benefits and harms of stakeholder engagement

Hypothesised benefits of engaging stakeholders in research include better

quality and appropriateness of research, empowering people, increasing uptake


of results, accountability, and social justice.6 11 Other, more specific potential

benefits include user-relevant research questions; user-friendly information,

questionnaires, and interview schedules; more appropriate recruitment

strategies; user-focused interpretation of data; and improved dissemination of

study results.8 However, few studies have formally evaluated the extent to

which stakeholder engagement has achieved these benefits and very few have

compared alternative ways of engaging stakeholders.5-13

In addition to concerns about a lack of evidence about the benefits of

stakeholder engagement, several other concerns have been raised. Poorly

planned and implemented efforts to engage stakeholders can create mistrust,

waste people’s time, and undermine future attempts at engagement.18 Other

concerns include:8 9 13 20

• the risk of tokenism and superficiality;

• a lack of clarity about the nature and purpose of stakeholder

engagement;

• researchers not having the necessary skills, time and resources to

effectively manage stakeholder engagement processes;

• individual stakeholders dominating decisions (unequal participation and

unbalanced representation);

• the boundaries between stakeholders and researchers becoming blurred

over time, resulting in stakeholders losing their perspective as users of

the results;

• potential conflicts between the perspectives of researchers and

stakeholders regarding criteria of quality and appropriateness; and

• slowing down research, making it costlier, and opportunity costs

(forgone opportunities as a result of using resources on stakeholder

engagement).

As with the hypothesised benefits, there is very little evidence of the extent to

which stakeholder engagement has had undesirable effects or the costs of

stakeholder engagement processes .5-13

Our purposes for engaging stakeholders in various aspects of designing,

evaluating and disseminating IHC learning resources for secondary schools are

to ensure that:

• Any concerns they have are heard and considered

• Problems are analysed, described and perceived correctly

• Appropriate solutions are identified

• Important barriers to scaling up use of the learning resources are

identified and addressed

• Promising implementation strategies are identified

• Decisions arising from collected feedback are appropriate and acceptable


• The results of the research are meaningful and accessible to them

However, efforts to engage stakeholders should respect the time they have

available and the value of their potential contributions. In addition to having a

clear purpose, stakeholders’ input should be considered, and it should be made

clear to them when and how they are able to influence decisions. To ensure that

stakeholders are appropriately engaged in the development of the IHC

secondary school resources, we will systematically consider how we can best

engage them in the different phases of the work, as well as factors that might

affect these efforts.19

Overview of the research project

The main aim of the project is to develop and evaluate two sets of digital

learning resources to enable young people (secondary school students) to make

informed personal choices about their health and to participate as scientifically

literate citizens in informed debate about health policies. The project will be

implemented in three East African countries: Uganda, Kenya and Rwanda. The

overall structure of East Africa’s secondary education system is divided into two

levels. In the ordinary level (O-level) students achieve a general certificate of

education after four years in all East African countries except Rwanda, where it

is achieved in three years. The advanced level (A-Level), or upper secondary

school, consists of two years for all East African countries except Rwanda,

where it is three years. O-level students are generally between 13 and 16 years

old, and A-level students are generally between 16 and 18 years old. Secondary

schools in East Africa can be public, private, or private with government

subsidies. The main language of instruction is English in all three participating

East African countries. In this project we will develop English learning

resources for O-level students (lower secondary school) in both government

and private schools in Uganda, Kenya and Rwanda. The entire project will

consist of the following studies, with separate protocols.


Overview of the research project (Figure 1)

Prioritisation of Key Concepts for secondary school learning

resources

We will start with prioritising the IHC Key Concepts21 that secondary school

students need to understand and apply to assess health claims and make

informed health choices. Together with teacher and student networks and

advisory groups in each country, we will determine which concepts should be

taught to O-level students.

Context analysis

We will explore issues which can impact use of the new resources, through

document analysis and interviews with key informants. These include:

• exploring the demand for learning resources for teaching critical

thinking about health

• mapping where teaching critical thinking about health best fits in the

curriculum

• identifying and examining relevant resources already in use

• exploring conditions for introducing new digital learning resources

• identifying opportunities and challenges for developing digital learning

resources.

Human-centred design of the learning resources

We will employ a user-centred approach to designing the learning resources.22

This is characterised by iterative cycles of idea generation, prototyping, testing,

analysis and revision. The aim is to design resources that teachers and students

will experience positively. We will collaborate with a small group of teachers,

students and other stakeholders through brainstorming and prototyping

Phase One

•Prioritisation of Key Concepts

Phase Two

•Context Analysis

Phase Three

•Human Centered design of the learning resources

Phase Four

•Evaluation of the learning resouces in RCTs.

Phase Five

•Process Evaluation


workshops,23 school visits, and piloting and user-testing prototypes of the

resources. Other stakeholders, including members of national advisory panels,

will be informed of progress, and we will invite their feedback and input into

the design of the resources.

Evaluation of the effects of the learning resources

We will use two-arm, cluster-randomised trials to test the effects of the

resources on the students’ ability to understand and apply the IHC Key Concepts

included in the learning resources. We will select representative samples of 90

to 100 rural and urban schools. The primary outcome measures will be

validated tests using multiple-choice questions to measure students’ ability to

understand and apply the IHC Key Concepts. We may also measure hypothetical

decisions, intended behaviours (with hypothetical scenarios), self-assessed

ability, attitudes, and actual decisions. Outcomes will be measured at the end of

the term in which the learning resources are used and again after one year.

Process Evaluation

We will conduct process evaluations alongside the trials. The main objectives of the

process evaluations will be to: explore the results of the trials; identify factors that may

affect the implementation, impact and scaling up of the resources, assuming they are

found to be effective; and to identify potential adverse and beneficial effects of the

intervention which are not included as outcomes in the trials. We will use structured

classroom observations at some of the intervention schools and interviews and focus

group discussions at purposively selected schools in each country. We will interview

teachers, students, policymakers, and parents. We will use a framework analysis

approach to guide data collection and analysis.

Objectives of the evaluation of stakeholder engagement

The primary objectives of this evaluation are:

• To ensure that stakeholders are effectively and appropriately engaged in

the design, evaluation and dissemination of the learning resources

• To evaluate the extent to which stakeholders were successfully engaged


Overview of the stakeholder engagement process (Figure 2)

•Stakeholders; Researchers, Policymakers, Curriculum Developers, International Advisory Panel, IHC network

•Levels of involvement; Consultation, Collaboration, Information

•Involvement Activities; Meetings

Phase One; Prioritisation

of Key Concepts

•Stakeholders; Researchers, Students, Teachers, Headteachers, National Advisory Panel

•Levels of involvement; Information, Consultation

•Involvement Activities; Meetings, Surveys, Workshops

Phase Two; Context Analysis

•Stakeholders; Researchers, Students (networks), Teachers (networks), Research Participants

•Levels of involvement; Consultation, Involvement, Collaboration, Delegation

•Involvement Activities; workshops, (piloting and user-testing)

Phase Three; Human Centred

Desigh

•Stakeholders; Research Participants; Students, Teachers, Headteachers

•Levels of involvement; Collaboration, Information

•Involvement Activities; Meetings, Workshops (Evaluation; cluster randomised trials)

Phase Four; Evaluation of effects of the

learning resources

•Stakeholders; Research participants; Students, Teachers, Headteachers, Policymakers, Parents

•Levels of involvement; Consultation, Involvement

•Involvement Activities; Meetings, Workshops, interviews

Phase Five; Process Evaluation


Methods

Identification and recruitment of stakeholders

The key stakeholders for this project are secondary school teachers and

students, i.e. the target users for the IHC secondary school resources. Other

stakeholders include school administrators (head teachers and directors),

policymakers, curriculum developers, parents, teacher trainers, educational

researchers, researchers with expertise in evidence-based medicine or clinical

epidemiology, and health professionals. We will engage teachers and students in

planning and implementing the project through teacher and student networks

in each country. We will engage other key stakeholders through a national

advisory group in each country, including policymakers, curriculum

development officers, school heads, and parent and civil society representatives.

In addition, we will engage researchers with expertise in education, health,

research methods, design, information and communication technology (ICT)

and science communication in an international advisory group. We also will

engage colleagues with an interest in adapting, developing, evaluating and

implementing IHC resources in other countries through the IHC network. 24

Teacher and student networks

Building on our experience with a network of primary school teachers in

Uganda,2 we will establish a network of secondary school teachers at the start of

the project in each of the three countries, Kenya, Rwanda and Uganda. We will

also establish networks of students in each country. We will work together with

the networks to design, evaluate, and disseminate the resources. The degree of

engagement with these groups will vary from informing them to delegating

some decisions to them. Some specific examples are:

• Informing them at the start of the project about the project’s aims, the

IHC Key Concepts, and development and evaluation plans

• Delegating to them the final decision about criteria for successful

engagement in the project (which input will feed into the content and

structure of the interview guides (questionnaires) to be developed for

the different stakeholder groups). Any resulting questionnaires will be

validated before being used.

• Consulting them, i.e. getting their input through interviews and focus

groups, regarding the demand for learning resources for teaching


students to think critically about health claims and choices; where

teaching these skills best fits in the curriculum; and what the conditions

are for introducing this into schools, including the availability of time and

resources, who the decision-makers are, and what influences their

decisions (as described in the protocol for a context analysis)

• Collaborating and consulting with them, seeking their input into

prioritising and sequencing the IHC Key Concepts in workshops (as

described in the protocol for prioritisation and sequencing of concepts)

• Involving them in generating resource ideas and prototypes together

with us in brainstorming and prototyping workshops

• Consulting with them about how best to ensure that, if the learning

resources achieve their objectives, their use will be scaled up and

sustained

• Consulting with them and getting their input into choice of success

criteria for evaluating the resources

• Consulting with them about how best to communicate the findings of our

research to teachers, students, and parents

Using a multi-stage stratified sampling method in each country, together with

the respective relevant educational authorities we will identify a representative

sample of secondary schools with varied characteristics that might modify the

effects of the learning resources that we will develop. This might include factors

such as the type of ICT available in schools, the size of schools, ownership of the

schools (public versus private), and the location of schools (rural, semi-urban,

or urban). With the relevant educational authorities , each country team will

then identify regions that are representative of the country with respect to the

key characteristics. To minimise travel time and expenses, we will recruit

teachers and students from schools in districts that are in proximity. Working

with the education authorities in each country, we will compile a list of lower

level secondary schools in those districts. A sample of schools will be selected

from those lists including schools with varied characteristics. As with our

previous study, schools involved in piloting resources will be ineligible for

participation in the randomised trial.

For example, the network of primary school teachers in Uganda was selected

from a list of schools in four districts near Kampala. We used stratified sampling

to select the schools. Government-funded and private schools were identified,

and schools were further divided into rural, semi-urban and urban schools. A

sample of schools was then randomly selected from the schools in each stratum.

As in our previous study involving primary schools, we will use stratified

sampling putting into consideration the school location (rural, semi-urban or

urban), ownership (private, public) and ICT facilities (under resourced, highly


resourced) to select schools from lists of schools in the districts identified by the

respective educational authorities in each country, with an aim to recruit a

diverse sample of students and secondary school teachers from those schools.

We will recruit between 20 and 25 people for each country student and teacher

networks, with a goal of having groups that are sufficiently large to include

diverse samples of teachers and students, while not being so large that it is

inefficient and difficult for members of the network to actively engage in

discussions and other activities, such as brainstorming workshops.

In collaboration with the education authority we will send an invitation letter to

each school, introducing the project and requesting to meet with the head

teacher. When we meet with the head teachers, we will introduce the project

and ask them to suggest one or more teachers to join the network. We will

suggest that this should be someone who would be interested, take the time to

participate, who has teaching experience, and who would be likely to contribute.

We will note the gender, level of education, years of experience, and subject

areas of the teachers who are suggested. We will use this information to identify

a diverse sample of teachers in relation to these criteria. We anticipate that the

teachers will primarily be science teachers.

We will send formal letters to the selected teachers, inviting them to be

members of the network. The letters will include: a brief introduction to the

project; an overview of what we expect of members of the network, what they

can expect to gain, and what they can expect of us; timing of the first meeting;

and an email address and telephone number for someone whom they can

contact for more information.

We will ask each teacher on the three country teachers’ networks to suggest

one or more students to join their respective student networks. We will suggest

that this should be someone who would be interested, be able to participate

without adversely affecting the student’s schoolwork, and who would be likely

to contribute. We will also suggest that the selected students should be a

mixture of top, average and low academic performers. We will note the gender

and age of the students suggested, again, to ensure a diverse sample.

We will send formal letters to the students and their parents, inviting the

students to be members of the network. These letters will include information

similar to that outlined in the letters to teachers, consent forms for the parents

to sign, and assent forms for the students to sign.

National advisory panels

The national advisory panels will include representatives from important

groups of stakeholders in each of the three participating countries. The research


team in each country will contact key groups, such as the national curriculum

committees, by phone or email and arrange face-to-face meetings to inform

them about the projects’ aims and objectives. Those groups will be asked to

identify suitable members for the national advisory panels and to identify other

key stakeholders that we should contact.

We will aim to identify 15 individuals in each country with a range of

experience, including: policymakers at the ministerial and regional or district

levels, members of the national curriculum committees, school directors, head

teachers, leaders of teacher unions, and representatives of parents’ groups and

civil society. We will include representatives of groups of stakeholders where

the group is likely to have an interest in being engaged, there are risks not

engaging the group, and the group is likely to have important input into

designing, evaluating, or disseminating the learning resources.

Criteria for selecting individuals to represent the different groups of

stakeholders include selection by their organisation or being influential within

their organisation, being interested, willing to take the time to participate, and

being likely to contribute. We will consult informal contacts, the teacher

networks, and initial invitees (snowballing). We will purposively select

members to ensure a diverse range of experience and expertise. Where we are

unable to recruit representatives of a group, we will report this, in the

evaluation of stakeholder engagement and discuss the implications.

We will collaborate with the national advisory panels; i.e. seek their advice and

incorporate it as far as possible. This includes advice on:

• The criteria for success of their engagement in the project (allowing

them to decide on the final criteria to be used)

• Protocols for each element of the project

• How best to connect the IHC secondary school resources to the

curriculum.

• How best to ensure that use of the resources will be scaled up and

sustained, given they are effective

• Success criteria for the evaluation of the resources

• How best to communicate the findings of our research to key stakeholder

groups

International advisory panel

The international advisory panel will include about 30 people with experience

and expertise in education and education policy, relevant areas of research (e.g.

education, health literacy, science communication, evidence-based practice,

critical appraisal and critical thinking), design, ICT, and learning games, as well


as representatives of key international organisations. We will identify potential

members through personal contacts, relevant publications, and snowballing. We

will aim to include people from a wide range of countries, including low-,

middle-, and high-income countries.

We will contact potential members by email, informing them about the project,

what is expected of members of the international advisory panel, and what they

can expect from us. The international advisory panel will be engaged through

consultation. We will seek their input and feedback on protocols, prioritisation

of Key Concepts, the design and evaluation of the learning resources, and

dissemination. A key role of this group will be to help us ensure that, with a

minimum of modification, the learning resources and our research findings have

the potential to be used in countries other than the three countries where they

will be initially developed and evaluated.

IHC Network

In addition to consulting the international advisory panel, we will consult the

IHC network.24 The IHC network is an informal collaboration of people in over

20 countries who are developing, evaluating or contextualising IHC primary

school resources. It is an international, multidisciplinary group with experience

in research methods, health services research, medicine, public health,

epidemiology, design, education, communication and journalism. It includes

contributors in Arab-speaking countries, Australia, Basque Country, Brazil,

Chile, China, Croatia, French-speaking countries, German-speaking countries,

Iran, Israel, Ireland, Italy, Kenya, Mexico, Norway, Poland, Rwanda, South Africa,

Spain, Uganda, United Kingdom, and United States. In addition to the

International advisory group, the IHC network will help ensure that the learning

resources will have a broader appeal in the different contexts but, in particular,

in their own countries.

Engagement of research participants

In addition to engaging stakeholders as described above, we will engage

research participants in the design and evaluation of the learning resources.

Teacher and student participants who are not necessarily members of the

teacher and student networks will be consulted and involved in designing the

resources through the user testing and piloting.

Participants in the trial will not be engaged in the design of the resources or

planning the research. They will be informed about the resources and the trial

during the trial recruitment process. After the trial is completed they will be

informed about the results.


Participants in the process evaluation will be consulted and involved in

interpreting the results of the trial identifying potential adverse and beneficial

effects of the intervention. If desired effects are achieved this will involve

identifying factors that may affect the implementation, impact and scaling up of

the intervention.

Success Criteria

Together with the teacher and student networks and the advisory groups, we

will establish measurable success criteria that reflect the objectives of engaging

stakeholders at the start of the project and evaluate the extent to which those

criteria were met at the end of the project.15 We will consider criteria related to

whether:

• The stakeholders were informed and engaged to an appropriate extent

• The approaches (of informing and engaging) that were used were

appropriate and worked as expected

• The level of involvement was appropriate

• The input was appropriate and whether it was used appropriately

• The intended outputs were delivered and appropriate

• The intended outcomes were achieved

• The efforts were worthwhile relative to what was achieved

• The appropriateness of group make-up (were important voices

missing/not represented) [the words ‘appropriate’ and ‘worthwhile’ beg

questions about how these words will be defined and how judgements

will be made]

The final decision about the success criteria will be delegated to the networks

and advisory groups.

We will also discuss with them how we plan to engage stakeholders and their

perceptions to judge whether our plans are likely to achieve success by the

agreed criteria. The final decision about these plans and any changes that are

made during the project will be made collaboratively with the networks and

advisory groups.

Data collection

The evaluation will include a basic description of what was done, including the

objectives, and how stakeholders were identified and engaged. This description

will include our plan at the start of the project, modifications made following

discussions with the networks and advisory groups, and any deviations or

modifications made during the project, including the reasons for those.


We will use questionnaires and semi-structured interviews to collect feedback

from the stakeholders and from the research team. The questionnaires will

reflect the agreed upon criteria for success. Separate questionnaires will be

prepared for and completed by the teacher networks, the student networks, the

advisory groups, and participants in the research. Decisions about what

information to collect from different stakeholders and participants and whom to

interview will be made collaboratively with the networks and advisory groups.

The interviews may be conducted in English or a local language (Luganda,

Swahili or Kinyarwanda), based on the preference of the person being

interviewed. Interviews will be recorded, transcribed, and, if necessary,

translated to English. Meetings with the networks and advisory groups will be

in English. At least two members of the research team will participate in

discussions, one of whom will take notes. The notes will be shared with and

approved by the relevant network or advisory group.

Analysis

We will use an interpretative description approach to collecting and analysing the

data.25 This approach borrows from grounded theory, naturalistic inquiry, and

ethnography. It differs from these approaches in that the investigators are looking

for findings with practical applications. Various verification strategies, such as

concurrent data collection and analysis, constant comparative analysis and iterative

analysis, serve to locate the findings within the framework of the existing body of

knowledge. The object of the exercise is a coherent conceptual description that can

inform effective strategies for engaging stakeholders in this and other research. We

will also present and discuss findings and interpretation of the findings with the

networks and advisory groups iteratively. We will record and report any

discrepancies between how findings are interpreted by the research team and other

stakeholders, any disagreements that have not been resolved by discussion within

each group, and differences in interpretation among groups or countries.

Interpretation and reporting of findings

The networks and advisory groups will be engaged in interpreting and

confirming interpretations of the findings. They will be invited to comment on

reports and be co-authors.

We will seek approval of the networks and advisory groups of a final report of

the evaluation of stakeholder engagement in this project.

When reporting the results, we will use the GRIPP2 Checklist for reporting

patient and public involvement in health and social care research.26


Ethical considerations

Ethics approval has been sought from local institutional review boards and

national research governing councils in each country. We have obtained ethics

approval from the Rwanda National Ethics Committee; approval number

00001973 and the Kenyan National Commission for Science, Technology and

Innovation; approval number NACOSTI/P119/1986. We are currently awaiting

ethics approval from the Uganda National Council of Science and Technology

having already submitted an application. Data collection will commence once

the study has been approved by all the participating sites.

We will obtain informed consent for all interviews. Data from questionnaires,

interviews, photos, videos and notes from discussions will be anonymised.

Teachers, students and policymakers who are invited to participate in any of the

networks will be informed of the purpose of their participation before written

permission is sought. The students who participate on the networks and user-

testing will be given information about the project to take to their parents,

where written permission will be obtained both from the parents in form of

consent and from the students in form of assent.

Consent for the students to participate in the pilot will be given by the

headteachers and teachers and students and their parents will have the same

right to refuse participation in piloting the resources as they do for any other

learning resources used in schools. Although this will be limited, teachers,

students and policymakers, whenever required to attend meetings off-site of

their professional premises (schools, ministry offices etc) will be compensated

for their transport expenses.

Choice of methods, risks and risk management

Although there are strong arguments for engaging stakeholders in research,

there is a paucity of research comparing different ways of doing this.9 By having

explicit goals for engaging stakeholders and monitoring the extent to which

those goals are being achieved, we will be able to make adjustments as needed

throughout the project. Our decision to engage teachers and students through a

network in each country is based on our prior experience with a teacher

network.2 We have prioritised engaging teachers and students because they are

the intended users of the resources that we will develop.

We will consider and discuss important risks of stakeholder engagement with

stakeholders, to plan how we will manage those risks, including potential waste

of resources, loss of credibility, conflicts, and adverse effects on the design,

evaluation, or dissemination of the learning resources.


List of abbreviations

IHC Informed Health Choices Project

Consent to publish

Not applicable.

Availablity of data and materials

The data files resulting from the stakeholder engagement processes will be

made available at the Norwegian Centre for Research Data

(http://www.nsd.uib.no/nsd/english/index.html).

Competing interests

All the authors declare that no competing interests exist.

Funding

This work will be funded by the Research Council of Norway

(https://www.forskningsradet.no/en/), Project number 284683, grant

no:69006 awarded to ADO. The funder will play no role in the study design, data

collection, data analysis, data interpretation, or writing of the report. The

principal investigator will have access to all the data in the study and have final

responsibility for the decision to submit for publication.

Authors’ contributions

All the investigators reviewed the protocol, provided input, and agreed on this

version.

http://www.nsd.uib.no/nsd/english/index.html

https://www.forskningsradet.no/en/


References

1. Nsangi A, Semakula D, Oxman AD, Austvoll-Dahlgren A, Oxman M, Rosenbaum S, et

al. Effects of the Informed Health Choices primary school intervention on the

ability of children in Uganda to assess the reliability of claims about treatment

effects: a cluster-randomised controlled trial. Lancet 2017;17:31226-6.

2. Nsangi A, Semakula D, Rosenbaum S, Oxman AD, Oxman M, Morelli A, et al.

Development of the Informed Health Choices resources in four countries to teach

primary school children to assess claims about treatment effects. Pilot and

Feasibility Studies 2020; 6:18..

3. Hillstrom K, Hillstrom LC, (Editors). Brainstorming. In: Encyclopedia of small

business, Farmington Hills: Gale Cengage 2002;1.

4. Deverka PA, Lavallee DC, Desai PJ, Esmail LC, Ramsey SD, Veenstra DL, et al.

Stakeholder participation in comparative effectiveness research: defining a

framework for effective engagement. Journal of comparative effectiveness research

2012;1(2):181-94.

5. Ray KN, Miller E. Strengthening stakeholder-engaged research and research on

stakeholder engagement. Journal of comparative effectiveness research

2017;6(4):375-89.

6. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in

research: moving from theory to practice. . J. Comp. Eff. Res. 2015;4(2):133-45.

7. Concannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, et al. A

systematic review of stakeholder engagement in comparative effectiveness and

patient-centered outcomes research. J Gen Intern Med 2014;29(12):1692-701.

8. Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al.

Mapping the impact of patient and public involvement on health and social care

research: a systematic review. Health Expectations 2014;17(5):637-50.

9. Nilsen ES, Myrhaug HT, Johansen M, et al. Methods of consumer involvement in

developing healthcare policy and research, clinical practice guidelines and patient

information material. . Cochrane Database Sys Rev 2006;CD004563

10. Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, et al. Stakeholder

involvement in systematic reviews: a protocol for a systematic review of methods,

outcomes and effects. Res Involv Engage. 2017;3(9).

11. Bowen DJ, Hyams T, Goodman M, West KM, Harris-Wai J, Yu J-H. Systematic Review

of quantitative measures of stakeholder engagement. Clinical Trials Sci

2017;10:314-36.

12. Harrison JD, Auerbach AD, Anderson W, Fagan M, Carrie M, Hanson C ea. Patient

stakeholder engagement in research: A narrative review to describe foundational

principles and best practice activities. Health Expectation 2019;22:307-16.

13. Rickinson M, Sebba J, Edwards A. Improving research through user engagement. .

Oxon, UK: Routledge 2011.


14. Haywood K, Brett J, Salek S, Marlett N, Penman C, Shklarov S, et al. Patient and

public engagement in health-related quality of life and patient-reported outcomes

research: what is important and why should we care? Findings from the first

ISOQOL patient engagement symposium. Qual Life Res 2015;24:1069-76.

15. NIHR INVOLVE. What is public involvement in research?

https://www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/. .

16. International Association for Public Participation. IAP2 Public Participation

Spectrum. International Association for Public Participation,.

https://cdn.ymaws.com/www.iap2.org/resource/resmgr/pillars/Spectrum_8.5x1

1_Print.pdf 2018.

17. Bruns B. Water Tenure Reform: Developing an Extended Ladder of Participation.

Presented at: Politics of the Commons: Articulating Development and

Strengthening Local Practices, . Chiang Mai, Thailand,

http://dlc.dlib.indiana.edu/dlc/handle/10535/453 2003;July 11-14.

18. INVOLVE. People & Participation: How to put citizens at the heart of decision-

making. London: Involve,

https://www.involve.org.uk/resources/publications/practical-guidance/people-

and-participation 2005.

19. The SURE Collaboration. Informing and engaging stakeholders. In: SURE Guides for

Preparing and Using Evidence-Based Policy Briefs. . Geneva: World Health

Organization,https://www.who.int/evidence/sure/guides/en/ 2011;Version

2.1(https://www.who.int/evidence/sure/guides/en/).

20. Hahn DL, Hoffmann AE, Felzien M, LeMaster JW, Xu J, Fagnan LJ. Tokenism in

patient engagement. Family Practice 2016;34(3):290-95.

21. Oxman AD, Chalmers I, Austvoll-Dahlgren A. Key Concepts for assessing claims

about treatment effects and making well-informed treatment choices [version 2;

peer review: 3 approved]. F1000Research, https://f1000research.com/articles/7-

1784/v2 2019;7(1784).

22. IDEO.org. The Field Guide to Human-Centered Design. San Francisco: IDEO.org,

2015;https://www.designkit.org/resources/1

23. Brainstorming. Encyclopedia of Small Business,. Farmington Hills: Gale Cengage,

2019; https://www.encyclopedia.com/entrepreneurs/encyclopedias-almanacs-

transcripts-and-maps/brainstorming.

24. Informed Health Choices Group. Informed Health Choices Newsletter.

https://www.informedhealthchoices.org/wp-content/uploads/2019/04/IHC-

Newsletter-2019.pdf 2019.

25. Thorne S, Kirkham SR, O’Flynn-Magee K. The analytic challenge in interpretive

description. Int J Qual Methods 2004;3:1-11.

26. Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2

reporting checklists: tools to improve reporting of patient and public involvement

in research. . BMJ; Research Involvement and Engagement. 2017;358:j3453(3):13.

http://www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/

http://www.iap2.org/resource/resmgr/pillars/Spectrum_8.5x11_Print.pdf

http://www.iap2.org/resource/resmgr/pillars/Spectrum_8.5x11_Print.pdf

http://dlc.dlib.indiana.edu/dlc/handle/10535/453

http://www.involve.org.uk/resources/publications/practical-guidance/people-and-participation

http://www.involve.org.uk/resources/publications/practical-guidance/people-and-participation

http://www.who.int/evidence/sure/guides/en/

http://www.who.int/evidence/sure/guides/en/)

http://www.designkit.org/resources/1

http://www.encyclopedia.com/entrepreneurs/encyclopedias-almanacs-transcripts-and-maps/brainstorming

http://www.encyclopedia.com/entrepreneurs/encyclopedias-almanacs-transcripts-and-maps/brainstorming

http://www.informedhealthchoices.org/wp-content/uploads/2019/04/IHC-Newsletter-2019.pdf

http://www.informedhealthchoices.org/wp-content/uploads/2019/04/IHC-Newsletter-2019.pdf

Protocol for evaluating stakeholder engagement in the ... · 5/25/2020 · engagement in the...

Documents

Transcript of Protocol for evaluating stakeholder engagement in the ... · 5/25/2020 · engagement in the...