Running head: EVALUATION PLAN 1

Brett Stachler & Kathryn Ward-Mytinger

Loyola University Chicago

EVALUATION PLAN 2

Contents Theoretical Framework ................................................................................................................... 4!History of Global Health Resident Program ................................................................................... 5!Global Health Certificate Details .................................................................................................... 7!

Goals and Objectives .................................................................................................................. 7!Goals, Objectives, and Components ........................................................................................... 8!Potential Limitations ................................................................................................................... 9!

Logic Model .................................................................................................................................. 10!Inputs ........................................................................................................................................ 10!Outputs ...................................................................................................................................... 11!Outcomes .................................................................................................................................. 11!Assumptions .............................................................................................................................. 13!External Factors ........................................................................................................................ 14!

Evaluation Approach .................................................................................................................... 14!Evaluation Questions and Focus ................................................................................................... 15!Evaluation Rationale ..................................................................................................................... 15!Quantitative Evaluation Approach ................................................................................................ 16!

Participants ................................................................................................................................ 17!Research Design ....................................................................................................................... 17!Survey Instrument ..................................................................................................................... 18!Implementation Plan ................................................................................................................. 20!Statistical Analyses ................................................................................................................... 22!Quantitative Results Presentation ............................................................................................. 23!

Quantitative Evaluation Approach ................................................................................................ 23!Participant Demographics ......................................................................................................... 24!Instrumentation and Validity .................................................................................................... 25!Implementation ......................................................................................................................... 26!Analysis .................................................................................................................................... 27!

Display Results ............................................................................................................................. 29!Next Steps ..................................................................................................................................... 29!References ..................................................................................................................................... 30!Appendix A: Northwestern Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Heath Logic Model ..................................................................................... 32!Appendix B: McGaw Global Health Curriculum ......................................................................... 34!Appendix C: McGaw Graduate Medical Education Certificate of Global Health Survey Construct Map and Item Matrix for PRE-TEST ........................................................................... 35!Appendix D: McGaw Graduate Medical Education Certificate of Global Health Survey Construct Map and Item Matrix for POST-TEST ........................................................................ 42!Appendix E: E-mail Draft of Pre-Test Survey .............................................................................. 46!Appendix F: Pre-Test Survey ........................................................................................................ 47!

EVALUATION PLAN 3

Appendix G: E-mail Draft of Post-Test Survey ............................................................................ 53!Appendix H: Post-Test Survey ..................................................................................................... 54!Appendix I: Pre-Test Resident Survey Email Reminder .............................................................. 60!Appendix J: Post-Test Resident Survey Email Reminder ............................................................ 61!Appendix K: General Survey Reminder Email ............................................................................ 62!Appendix L: Resident Semi-Structured Interview Protocol ......................................................... 63!Appendix M: Semi-Structured Interview Resident Demographic Questionnaire ........................ 66!Appendix N: Semi-Structured Resident Interview Protocol Summary Notes .............................. 67!Appendix O: Resident Semi-Structured Student Interview E-Mail .............................................. 68!Appendix P: Mentor Semi-Structured Interview Protocol ............................................................ 69!Appendix Q: Semi-Structured Interview Mentor Demographic Questionnaire ........................... 72!Appendix R: Mentor Semi-Structured Interview Protocol Notes ................................................. 73!Appendix S: Mentor Semi-Structured Student Interview E-Mail ................................................. 74!Appendix T: Lecturer Semi-Structured Interview Protocol .......................................................... 75!Appendix V: Lecturer Semi-Structured Interview Protocol Notes ............................................... 79!Appendix W: Lecturer Semi-Structured Lecturer Interview E-Mail ............................................ 80!

EVALUATION PLAN 4

Evaluation Plan for McGaw Global Health Clinical Scholars Certificate Program

The world has become increasingly interconnected and globalization now affects

virtually every person’s life. Increases in the flow of people, products, services, and information

between and among countries and continents are having a dramatic influence on the world’s

health and health care delivery. This evaluation proposal will give an overview of the McGaw

Global Health Clinical Scholars Certificate Program, the reasoning for the inquiry; how the

assessors will approach evaluate the program from both a quantitative and qualitative evaluative

approach, and recommended next steps for key stakeholders. It is our goal to provide a solid

evaluative foundation to this innovative and groundbreaking program.

Theoretical Framework

Delivering care from a global health perspective is necessary for physicians and health

care providers. Researchers in the field contend residents who complete a focused global health

curriculum learn to practice medicine with limited access to laboratory tests and expensive

diagnostic procedures, relying on strengthened physical examination skills and depending less on

laboratory values, radiologic imaging, and other diagnostic testing, and they develop a deeper

appreciation for global public health issues and become more culturally sensitive (Drain,

Holmes, Skeff, Hall, & Gardner, 2009). Within the medical community, the term “global health”

refers to a broad spectrum of health issues encountered by physicians worldwide.

Acknowledgment of its importance in helping health care workers learn to provide quality care

for patients is often overlooked, likely because the definition of global health is sometimes

confused or used interchange- ably with both “international health” and “public health”. This

was addressed by Koplan, Merson, Reddy, Rodriguez, & Sewankambo (2009):

[Global health is] an area for study, research, and practice that places a priority on

EVALUATION PLAN 5

improving health and achieving equity in health for all people world- wide. Global

health emphasizes transnational health issues, determinants, and solutions; involves many

disciplines within and beyond the health sciences and promotes interdisciplinary

collaboration; and is a synthesis of population-based prevention with individual-level

clinical care (p. 1993).

Global health training programs are necessary in graduate medical education given resident

interest across all specialties as well as the globalization of health (Chase & Everret, 2011; Kern,

Thomas, & Hughs, 2009). There is a paucity of such training programs, particularly

competency-driven programs, in the United States (A. Doobay-Persaud, personal

communication, September 2, 2014). In addition, many residents across universities participate

in international rotations that have not been evaluated for safety, educational value, social

responsibility, or accountability (Cherniak, Drain, & Brewer, 2013). Of note, recent surveys

have shown that between 50-60% of residents have chosen their programs based on global health

training opportunities (Drain, et al., 2009).

History of Global Health Resident Program

In early 2014 the Center for Global Health, housed within the world renowned Fienberg

School of Medicine at Northwestern University, in order to answer the call by medical

residences and global need for global health practitioners, took steps to join an emerging group

of elite residency programs that offer comprehensive competency-based Global Health

Certification (A. Doobay-Persaud, personal communication, September 2, 2014). Developed

and managed by Dr. Ashti Doobay-Persaud, the Faculty Development Chair at the Center for

Global Health and Director of the Section of Global Health within the Division of Hospital

Medicine at Northwestern University, program development began in 2013 when Dr. Doobay-

EVALUATION PLAN 6

Persaud invited a small group of medical practitioners dedicated to global health to a round table

to engage in a substantive dialogue on the development of a Global Health curriculum for

medical residents. The dialogue led Dr. Doobay-Persaud to collaborate with the McGaw

Medical Center of Northwestern University, an Illinois not-for-profit corporation that sponsors,

administers, and supports graduate medical education, to assist financially in the development

and implementation of a three-year global health certificate for medical residents of the Fienberg

School of Medicine. While many global health programs are offered at various medical

institutions very few are grounded in the Accreditation Council for Graduate Medical Education

(ACGME) recommended competencies for the training of residents (A. Doobay-Persaud,

personal communication, September 2, 2014). An important tenant of the certificate program is

to focus on building competency essential for practice in under-resourced or disaster-effected

areas at home and abroad.

The process to develop the preliminary certificate curriculum involved two steps: a

review of existing programs and competencies as well as interviewing and assessing needs from

key stakeholders; and an internet search of existing programs generated with the following

terms: health; medicine; global; international; residency; curriculum; competencies; and

education (A. Doobay-Persaud, personal communication, September 2, 2014). Twenty residency

programs were reviewed, and Dr. Doobay-Persaud contacted the directors of the five programs

that had competencies for guidance; four of the five responded (A. Doobay-Persaud, personal

communication, September 2, 2014). In addition to corresponding with peer medical

institutions, Dr. Doobay-Persaud conducted a PubMed literature search limited to the last ten

years, using the internet search terms world health, international educational exchange, program

development, graduate medical education and internship; 448 articles were retrieved, and 33

EVALUATION PLAN 7

were included as they described specific competencies and skills (A. Doobay-Persaud, personal

communication, September 2, 2014). Finally, current expert opinion from the Consortium of

Universities for Global Health on inter-professional global health competencies also guided

program development (A. Doobay-Persaud, personal communication, September 2, 2014).

Utilizing information gathered from peer expertise, research, and existing ACGME competencies

Dr. Doobay-Persaud drafted the preliminary certificate curriculum. On September 2, 2014, the

Dean of Graduate Medical Education, program directors, senior faculty across specialties, the

Center for Global Health and the Division of Hospital Medicine leadership, as well as

development and administrative professionals, met to review the preliminary curriculum for

approval.

Global Health Certificate Details

The meeting on September 2, 2014 produced a robust and comprehensive curriculum

plan for the certificate program officially entitled, the McGaw Graduate Medical Education

Certificate of Global Health. The certificate grounded in ACGME competencies will require

interested residents to complete eight certificate components over a two-year period.

Goals and Objectives

The components combine to meet the goal initially established by Dr. Doobay-Persaud,

to create a global health certificate program with an educational curriculum for residents and

fellows that will train them in clinical care delivery in resource-limited settings worldwide. The

concept of the global health education began within the Feinberg School of Medicine as a

combination of core concepts such as high-quality global health practicums, clinical experiences

with mentorships and evaluations, and a scholarly project. However, the September 2, 2014

meeting led to several additions, requests, and changes that have been approved to better support

EVALUATION PLAN 8

the goals and objectives of the program. Program directors requested, in acknowledgement of

limited time and schedule diversity of residents, the didactic elements be made available online

(A. Doobay-Persaud, personal communication, September 22, 2014). In addition, senior faculty

specialists introduced several new lecture topics they believed critical to a well-rounded global

health education. Furthermore, each program director and faculty member present committed to

leveraging global health clinic partnerships to provide structured ACGME-based global

experiences. Directors and faculty members also committed to creating relationships with onsite

mentors for residents when completing the global clinic visit component of the program (A.

Doobay-Persaud, personal communication, September 22, 2014). At the conclusion of the

September 2, 2014, after comprehensive debate and cross-specialty dialogue, the goals,

objectives, and components of the McGaw Graduate Medical Education Certificate of Global

Health were established.

Components

The McGaw Graduate Medical Education Certificate of Global Health requires Fienberg

School of Medicine residents to complete eight components within two years to attain a

certificate of global health. The eight components are: nine core lectures, five specialty-specific

lectures, participate in a quarterly journal club, seek faculty mentorship, utilize web-based

training modules, contribute to cross-residency skills exchange, travel to global health clinic for

skills training and exchange, and produce and present scholarly project upon return from their

global experience (See Appendix B). These components were created to meet the following

global health competencies: describe global burden of disease and demonstrate understanding of

epidemiologic tools and methods; understand health implications of travel, migration, and trade;

recognize the major determinants of health; explain the role of community-engagement strategies

EVALUATION PLAN 9

in capacity strengthening; demonstrate high standards of ethical conduct and quality in global

healthcare; practice and professionalism in a resource-limited setting: provide culturally sensitive

care and support to patients; demonstrate a basic understanding of the relationship between

health and human rights; develop global health focused scholarly work online, currently, there is

not a universal assessment tool to ensure standards are met across programs, it is dependent upon

the program to individually assess each resident prior to certification. (A. Doobay-Persaud,

personal communication, September 22, 2014). The components and corresponding

competencies are ambitious, however Dr. Doobay-Persaud believes the breadth and depth of the

certificate will attract residents eager to expand their medical knowledge, improve patient care,

and increase their understanding of global need (See Appendix B).

Key Stakeholders

Along with Dr. Doobay-Persaud many within the Fienberg School of Medicine and the

McGaw Medical Graduate program are dedicated to the creation and implementation of this

certificate program. Deans of both McGaw and Fienberg have committed support through

funding, promotion, and authoring Program Directors to allow residents time to participate in a

global site rotation. In addition to both schools, the Center for Global Health at Fienberg School

of Medicine has committed funding, physical office and meeting space, web-hosting and

recording of lectures, and faculty. The McGaw Global Health Resident Certificate program is a

new program with significant support across the diverse medical community of Northwestern

University.

Potential Limitations

As a new program, Dr. Doobay-Persaud and key stakeholders shared concerns in regards

to the next steps of resident recruitment and program implementation. The primary concern is

EVALUATION PLAN 10

certificate funding, which will affect both the recruitment of residents as well as implementation

of the certificate program. As of September 25, 2014 multiple departments, including McGaw

Graduate Medical Education Organization, the Center for Global Health, and the Fienberg

School of Medicine have verbally committed to financial support of both the administrative,

faculty, and resident costs of the program online (A. Doobay-Persaud, personal communication,

September 28, 2014). However, no funds have been received. Additionally, a program funded

by multiple entities will be subject to a complex set of financial and administrative oversight that

will add a layer of complexity to the evaluative goals and process.

Logic Model

The McGaw Global Health Resident Certificate program logic model describes the

resources and planning before the classes begin the role of residents, faculty, and staff while the

program is in session, and the outcomes from a short-to long-term timeline based on residents’

participation in the program (Wholey, Hatry, & Newcomer, 2010). For a visual graph of the

logic model, please refer to Appendix A.

Inputs

The inputs describe the research, planning, and supplies needed before the program is

able to begin and while it is in session. As described earlier, the individuals involved in the

program begin with the staff members who have previously evaluated the needs for the McGaw

Global Health Resident Certificate program. They have also constructed the curriculum for the

faculty members who will serve as lecturers and mentors for the residents enrolled in the McGaw

Global Health Resident Certificate program. Finances have been verbally committed from

various Feinberg School of Medicine entities to cover the staff creating and planning the

program, as well as the faculty who will be providing the lectures and mentorship to residents.

EVALUATION PLAN 11

With the lectures of the program in the evening after residency hours during the day, staff will be

providing dinners for the residents as an incentive to enroll in the program. Finances are

provided for various items related to residents visiting their global health sites. Time and

materials relate to the planning and implementation of the McGaw Global Health Resident

Certificate program. Research, a needs evaluation, and development of the program came from

various stakeholders within the Feinberg School of Medicine, with stakeholders now marketing

and enrolling residents to begin the program in 2015.

Outputs

The courses will be taught in-class by faculty members unless residents are physically

located in a space they cannot leave due to conflicts with their residency status (i.e., Family

Medicine). The residents enrolled in the program will complete 9 of the 12 core curriculum

lectures, and 5 of the 6 specialty lectures, participate in a quarterly journal club, and engage in

various scholarly and professional development projects. The global health sites will be in a

variety of different locations. Medical insurance will be provided to residents, and global health

sites will have medical practitioners and faculty providing mentorship. Prior to departing to

global health sites, residents will participate in a mandatory pre-departure training.

Outcomes

Outcomes are defined as the results of residents engaging and participating in the McGaw

Global Health Resident Certificate program during their participation, upon completion, and in

years after completing (Wholey, et al., 2010). The short-term goals focus on the experiences of

residents during the three years of engagement in the program. Residents will begin to

understand the global context and inequities of health and medicine, in both their lectures and in

global health visits to various countries. In discovery of other health and medical contexts,

EVALUATION PLAN 12

residents will discover different sociopolitical contexts in the countries they study in, which will

highlight different socio-economic restraints and inequalities and political contexts that may

reject or find western health and medicine hostile. Residents will begin to examine health and

medicine outside the paradigm of western medicine, further identified as traditional medicine

(not recognized by the Accreditation Council for Graduate Medical Education). Residents are

also exposed to different ethical dilemmas related to global health and medicine, and are offered

a lecture to explore these concepts further. During all of these various experiences and

engagement, residents will also develop professional skills such as creating scholarly works, and

presenting on various topics. Residents will also be exposed to different careers in global health,

different mentors, and connections to supplement their experiences during and after the McGaw

Global Health Resident Certificate program.

Medium-term outcomes are defined as the results of residents completing the two-year

program upon time of completion into their professional positions as fellows or licensed medical

practitioners. Residents will be able to define various global health contexts and inequities, and

use their knowledge in practice with a high level of ethical medical care. Residents will in

addition complete the program with knowledge of systemic structures and policies that shape

healthcare and medicine to where graduates can work within and shift those structures when

necessary. Residents will also gain knowledge and know how to practice medicine within the

community and cultural context, and know how to operate healthcare under financial restrictions.

As the capstone to the certificate program, residents will engage in scholarly work related to

global health. In relation to long-term outcomes, graduates will contribute to the medical

community with the knowledge and experience of global heath differences and inequity.

EVALUATION PLAN 13

Graduates are expected to continue to strive for global health equity in their various capacities

and scholarly work.

Assumptions

The evaluators must carefully consider the assumptions made of the residents, faculty,

and staff involved in the McGaw Global Health Resident Certificate program, which may

influence the accuracy of the survey, and the results outlined from the evaluation (Wholey, et al.,

2010). Considering the McGaw Global Health Resident Certificate has input from various

stakeholders throughout the Feinberg School of Medicine, and is in its infancy of being

implemented, the inputs and outputs of the program must be accounted for as assumptions. Prior

knowledge of a programs’ development is one of the key factors in understanding a program’s

inputs, outputs, assumptions, and external factors (Weiss, 1998). The understanding of the

program may change as it developed, and must be continuously monitored for the evaluation to

be effective (Weiss, 1998). Residents are currently being enrolled, but we are under the

assumption they are interested in the program, can attend the on-campus lectures, participate in

the global sites, and complete the program in three years. Assumptions are also made regarding

the program curriculum and process. We do not know if the lectures, mentorship, global site

visits, and other components of the program will meet the outcomes outlined by McGaw Global

Health Resident Certificate faculty and staff, and the Accreditation Council for Graduate

Medical Education (ACGME). Some assumptions under this umbrella to consider are that: the

ACGME can be taught by lectures and understood by residents, whether the global site mentors

have training in ACGME outcomes, and whether ACGME outcomes are desirable outcomes

outside of the United States medical context.

EVALUATION PLAN 14

External Factors

The evaluators must consider external factors that can alter the intended outcomes of the

McGaw Global Health Resident Certificate program. Among the most fundamental of external

factors is the Feinberg School of Medicine continuing to enroll residents, which are needed for

the McGaw Global Health Resident Certificate to exist. Program stakeholders have prepared for

this reality, but have already seen progressive enrollment as off December 2014. Program

funding must also continue and key stakeholders for the program must remain in their positions

for the program to meet the outcomes for students. Within the context of the global sites,

communication between faculty and staff of Feinberg and the global health sites must be strong

to ensure the sites are meeting ACGME outcomes. It is difficult to control the geopolitical and

sociopolitical contexts at the global sites, which will factor resources at the sites, the type of

health and medical experiences residents will have, and the overall safety of the site and resident.

It will be difficult to control external factors once the residents graduate, which may challenge

long-term goals specifically.

Evaluation Approach

Our approach will be evaluating the outcomes of the McGaw Global Health Resident

Certificate by focusing on the program in its infancy, constituting a formative approach, which

seeks to provide evaluative information for the purpose of program improvement (Fitzpatrick,

Sanders, & Worthen, 2011). Enrollment for winter 2015 is underway and has multiple residents

enrolled. Staff and faculty facilitated a needs assessment for the creation of the McGaw Global

Health Resident Certificate program supplementing the creation of learning strategies used in the

programs’ curriculum. Unique to this global health program is the desire to meet ACGME

EVALUATION PLAN 15

standards, which are often not met in graduate medical global health programs (A. Doobay-

Persaud, personal communication, September 22, 2014).

Evaluation Questions and Focus

These same outcomes created by program faculty and staff, under the standards of

ACGME will be under evaluation by the authors. The overarching themes guiding our

evaluation stem from the educational strategies and outcomes the McGaw Global Health

Resident Certificate program anticipates for students who finish their certificate. Will students

describe global burden of disease and demonstrate understanding of epidemiologic tools and

methods? Will they understand health implications of travel, migration, and trade? Will they

recognize the major determinants of health? Will they understand the role of community-

engagement strategies in the context of healthcare? Will they demonstrate high standards of

ethical conduct and quality in global healthcare? Will they practice professionalism in a resource

limited setting by providing culturally sensitive care, support to patients, and understanding the

role of traditional medicines? Will they demonstrate a basic understanding of the relationship

between health and human rights? Will they develop global health focused scholarly work?

These outcomes will guide our approach in evaluating the McGaw Global Health Resident

Certificate program.

Evaluation Rationale

The program outcomes-orientated approach will allow the authors to evaluate the

specified outcomes and purposes of the McGaw Global Health Resident Certificate program to

focus on the extent to which the outcomes and purposes are achieved (Fitzpatrick, et al., 2011).

Using quantitative and qualitative measurements, we will attempt to measure the developmental

changes residents undergo related to the outcomes created by the McGaw Global Health

EVALUATION PLAN 16

Resident Certificate program. We also hope to measure students retaining the short-term

outcomes and purposes, thus leading to medium- and long-term stated outcomes measured in

future evaluations. Because this program has been established and is ready to be measured for

its outcomes for the first time, our assessment will be formative by measuring the program in its

progress, rather than assessing whether or not the program should be continued (Fitzpatrick, et

al., 2011).

One of the two evaluators has extensive knowledge of the program, although a

developing context of global health programs outside of Feinberg. Although familiarity with the

program is a strength for the evaluation plan, it has the potential to cause bias in contributing to

the final product. Additionally, global heath contexts will be needed from stakeholders.

Although the formative evaluation seeks to define the context for programs, the McGaw Global

Health Resident Certificate program will continue to develop and change due to a variety of

external factors (Fitzpatrick, et al., 2011). The evaluators will continue to evaluate these external

factors, and recognize changes with the McGaw Global Health Resident Certificate program.

Quantitative Evaluation Approach

This section will detail the implementation of quantitative measurements for our

evaluation of the McGaw Graduate Medical Education Certificate of Global Health, including

our research design, statistical analysis, survey instrument, and implementation plan. The logic

model contextualizes the quantitative approach and methodology by seeking to understand short,

medium, and long-term goals of the McGaw Graduate Medical Education Certificate of Global

Health.

EVALUATION PLAN 17

Participants

Fienberg School of Medicine residents in the OBGYN, Dermatology, Family Medicine,

General Medicine, and Surgery programs who enroll in the McGaw Graduate Medical Education

Certificate of Global Health will be the focus of this evaluation. The McGaw Graduate Medical

Education Certificate of Global Health is a new program; therefore, enrollment projections for

the inaugural winter 2015 quarter are unknown. The sampling will be census, meaning the entire

portion of the evaluative population, inviting all students enrolled in the McGaw Graduate

Medical Education Certificate of Global Health to complete the pre- and post- test surveys as a

proposed requirement for receiving their certificate, pre-Institutional Review Board approval

(Schuh, 2009).

Research Design

The most effective quantitative measure to assess the outcomes of the McGaw Graduate

Medical Education Certificate of Global Health is a pre-experiential longitudinal study using a

pre-test and post-test survey (Creswell, 2009). The McGaw Graduate Medical Education

Certificate of Global Health is interpreted as an intervention, as it is a unique program with a set

of outcomes not currently present in the Feinberg School of Medicine. The quantitative

measurements will be used to measure the direction of change, the amount of change that

occurred, and the amount of consistency in which the resident changed by using a set of Likert

scale questions (Creswell, 2009). The quantitative portion of the longitudinal study will begin

with residents taking the pre-test before their lectures begin in their first quarter, and end with

residents completing the post-test upon completion of the McGaw Graduate Medical Education

Certificate of Global Health curriculum. As mentioned above, the type of sampling used for the

longitudinal study will be census, inviting all residents enrolled to participate (Wholey, et al.,

EVALUATION PLAN 18

2010). Participation in the quantitative portion of the evaluation is being proposed as required to

the IRB for students to receive their certificate, ensuring near a 100% response rate, limiting the

internal threat of mortality (Creswell, 2009). Though census sampling will provide a holistic

context of the residents in the McGaw Graduate Medical Education Certificate of Global Health,

the small sample size of approximately 5-7 students per quarter will also adversely affect the

validity of the findings (Wholey, et al., 2010).

The pre-experiential nature of the quantitative design will measure the single group of

residents participating in the McGaw Graduate Medical Education Certificate of Global Health,

without a control group to measure, classified as a one-group pre-test post-test design (Creswell,

2009). A control group will not be utilized for this evaluation, limiting validity that is offered in

quasi- and true experimental design (Creswell, 2009). However, a control group and random

selection are not feasible in this context, since the material and time, resources do not exist to

administer a control group, and the population to survey is not large enough for random selection.

The survey questions were constructed using a survey construct matrix to link survey questions

to the short-, medium-, and long-term outcomes detailed in the logic model of the McGaw

Graduate Medical Education Certificate of Global Health (Appendix A).

Survey Instrument

The McGaw Graduate Medical Education Certificate of Global Health program, designed

to be completed in a semi-linear manner, will result in the sample of residents completing the

program at various times throughout the calendar year. Therefore, the ideal evaluation

instrument is an online survey to be emailed upon the submission of their request for certificate

completion (Appendix E; Creswell, 2009). To account for non-sequential completion, an online

survey will also attend to variation of schedules and limited time medical residents cite as

EVALUATION PLAN 19

roadblocks to completing similar surveys. Primary stakeholder, Dr. Doobay-Persaud has agreed

the completion of the survey be a condition of receipt of the official certificate of completion

(written communication, 2014). The required completion of the online survey therefore is a

feasible option for assessing the McGaw Graduate Medical Education Certificate of Global

Health outcomes.

The McGaw Graduate Medical Education Certificate of Global Health program will

enroll the first class of residents in November of 2014, to begin in January of 2015; therefore, it

is essential the survey be designed specifically for this program to assess desired outcomes.

However, while the program may be in infancy, program assessment surveys have become

standard throughout higher education. As such, many questions have been borrowed directly or

modified slightly from existing survey questions. The sources from which these questions were

drawn include: Center for Global Education at the University of California, Los Angeles SOAR

Survey (2012), Center for Experiential Learning at Loyola University Chicago Survey (2013),

and Central Connecticut State University Center for International Education Study Abroad

Survey (2013).

The development of the survey began with a request from key stakeholders to assess if

the certificate program is a successful intervention in expanding residents’ global health

awareness and knowledge in tandem with increasing medical competency and patient care. To

meet this request a survey construct map (Appendix A) allow for the ordering of questions and

ease of relation to the program outcomes stated in the logic model. The survey construct map

(Appendix C) includes both pre and post – test survey questions; the questions are intended to

measure if residents who completed the certificate program met the intended outcomes. The pre-

test survey begins by requiring the student identification number so the pre and post-test surveys

EVALUATION PLAN 20

may be compared. In addition, several general questions ascertain the residents’ previous

experience abroad to control for prior knowledge or experience. The assessment tool is intended

to assess resident knowledge, aptitude, and perception of global healthcare prior to the

completion of the program. The pre-test consists of 16 questions divided into seven distinct

groups.

Of the seven groups, three groups: Skill Development, Global Health Awareness, and

Professionalism addressed short-term outcomes with a four-option Likert scale. For continuity, a

four-option Likert scale is standard throughout the pre and post-test surveys. While Wholey, et

al. (2010), recommends five-option Likert scale, this survey deviates from that recommendation

to eliminate the middle choice, to encourage deep assessment of global health experience and

awareness. In addition, the exclusion of a neutral option in this initial assessment may give a

clearer idea of effectiveness of program components. The pre-test survey continues with

questions designed to assess the residents’ familiarity with the core concepts of the certificate

program. The medium outcomes are where the greatest change between pre and post-test are

anticipated. Finally, the pre-test survey concludes with a series of questions that seek to ascertain

general perception of the necessity and impact of global health. Similar to the pre-test, the post-

test survey will include all questions except demographic information, as we may link the pre

and post-test via the student number. In addition to the pre-test questions, additional process goal

questions have been included to assess additional aspects of the program.

Implementation Plan

In order to attend to the inaugural implementation of the program and survey, a pilot test

will be conducted to improve clarity of survey questions as well as create a baseline for validity

(Creswell, 2009). The pilot test, of the pre-test survey, will be administered to a convenience

EVALUATION PLAN 21

sample of current third year residents (not eligible for two year program) associated with the

Center for Global Health within the Fienberg School of Medicine. Once the pilot test and

feedback from those surveyed have been collected, reviewed, and assessment instrument refined

the pre-test survey will be emailed to the participating resident upon acceptance to the Global

Healthcare Residence Certificate Program. The email will communicate the estimated time

needed to complete the survey and require the resident to complete the survey within one week;

with email reminders occurring each day the survey remains incomplete. If incomplete after two

business days, the resident will be contacted via phone and email by the certificate program

director, Dr. Doobay-Persaud (written communication, 2014).

To complete the survey loop, the post-test mirrors the pre-test in distribution mode. The

post-test assessment will be conducted via online survey. The survey link will be emailed to the

resident upon the approval of certificate completion. However, to ensure survey completion the

email will communicate the survey must be completed within one week. If incomplete after one

week, the resident will be contacted via phone and email by the program assistant to make sure

email was received and encourage completion of survey (written communication, 2014). It is the

stakeholder’s hope that through verbal and written reminders to complete the survey, along with

the dedication of residents to Global Health the survey will have a high completion rate, at this

time the stakeholders are unwilling to provide additional incentives (written communication,

Doobay-Persaud, 2014). Finally, upon receipt of both the pre and post-test the residents will

receive an automated email thanking them for their time and invite them to provide additional

feedback via email or in-person meeting (written communication, Doobay-Persaud, 2014).

EVALUATION PLAN 22

Statistical Analyses

The statistical analyses will help evaluate and understand the direction, amount, and

consistency residents changed during the McGaw Graduate Medical Education Certificate of

Global Health. SPSS will be used to conduct measures of central tendency, a paired sample t-

test, and an ANOVA analysis, to evaluate the previous aspects mentioned. The null hypothesis

we seek to reject in the ANOVA is that there are no mean differences between the different

groups we seek to test, such as time 1 and time 2 differences in global health awareness, beyond

a statistical significance.

The descriptive statistics will describe the pre- and post-test changes between the mean,

median, and mode from the Likert scale questions asked in the survey, providing general patterns

(Wholey, et al., 2010). Central tendency measures will also provide a context of data in

relationship with demographics of the residents such as race/ethnicity, gender, sexual orientation,

program year, intended professional practice, and international experience by comparing means,

medians, and modes between the pre-test and post-test. Cross tabulations and chi-square test will

determine if these categorical variables differed across demographic categories. These analyses

will provide a basic understanding of residents’ development throughout the program.

A paired sample t-test will help us evaluate the pre- and post-test results by comparing

data between the pre- and post-test. This will allow us to understand if significant changes

happened across different groups, and across different dependent variables (our different

outcomes measured in the Likert scale questions). An ANOVA test will expand our argument by

seeking to invalidate the null hypothesis depending on whether the results establish a threshold

for statistical significance (Wholey, et al., 2010). The ANOVA will also allow us to compare

scores across three or four groups, provide deeper analysis by determining if there is statistical

EVALUATION PLAN 23

significance between three or more different identities in the goals and outcomes measured in the

Likert scale. Regression tests will be used to determine the variance in any statistically

significant results.

Quantitative Results Presentation

The results of the assessment are essential to the development, improvement, funding,

and continuation of the certificate program, as many stakeholders appear to be data driven. Key

stakeholders anticipate presenting the results through professional presentations that include

poster presentation, global health panels, and the annual Consortium of Universities for Global

Health in 2017. In addition, several stakeholders have expressed interest in authoring a peer-

reviewed paper on the success of Global Health programs for residents. Finally, the results will

be used to improve resident experience, streamline program operations, and increase funding for

additional global site experiences.

Quantitative Evaluation Approach

This section will outline the quantitative approach to the evaluation of the McGaw Global

Health Resident Certificate Program to add depth and meaning to the evaluation results by using

methods that utilize in-depth interviews (Schuh, 2009). For the quantitative approach to the

evaluation, semi-structured interviews will be utilized with residents, lecturers, and resident

mentors. Semi-structured interviews will give further meaning and depth to the sample provided

in the qualitative results, and increases the sample size by gathering a larger amount of

information from a small group of individuals (Schuh, 2009). Semi-structured will be also be

utilized, as opposed to focus groups, due to the vast differences in experiences, schedules, and

development we foresee interviewing between three different populations comprising the

McGaw Global Health Resident Certificate Program. The Semi-structured interviews will

EVALUATION PLAN 24

expand on the questions asked in the quantitative survey, deepen their reflection on their

experiences in the program, and provide the interviewers a chance to provide deeper

explanations related to the qualitative findings. The questions will be orientated around the logic

model outcomes, particularly the short- and medium-term outcomes.

Participant Demographics

All of the medical resident participants for the interview will be gathered from the

surveyed participants. The last question of our survey asks participants to disclose their e-mail

address if they are interested in participating in the interview process. For the lecturers and

mentors, we will email (See Appendices S & W), inviting them to participate. The respondents

will then participate in a semi-structured interview process to capture both the residents story as

well as the mentors and lecturers who the program placed in a position of strong influence on

both the program and participants for more complete assessment (Wholey et al., 2010). It is

important to the key stakeholders that there is at least one resident and mentor from each global

site, to had better evaluate the site as well as the overall program. However, the beneficiary

group, medical residents, is historically difficult to schedule due to rigorous professional and

academic schedules. Therefore, while bias will increase, it is more realistic to first see that

volunteers for the qualitative portion of the program evaluation.

The key stakeholders have indicated they would ideally have at minimum two Resident

and Mentor participants from each global site, as well as, all lecturers and mentors. As this

group will be the first to complete the certificate program, it is essential to capture many

experiences and perspectives. It is especially important to focus on the global site experience

within the qualitative portion, as there will be no standard experience to measure qualitatively, as

each resident will have an experience unique to each global site. In addition to the residents, the

EVALUATION PLAN 25

lecturers and mentors will provide a critical eye as experienced professionals within the field of

global health, which will enable the key stakeholders to improve and grow the program.

Instrumentation and Validity

The protocol instruments (Appendices L, P, & T) that we have created focuses on several

distinct subjects: reasons for enrolling in the McGaw Global Health Resident Certificate Program,

experiences while enrolled such as mentorship and global health site experiences, and areas of

improvement for the program. The protocol seeks to guide the questions for semi-structured

interviews and includes probes in case the participants do not answer the questions with the

depth sought in the questions (Wholey, et al, 2010). The protocol’s focus on reasons for

enrolling seeks to understand how students heard about the program, since it is a new and

developing program. Probes include questions based on different facets of the program weighing

on their decision to enroll such as: medical colleagues, different global health sites, or their

different program directors. One probe also asks if their experiences as a resident have changed

since their enrollment in the McGaw Global Health Resident Certificate Program. The

protocol’s focus on experiences while enrolled asks residents about their experiences while

enrolled generally, experiences with global health sites, and mentorship relationships. Specific

probes ask for more depth information on the three subjects and include: most rewarding and

challenging experiences, and the impacts the latter two pieces of the McGaw Global Health

Resident Certificate Program had on the residents. The last specific protocol focus is

effectiveness and needs and seeks to understand the strengths and weaknesses of the program so

that it can further be developed. Specific probes include asking if any new initiatives would help

develop the program, or how any of the existing pieces of the program could further be

developed.

EVALUATION PLAN 26

Due to strict time constraints of those participating in the semi-structured interviews, the

time of the interviews is to be no longer than 60 minutes. Although the quantitative portion of

the evaluation is a program requirement for residents to complete the certificate, the qualitative

section will not be required for residents. Completion time of the semi-structured interviews

may be impacted if interpretation assistance is needed for residents who do not use English as a

first language.

In order to improve the validity of the interview protocol, members of the McGaw Global

Health Resident Certificate Program, including Dr. Ashti Doobay-Persaud and Kathryn Ward-

Mytinger, will review the materials. By having the protocol reviewed by stakeholders, we hope

to improve the face validity of the qualitative data. Face validity ensures the experts evaluate the

content of the research instrument, and indicate the degree to which the measurement criteria or

objectives are valid (Schuh, 2009). To further develop the accuracy of the qualitative instrument,

we will administer a pilot test with a group of no more than seven students, with the hope of one

student per demographic group.

Implementation

As program director, Ashti Doobay-Persaud has requested that Katie Ward-Mytinger, the

program graduate intern and co-constructor of the survey, conduct the semi-structured interviews.

In addition, it is again important to attend to the complex and saturated schedules of our

interview participants. Therefore, participants who indicated willingness to participate will be

contacted via email (See Appendices O, S, & W) to alert them of their selection, time

commitment of 30 minutes, a request to complete an online form indicating dates and times

available for interview, and the demographic questionnaire. The form will also contain a choice

stating “unavailable for stated interview times,” if the participant checks that box they will be

EVALUATION PLAN 27

prompted to contact the interviewer to set up a time and date outside the offering, possibly via

Skype.

Interviews will take place at the Center for Global Health offices that are conveniently

located within a block of Northwestern Memorial Hospital, where a majority of the participants

work. This space provides both relaxed and formal spaces that have been committed to the

program for interview purposes. Interviewees will have the opportunity to choose the interview

space in an effort to create a comfortable interview space. Those interviewees who are not able

to interview in person will be asked to interview over Skype to best mirror a face-to –face

environment. However, there is a possible bias created in using two different modes of

interviewing. It is our hope that the mode of interview, in-person versus virtual, will not

significantly impact the data collected.

Analysis

The focus group and semi-structured interviews facilitated will be analyzed via

transcription and notes. Participants are aware their responses are being recorded, and particular

themes will require facilitators to take notes. The data will be analyzed by administrators in the

McGaw Global Health Resident Certificate Program, using software for qualitative data analysis

(i.e., NVivo, Atlas.ti).

The methodical approach evaluators will take, once data are collected and organized, is

open coding. By using open coding, organize and prepare the data, divide the data into

categories, determine how categories will be represented, interpret and make meaning of the data

based off the categories, and connect the categories to the qualitative data and assessment

objectives (Schuh, 2009). This will allow us to describe the data using themes, which are then

EVALUATION PLAN 28

discussed and interpreted, backed up by quotes, and put into conversation with similar and

different perspectives (Schuh, 2009).

To assess the validity of the results, we will use several strategies including member

checking, peer debriefing, and triangulation (Schuh, 2009). Member checking is described by

taking the initial findings of the qualitative instruments back to the participants to determine their

accuracy (Schuh, 2009). In order for member checking to feasibly happen, transcriptions and

coding will need to happen quickly after the focus group sessions, so that interviewees can have

a polished view of the data analysis as opposed to the raw transcripts (Creswell, 2009). We will

also utilize peer debriefing by having individuals not familiar with the study, but familiar to the

field of global health, to review our findings and ask questions (Schuh, 2009). To further assess

the validity of our qualitative results, we will triangulate it with our quantitative results, and

continue to use existing results to enrich the data found in future evaluations (Schuh, 2009).

We will discuss limitations in an effort to control for them as much as possible. Personal

bias is possible from the interview process, to coding the data, and describing the data in its

written and final form. Individuals involved with the evaluation will be asked to participate in

trainings, construct expectations, participate in personal reflections, and create strategic plans so

that personal bias can be limited as much as possible. With interviews, personal bias can

interfere as interviewers can ask leading questions and/or close-ended questions, hindering the

responses of the interviewees. Although the evaluators will seek to create rich qualitative data

by using coding and direct quotes, the evaluators will be presented with a large amount of data.

The evaluations will need to reflect and be mindful of inter-rater reliability, especially as the

codes are developed, since data may be interpreted in multiple ways, based on each evaluator’s

bias. A balance must be created between richness and feasibility to ensure the assessment, which

EVALUATION PLAN 29

is orientated towards action as opposed to research, strives for its intention to create meaningful

change (Banta & Blaich, 2010).

Display Results

The final report will be a combination of statistical analyses as well as direct quotations

from the focus group interviews. Direct quotes will serve as a powerful and detailed way to

understand the aspects of the McGaw Global Health Resident Certificate Program that achieved

the intended outcomes as well as aspects that need to be omitted or altered in future years.

Similarly, direct quotes will allow evaluators to make meaning of aspects the residents

experienced that were unanticipated and unaccounted for in the research design. The coding

process and data presentation will allow stakeholders to quickly see the extent to which students

shared experiences that were suggestive of similar themes.

Next Steps

The evaluators will meet with key stakeholders in the coming days for final approval; this

meeting is set for December 4, 2014. Any sort of revisions will be completed before residents

are enrolled in the second week of January. The evaluators will submit the final mock up and

pilot test to begin building the quantitative sections, and to determine if any revisions need to be

made to the qualitative and quantitative questions. The final proposal will include a budget

request, requesting the budget is approved and secured before the evaluation begins. Once

residents are enrolled, they will be asked to complete the pre-test.

EVALUATION PLAN 30

References

Banta, W. T. & Blaich, C. (2010). Closing the assessment loop. Change: The Magazine of

Higher Learning, 43, 22-27.

Central Connecticut State University. (2013). Center for International Education Study Abroad

Survey. Retrieved from:

web.ccsu.edu/uploaded/departments/AdministrativeDepartments/Center_for_Internationa

l_Education/Courses_Abroad/Study_Abroad_Student_Assessment_Form.pdf

Chase, J.A. & Evert, J. (Eds.) (2011). Global health training in graduate medical education: A

guidebook (2nd Ed.). San Francisco, CA: iUniverse, Inc.

Cherniak, W.A., Drain, P.K., & Brewer, T.F. (2013). Educational objectives for international

medical electives: A literature review. Academic Medicine. 88, 1778-1781.

Creswell, J. (2009). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches

(3rd Ed.). SAGE Publication: Thousand Oaks, CA.

Drain, P.K., Holmes, K.K., Skeff, K.M., Hall, T.L., & Gardner, P. (2009). Global health training

and international clinical rotations during residency: Current status, needs, and

opportunities. Acad Med. 84, 320-325.

Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches

and practical guidelines (4th Ed.). New York: Longman.

Gupta, A., Talavlikar, R., Ng, V., Chorny, Y., Chawla, A., Farrugia, M., Lorette, J., Raza, D., &

Vyvey, M. (2012). Global health curriculum in family medicine: Resident perspective.

Canadian Family Physician, 58, 143-6.

Koplan, J.P., Bond T.C., Merson, M.H., Reddy, K.S., Rodriguez, M.H, & Sewankambo, N.K.

(2009). Toward a common definition of global health. Lancet. 373 (9679), 1993-5.

EVALUATION PLAN 31

Kern, D.E., Thomas, P.A., & Hughs, M.T. (2009). Curriculum development for medical

education: A six-step approach. Baltimore, MD: The John Hopkins University Press.

Loyola University, Chicago. (2013) Center for Experiential Learning LUROP Student Survey.

Retrieved from: http://www.luc.edu/lurop/luropstudentsurvey/

Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA:

Jossey-Bass.

University of California, Los Angeles. (2012) Center for Global Education SOAR Survey.

Retrieved from: http://www.globaled.us/cccsoar/about-ccc-soar-surveys.asp#top

Weiss, C. H. (1998). Evaluation (2nd Ed.) Upper Saddle River, NJ: Prentice Hall.

Wholey, J.S., Hatry, H.P., & Newcomer, K.E. (Eds.) (2010). Handbook of practical program

evaluation). (3rd Ed.) Jossey-Bass: San Francisco

Central Connecticut State University. (2013) Center for International Education Study Abroad

Survey. Retrieved from:

http://web.ccsu.edu/uploaded/departments/AdministrativeDepartments/Center_for_International_

Education/Courses_Abroad/Study_Abroad_Student_Assessment_Form.pdf

EVALUATION PLAN 32

Appendix A: Northwestern Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Heath Logic Model

Inputs

Outputs Outcomes -- Impact

Activities Participation Short Medium Long

People • 5 Program Directors • Program Manager • 2 Deans: Dean of

Feinberg; Dean of Graduate Medical Education

• 8 members in the Center for Global Health

• (?#) of Feinberg Residents

• Mentors for enrolled residents

Finances • Hiring staff for program

lectures • Dinner for Residents • Discretionary funds for

global sites • Printed materials • Transportation to and

from Global Medical Sites

• Medical Insurance Time

• Research and development

• Program design • Benchmark and planning

meetings • Pre-departure

orientation • Curriculum design • Teaching courses

Materials • Simulation lab • Online lecture software • Website • Poster explaining

program • Classroom and office

space Physical Global Medical

Sites • Comprehensive list in

Appendix …

Faculty

• Building university partnerships

• Mentoring • Lectures

Residents

• Residents complete 9 out of 12 core lectures

• Residents complete 5 out of 6 specialty lectures

• Quarterly journal club • Cross residence skills

practicum • Pre-departure training • Global site specific

training • Mentorship program at

Northwestern and global specific site

• Global field experience • Scholarly project • Professional

presentation

• Faculty • Residents • Deans • Residency Program

Directors and Chief Residents

• Local and global healthcare workers

• Local and global healthcare facilities

• Center for Global Health administrators and staff

During students 1- 1 ½

year tract in program

• Residents understand global contexts in health.

• Residents understand global health inequities.

• Residents create cross residency skills to diversify medical knowledge and improve patient care.

• Development of professional skills (i.e. scholarly presentations, writing)

• Increased medical knowledge of global medical issues

• Develop knowledge of traditional medicine (otherwise defined as non-ACGME board certified medicine)

• Expanded understanding of ethics in global healthcare

• Understanding of sociopolitical contexts in relation to global and domestic health

• Exposure to career opportunities in global medicine

Upon completion of

certificate program

• Residents can describe global burden of disease and demonstrate understanding of epidemiologic tools and methods

• Residents will understand health implications of travel, migration and trade

• Residents will recognize the major determinants of health

• Residents can explain the role of community-engagement strategies in heath contexts

• Residents will demonstrate high standards of ethical conduct and quality in global healthcare

• Residents will practice and professionalism in a resource limited setting by providing culturally sensitive care and support to patients, and by understanding the role of traditional medicine

• Residents will demonstrate a basic understanding of the relationship between health and human rights

• Residents will develop global health focused scholarly work

Years following

certificate program

combined with practical

experience.

• Graduates will build action capacity for addressing health resource inequities domestically and globally.

• Graduates will work toward global health equity

• Graduates will be able to provide context outside of western medicine

• Graduates can create global health initiatives within local communities

• Global healthcare equity

EVALUATION PLAN 33

Assumptions

External Factors (thinking broadly) • Residents will be interested in Global Health Certificate program. • Residents will be available to attend on-campus lectures or online lectures if necessary. • Faculty is available to teach lectures. • Faculty is willing to mentor residents. • Lectures will engage students and meet learning outcomes. • Global Health curriculum meets Accreditation Council of Graduate Medical Education. • Initial funding is enough to sustain program. • Residents will be able to complete the certificate program in the proposed timeline of

three years. • Global sites will available and maintain accreditation. • Residents will be able to complete their required global site visit. • Pre-departure component will adequately prepare residents for site visit.

• Program external factors o Residents are developmentally prepared for global health curriculum. o Funding from internal and external sources. o Communication is in place between domestic and global sites. o Key stakeholders will remain in decision-making positions. o Feinberg School of Medicine continues to offer residency programs.

• Global external factors o Political/geographical stability of location o Resource distribution at global health sites o Availability of medical factuality to train residents and meet mentorship

requirements.

EVALUATION PLAN 34

Appendix B: McGaw Global Health Curriculum

EVALUATION PLAN 35

Appendix C: McGaw Graduate Medical Education Certificate of Global Health Survey Construct Map and Item Matrix for PRE-TEST

Pre$test:(McGaw(Graduate(Medical(Education(Certificate(of(Global(Health(Logic(Model(Component( SQ(#( Question( Answer(Type(( Answer(Options((

General(Information(

Blank&& Student&ID&#&Text&Box&

N/A&&

1& Please&indicate&which&residency&program&you&are&currently&participating&in& N/A&&

2& Which&statement&best&describes&your&primary&reason&for&participating&in&the&program&&

CheckFbox/Likert&Scale&&

4&point,&essential&to&not&important&

3& Please&indicate&the&site&you&are&interested&in&completing&your&global&practicum&& CheckFlist&

N/A&&

4& Have&you&had&any&experience&in&global&healthcare&settings?& Answer&option&yes/no&

5& If&yes,&please&describe&indicate&location(s)&and&briefly&describe&responsibility(s)&& Text&Box& N/A&&

12& Please&indicate&your&program&year& CheckFlist& N/A&13& Please&describe&how&you&identify&your&gender&

Text&Box&

N/A&14& Please&describe&how&you&identify&your&race/ethnicity& N/A&15& Please&describe&how&you&identify&your&sexual&orientation& N/A&16& Please&describe&your&intended&professional&practice& N/A&&

17& Prior&to&this&certificate&program,&what&kinds&of&international&experience&have&you&had?&

CheckFlist&&

Answer&options:&no&past&international&experience,&vacation,&lived&abroad&for&more&than&3&months,&volunteered&abroad,&worked&abroad,&served&in&the&military&abroad&for&more&than&3&months,&participated&in&a&study&abroad&program&&&

EVALUATION PLAN 36

&&

&& && && && &&Logic(Model(Component( SQ(#& Question& Answer(Type(& Answer(Options(&

Short(Term(Criteria(1($(Skill(Development((

6& Prompt:&How&would&you&currently&rate&yourself&in&the&following&areas:&&

CheckFbox/Likert&

Scale&&

Very&experienced,&some&experience,&little&

experience,&no&experience&

6A& Scholarly&Writing&6B& Scholarly&Presentations&6C& Ability&to&work&with&diverse&groups&of&people&6D& Ability&to&work&in&teams&6E& Patient&care&&6F& Ability&to&take&initiative&

&& && && && &&

Short(Term(Criteria(2($(Global(Health(Awareness((

7& Prompt:&How&would&you&currently&rate&your&understanding&in&the&following&areas:&

CheckFbox/Likert&

Scale&

Very&knowledgeable,&some&knowledge,&little&

knowledge,&none&

7B& The&relationship&between&health&and&human&rights&7C& Major&global&&health&determinants&7D& Familiarity&with&nonFwestern&health&practices&7E& The&role&of&community&engagement&stratifies&in&healthcare&7F& SocioFpolitical&contexts&in&relationship&to&global&health&&

&& && && && &&

Short(Term(Criteria(3($(Professionalism(

&

&

8& Prompt:&To&what&extent&do&you&agree&or&disagree&with&the&following:&&

CheckFbox/Likert&

Scale&

Strongly&agree,&agree,&disagree,&strongly&

disagree&8A&

Global&health&practitioners&have&the&ability&to&communicate&effectively&with&patients&and&health&practitioners&with&a&different&culture/ethnicity&different&than&their&own.&

8B& Global&health&practitioners&have&a&diverse&skill&set,&outside&their&chosen&specialty&

EVALUATION PLAN 37

& 8C& Global&health&practitioners&develop&relationships&with&other&experienced&healthcare&professionals&

8D& Global&health&practitioners&share&global&health&knowledge&with&novices&in&the&field&

8E& Global&health&practitioners&expand&the&understanding&of&ethics&in&global&healthcare&

&& && && && &&

Medium(Term(Criteria(1($(Understand(Core(Concepts(

9& Prompt:&How&would&you&currently&rate&your&understanding&in&the&following&areas:& CheckF

box/Likert&Scale&

Very&knowledgeable,&some&knowledge,&little&

knowledge,&none&9A& Global&burden&of&disease&9B& Global&burden&of&maternal&and&child&health&

Logic(Model(Component( SQ( Question& Answer(Type(& Answer(

EVALUATION PLAN 38

#& Options(&

Medium(Term(Criteria(1($(Understand(Core(Concepts(

9C& Global&burden&of&mental&health&

& &

9D& Global&burden&of&reproductive&health&9E& Global&burden&of&surgical&disease&&9F& Global&health&systems&&9G& Health&and&Humanitarian&Disaster&Response&9H& Impact&of&climate&change&on&global&heath&9I& Effective&use&of&local&medicine&

&& && && && &&

Medium(Term(Criteria(2($(Scholarly(Work(

10& Prompt:&How&would&you&currently&rate&yourself&in&the&following&areas:&&

CheckFbox/Likert&

Scale&

Very&experienced,&

some&experience,&

little&experience,&no&experience&

10A& Ability&to&synthesize&&global&health&concepts&into&written&work&

10B&

Familiarity&with&current&research&in&global&healthcare&10C& Ability&to&inform&others&about&the&global&health&context& && &&

&& && && && &&

Long(Term(Criteria($(Transference(

11& To&what&extent&do&you&agree&or&disagree&with&the&following:&&

CheckFbox/Likert&

Scale&

Strongly&agree,&agree,&disagree,&strongly&disagree&

11A& Equitable&healthcare&is&a&human&right&11B&

Healthcare&capacity&of&underFresourced&community&directly&effects&the&health&of&all&communities&

11C& I&have&an&ethical&obligation&to&have&competency&in&global&healthcare&11D&

I&believe&medical&practice&outside&the&western&tradition&is&vital&to&global&healthcare&

11E& I&presently&incorporate&global&health&practices&into&my&current&practice&11F& I&currently&participate&in&global&health&initiatives&

EVALUATION PLAN 39

Logic(Model(Component( SQ(#(

Question( Answer(Type((

Answer(Options((

General(Information( Blank&&

Student&ID&#& Text&Box& N/A&&

1& Please&indicate&which&residency&program&you&are&currently&participating&in&

CheckFlist& N/A&&

7& At&which&site&did&you&complete&your&global&health&experience& Text&Box& N/A&&11& While&participating&in&the&certificate&program&did&you&complete&any&

additional&global&health&rotations?&&&Check&Box& Yes/No&

12& If&yes,&please&indicate&the&program,&location,&and&duration:& Text&Box& N/A&&13& Who&is&&your&global&health&mentor& Text&Box& N/A&&14& Please&describe&your&intended&professional&practice& Text&Box& N/A&&

&& && && && &&Short(Term(Criteria(1($(Skill(

Development((2& Prompt:&How&would&you&currently&rate&yourself&in&the&following&areas:&& CheckF

box/Likert&Scale&&

Very&experienced,&

some&experience,&

little&experience,&no&experience&

A& Scholarly&Writing&B& Scholarly&Presentations&C& Ability&to&work&with&diverse&groups&of&people&D& Ability&to&work&in&teams&E& Patient&care&&F& Ability&to&take&initiative&

&& && && && &&Short(Term(Criteria(2($(Global(Health(

Awareness((3& Prompt:&How&would&you&currently&rate&your&understanding&in&the&

following&areas:&CheckF

box/Likert&Scale&

Very&knowledgeable,&

some&knowledge,&

little&knowledge,&

none&

A& The&relationship&between&health&and&human&rights&B& Major&global&&health&determinants&C& Familiarity&with&nonFwestern&health&practices&D& The&role&of&community&engagement&stratifies&in&healthcare&E& SocioFpolitical&contexts&in&relationship&to&global&health&&

&& && && && &&Short(Term(Criteria(3($(

Professionalism(4& Prompt:&To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF

box/Likert&Scale&

Strongly&agree,&agree,&

disagree,&strongly&disagree&

A& Global&Health&Practitioners&have&the&ability&to&communicate&effectively&with&patients&and&health&practitioners&with&a&different&culture/ethnicity&different&than&their&own.&

B& Global&Health&Practitioners&have&a&diverse&skill&set,&outside&their&chosen&specialty&

C& Global&Health&Practitioners&develop&relationships&with&other&experienced&healthcare&professionals&

EVALUATION PLAN 40

D& Global&Health&Practitioners&share&global&health&knowledge&with&novices&in&the&field&

E& Global&Health&Practitioners&expand&the&understanding&of&ethics&in&global&healthcare&

&& && && && &&Medium(Term(Criteria(1($(Understand(

Core(Concepts(5& Prompt:&How&would&you&currently&rate&your&understanding&in&the&

following&areas:&CheckF

box/Likert&Scale&

Very&knowledgeable,&

some&knowledge,&

little&knowledge,&

none&

A& Global&burden&of&disease&B& Global&burden&of&maternal&and&child&health&C& Global&burden&of&mental&health&D& Global&burden&of&reproductive&health&E& Global&burden&of&surgical&disease&&F& Global&health&systems&&G& Health&and&Humanitarian&Disaster&Response&H& Impact&of&climate&change&on&global&heath&I& Effective&use&of&local&medicine&

&& && && && &&Medium(Term(Criteria(2($(Scholarly(

Work(6& Prompt:&How&would&you&currently&rate&yourself&in&the&following&areas:&& CheckF

box/Likert&Scale&

Very&experienced,&

some&experience,&

little&experience,&no&experience&

A& Ability&to&synthase&global&health&concepts&into&written&work&B& Familiarity&with&current&research&in&global&healthcare&

C& Ability&to&inform&others&about&the&global&health&context& && &&&& && && && &&Long(Term(Criteria($(Transference( 7& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF

box/Likert&Scale&

Strongly&agree,&agree,&

disagree,&strongly&disagree&

A& Equitable&healthcare&is&a&human&right&B& Healthcare&capacity&of&underFresourced&community&directly&effects&the&

health&of&all&communities&C& I&have&an&ethical&obligation&to&have&competency&in&global&healthcare&D& I&believe&medical&practice&outside&the&western&tradition&is&vital&to&global&

healthcare&E& I&presently&incorporate&global&health&practices&into&my&current&practice&

EVALUATION PLAN 41

F& I&currently&participate&in&global&health&initiatives&&& && && && &&Process(Goal(1(Criteria($(Mentorship(

Experience(8& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF

box/Likert&Scale&

Strongly&agree,&agree,&

disagree,&strongly&disagree&

A& My&mentorship&experience&benefited&my&practice&of&medicine&B& My&mentor&was&committed&to&my&success&C& My&mentor&expanded&my&knowledge&of&global&healthcare&&D& My&mentor&and&I&effectively&communicated&while&I&was&abroad&E& My&mentor&assisted&in&the&development&of&my&professional&presentation&&

&& && && && &&Process(Goal(2(Criteria($(Global(Site(

Experience(9& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF

box/Likert&Scale&

Strongly&agree,&agree,&

disagree,&strongly&disagree&

A& I&felt&the&preFsite&orientation&prepared&me&for&my&global&site&experience&B& The&lecture&series&enhanced&my&site&experience&C& My&onsite&mentor&worked&to&expand&my&medical&and&professional&skills&D& My&global&experience&was&positive&E& My&site&experience&gave&me&the&chance&to&add&to&improve&my&medical&

skills&F& My&site&supervisor&was&committed&to&making&it&a&good&learning&

experience&&& && && && &&

Process(Goal(3(Criteria($(Didactic(Instruction(

10& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckFbox/Likert&

Scale&

Strongly&agree,&agree,&

disagree,&strongly&disagree&

A& The&Global&Health&Lecture&series&expanded&my&medical&knowledge&outside&my&specialty&

B& The&Global&Health&Lecture&series&schedule&worked&well&with&my&residency&program&schedule&

C& The&lecture&series&met&my&expectation&D& The&lecturers&were&informative&E& The&cross&residency&skill&exchange&improved&my&patient&care&at&my&global&

site&

EVALUATION PLAN 42

Appendix D: McGaw Graduate Medical Education Certificate of Global Health Survey Construct Map and Item Matrix for POST-TEST

Logic(Model(Component( SQ(

#(Question( Answer(

Type((Answer(Options((

General(Information( Blank&&

Student&ID&#& Text&Box& N/A&&

1& Please&indicate&which&residency&program&you&are&currently&participating&in&

CheckFlist& N/A&&

7& At&which&site&did&you&complete&your&global&health&experience& Text&Box& N/A&&11& While&participating&in&the&certificate&program&did&you&complete&any&

additional&global&health&rotations?&&&Check&Box& Yes/No&

12& If&yes,&please&indicate&the&program,&location,&and&duration:& Text&Box& N/A&&13& Who&is&&your&global&health&mentor& Text&Box& N/A&&14& Please&describe&your&intended&professional&practice& Text&Box& N/A&&

&& && && && &&Short(Term(Criteria(1($(Skill(

Development((2& Prompt:&How&would&you&currently&rate&yourself&in&the&following&areas:&& CheckF

box/Likert&Scale&&

Very&experienced,&

some&experience,&

little&experience,&no&experience&

A& Scholarly&Writing&B& Scholarly&Presentations&C& Ability&to&work&with&diverse&groups&of&people&D& Ability&to&work&in&teams&E& Patient&care&&F& Ability&to&take&initiative&

&& && && && &&Short(Term(Criteria(2($(Global(Health(

Awareness((3& Prompt:&How&would&you&currently&rate&your&understanding&in&the&

following&areas:&CheckF

box/Likert&Scale&

Very&knowledgeable,&

some&knowledge,&

little&knowledge,&

none&

A& The&relationship&between&health&and&human&rights&B& Major&global&health&determinants&C& Familiarity&with&nonFwestern&health&practices&D& The&role&of&community&engagement&stratifies&in&healthcare&E& SocioFpolitical&contexts&in&relationship&to&global&health&&

&& && && && &&Short(Term(Criteria(3($( 4& Prompt:&To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF Strongly&agree,&

EVALUATION PLAN 43

Professionalism( A& Global&Health&Practitioners&have&the&ability&to&communicate&effectively&with&patients&and&health&practitioners&with&a&different&culture/ethnicity&different&than&their&own.&

box/Likert&Scale&

agree,&disagree,&strongly&disagree&B& Global&Health&Practitioners&have&a&diverse&skill&set,&outside&their&chosen&

specialty&C& Global&Health&Practitioners&develop&relationships&with&other&experienced&

healthcare&professionals&D& Global&Health&Practitioners&share&global&health&knowledge&with&novices&

in&the&field&E& Global&Health&Practitioners&expand&the&understanding&of&ethics&in&global&

healthcare&&& && && && &&Medium(Term(Criteria(1($(Understand(

Core(Concepts(5& Prompt:&How&would&you&currently&rate&your&understanding&in&the&

following&areas:&CheckF

box/Likert&Scale&

Very&knowledgeable,&

some&knowledge,&

little&knowledge,&

none&

A& Global&burden&of&disease&B& Global&burden&of&maternal&and&child&health&C& Global&burden&of&mental&health&D& Global&burden&of&reproductive&health&E& Global&burden&of&surgical&disease&&F& Global&health&systems&&G& Health&and&Humanitarian&Disaster&Response&H& Impact&of&climate&change&on&global&heath&I& Effective&use&of&local&medicine&

&& && && && &&Medium(Term(Criteria(2($(Scholarly(

Work(6& Prompt:&How&would&you&currently&rate&yourself&in&the&following&areas:&& CheckF

box/Likert&Scale&

Very&experienced,&

some&experience,&

little&experience,&no&experience&

A& Ability&to&synthase&global&health&concepts&into&written&work&B& Familiarity&with&current&research&in&global&healthcare&

C& Ability&to&inform&others&about&the&global&health&context& && &&&& && && && &&Long(Term(Criteria($(Transference( 7& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF Strongly&agree,&

EVALUATION PLAN 44

A& Equitable&healthcare&is&a&human&right& box/Likert&Scale&

agree,&disagree,&strongly&disagree&

B& Healthcare&capacity&of&underFresourced&community&directly&effects&the&health&of&all&communities&

C& I&have&an&ethical&obligation&to&have&competency&in&global&healthcare&D& I&believe&medical&practice&outside&the&western&tradition&is&vital&to&global&

healthcare&E& I&presently&incorporate&global&health&practices&into&my&current&practice&F& I&currently&participate&in&global&health&initiatives&

&& && && && &&Process(Goal(1(Criteria($(Mentorship(

Experience(8& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF

box/Likert&Scale&

Strongly&agree,&agree,&

disagree,&strongly&disagree&

A& My&mentorship&experience&benefited&my&practice&of&medicine&B& My&mentor&was&committed&to&my&success&C& My&mentor&expanded&my&knowledge&of&global&healthcare&&D& My&mentor&and&I&effectively&communicated&while&I&was&abroad&E& My&mentor&assisted&in&the&development&of&my&professional&presentation&&

&& && && && &&Process(Goal(2(Criteria($(Global(Site(

Experience(9& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF

box/Likert&Scale&

Strongly&agree,&agree,&

disagree,&strongly&disagree&

A& I&felt&the&preFsite&orientation&prepared&me&for&my&global&site&experience&B& The&lecture&series&enhanced&my&site&experience&C& My&onsite&mentor&worked&to&expand&my&&medical&and&professional&skills&D& My&global&experience&was&positive&E& My&site&experience&gave&me&the&chance&to&add&to&improve&my&medical&

skills&F& My&site&supervisor&was&committed&to&making&it&a&good&learning&

experience&&& && && && &&

Process(Goal(3(Criteria($(Didactic(Instruction(

10& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckFbox/Likert&

Scale&

Strongly&agree,&agree,&

disagree,&strongly&disagree&

A& The&Global&Health&Lecture&series&expanded&my&medical&knowledge&outside&my&specialty&

B& The&Global&Health&Lecture&series&schedule&worked&well&with&my&residency&program&schedule&

C& The&lecture&series&met&my&expectation&D& The&lecturers&were&informative&

EVALUATION PLAN 45

E& The&cross&residency&skill&exchange&improved&my&patient&care&at&my&global&site&

Running head: EVALUATION PLAN 46

Appendix E: E-mail Draft of Pre-Test Survey

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY

You are being invited to complete this survey as an enrolled student in the McGaw Graduate

Medical Education Certificate of Global Health. Although your participation is a prerequisite for

completion of the McGaw Graduate Medical Education Certificate of Global Health, your

experience will aid our evaluation of the certificate as a means to continuously improve your

experience.

The pre-test is the first component of the evaluation procedure. At the end of the course, you will

be invited to take a post-test survey. All information you provide will remain confidential, but

please indicate your student ID number so we can effectively analyze results.

Student ID Number:

Should you have any questions please feel free to contact: Ashti Doobay-Persaud, MD, Faculty Development Chair adoobayp@nmh.org

EVALUATION PLAN 47

Appendix F: Pre-Test Survey

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY

Questions 1 – 5 will ask you general information: 1. Please indicate which residency program you are currently participating in:

Drop Down Menu

2. How important is this Global Health training to your medical education:

Essential Very Important

Somewhat Important

Not Important

3. Please indicate the site you are interested in completing your global practicum:

Check list will be compiled once sites are finalized, TBA.

4. Have you had any experience in global healthcare settings?

Yes No

5. If yes, please describe indicate location(s) and briefly describe responsibility(s):

Question 6 will ask you to rate your various skills:

6. How would you currently rate yourself in the following areas:

A Major Strength

Somewhat Strong

Average Somewhat weak

A Major weakness

Scholarly Writing

Scholarly Presentations

Ability to work with diverse groups of people

Ability to work in teams

Patient care

Ability to take initiative

EVALUATION PLAN 48

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY

Question 7 will ask about your awareness in global health:

7. How would you currently rate your understanding in the following areas:

Very Knowledge-

able

Some Knowledge

Little Knowledge

None

The relationship between health and human rights

Major determinants of global

health

Familiarity with non-western health practices

The role of community engagement strategy in healthcare

Socio-political contexts in relationship to global health

EVALUATION PLAN 49

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY

Question 8 will ask about your opinions and attitudes:

8. To what extent do you agree or disagree with the following:

Strongly Agree

Agree Disagree Strongly Disagree

Global health practitioners have the ability to communicate effectively with patients and health practitioners with a culture/ethnicity different from their own.

Global health practitioners have a diverse skill set, outside their chosen specialty

Global health programs develop relationships with other experienced healthcare professionals

Global health programs share global health knowledge with novices in the field

Global health programs expand the understanding of ethics in global healthcare

EVALUATION PLAN 50

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY

Question 9 and 10 will ask about your awareness of core concepts in global health

9. How would you currently rate your understanding in the following areas:

Very Knowledge-

able

Some Knowledge

Little Knowledge

None

Global burden of disease

Global burden of maternal and child health

Global burden of mental health

Global burden of reproductive health

Global burden of surgical

disease

Global health systems

Health and humanitarian disaster response

Impact of climate change on global heath

Effective use of local medicine

10. How would you currently rate yourself in the following areas:

Very Knowledge-

able

Some Knowledge

Little Knowledge

None

Ability to synthesize global health concepts into written work

Familiarity with current research in the importance of equitable global healthcare

Ability to inform others about the importance and impact of global health

EVALUATION PLAN 51

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY

Question 11 will ask about your opinions and attitudes:

11. To what extent do you agree or disagree with the following:

Strongly Agree

Agree Disagree Strongly Disagree

Equitable (access, care, affordability) healthcare is a human right

Healthcare capacity of an under-resourced community directly effects the health of all communities

I have an ethical obligation to have competency in global healthcare

I believe medical practice outside the western tradition is vital to global healthcare

I presently incorporate global health health care practice into my current practice

I currently participate in global health initiatives (volunteer, research, education initiatives, community outreach, etc.)

Questions 12 – 17 will ask you general information:

12. Please indicate your program year: Check list will be compiled once residency ranking is finalized, TBA

13. Please give your current gender identity: Fill in Blank

14. Please give your race/ethnicity: Fill in Blank

15. Please give your sexual orientation (voluntary):

EVALUATION PLAN 52

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY

17. Prior to this certificate program, what kinds of international experience have you had?

Check Box If, Yes please give number of occurrences

No past international experience

Vacation

Lived abroad for more than 3 months

Volunteered abroad

Worked abroad

Served in the military abroad for more than 3

months

Participated in a study abroad program

Thank you for taking the time to complete the McGaw Graduate Medical Education Certificate

of Global Health pre-test survey!

EVALUATION PLAN 53

Appendix G: E-mail Draft of Post-Test Survey

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: EMAIL POST-TEST SURVEY

You are being invited to complete this follow-up survey as an alumnus of the McGaw Graduate

Medical Education Certificate of Global Health. Although your participation is a prerequisite for

completion of the McGaw Graduate Medical Education Certificate of Global Health, your

experience will aid our evaluation of the certificate as a means to continuously improve residents’

experiences.

All information you provide will remain confidential, but please indicate your student ID number

so we can effectively analyze results.

Student ID Number:

Should you have any questions please feel free to contact: Ashti Doobay-Persaud, MD, Faculty Development Chair adoobayp@nmh.org

EVALUATION PLAN 54

Appendix H: Post-Test Survey

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: POST-TEST SURVEY

Questions 1 will ask you general information: 1. Please indicate which residency program you are currently participating in:

2. At which site did you complete your global health experience:

Question 3 will ask you about your various skills:

3. How would you currently rate yourself in the following areas:

Very Experienced

Some Experience

Little Experience

No Experience

Scholarly Writing

Scholarly Presentations

Ability to work with diverse groups of people

Ability to work in teams

Patient care

Ability to take initiative

Question 4 will ask about your awareness in global health:

4. How would you currently rate your understanding in the following areas:

Very Knowledge-

able

Some Knowledge

Little Knowledge

None

The relationship between health and human rights

Major global health

determinants

Familiarity with non-western health practices

The role of community engagement stratifies in healthcare

Socio-political contexts in relationship to global health

EVALUATION PLAN 55

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical

Education Certificate of Global Health: POST-TEST SURVEY Question 5 will ask about your opinions and attitudes:

5. To what extent do you agree or disagree with the following:

Strongly Agree

Agree Disagree Strongly Disagree

Global health practitioners have the ability to communicate effectively with patients and health practitioners with a culture/ethnicity different from their own.

Global health practitioners have a diverse skill set, outside their chosen specialty

Global health programs develop relationships with other experienced healthcare professionals

Global health programs share global health knowledge with novices in the field

Global health programs expand the understanding of ethics in global healthcare

EVALUATION PLAN 56

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical

Education Certificate of Global Health: POST-TEST SURVEY Question 6 and 7 will ask about your awareness of core concepts in global health

6. How would you currently rate your understanding in the following areas:

Very Knowledge-

able

Some Knowledge

Little Knowledge

None

Global burden of disease

Global burden of maternal and child health

Global burden of mental health

Global burden of reproductive health

Global burden of surgical

disease

Global health systems

Health and humanitarian disaster response

Impact of climate change on global heath

Effective use of local medicine

7. How would you currently rate yourself in the following areas:

Very Knowledge-

able

Some Knowledge

Little Knowledge

None

Ability to synthesize global health concepts into written work

Familiarity with current research in the importance of equitable global healthcare

Ability to inform others about the importance and impact of global health

EVALUATION PLAN 57

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: POST-TEST SURVEY

Question 8-11 will ask about your opinions and attitudes:

8. To what extent do you agree or disagree with the following:

Strongly Agree

Agree Disagree Strongly Disagree

Equitable (access, care, affordability) healthcare is a human right

Healthcare capacity of an under-resourced community directly effects the health of all communities

I have an ethical obligation to have competency in global healthcare

I believe medical practice outside the western tradition is vital to global healthcare

I presently incorporate global health care practice into my current practice

I currently participate in global health initiatives (volunteer, research, education initiatives, community outreach, etc.)

9. To what extent do you agree or disagree with the following:

Strongly Agree

Agree Disagree Strongly Disagree

My mentorship experience improved my practice of medicine

My mentor was committed to my professional development

My mentor was committed to academic development

My mentor expanded my medical knowledge of global healthcare

My mentor expanded my cultural knowledge of global healthcare

My mentor and I effectively communicated while I was abroad

My mentor assisted in the development of my professional presentation

EVALUATION PLAN 58

10. To what extent do you agree or disagree with the following:

Strongly Agree

Agree Disagree Strongly Disagree

I felt the pre-site orientation prepared me for my global site experience

The lecture series enhanced my site experience

My onsite mentor worked to expand my medical and professional skills

My global experience was positive

My site experience gave me the chance to add to improve my medical skills

My site supervisor was committed to making it a good learning experience

11. To what extent do you agree or disagree with the following:

Strongly Agree

Agree Disagree Strongly Disagree

The Global Health Lecture series expanded my medical knowledge outside my specialty

The Global Health Lecture series schedule worked well with my residency program schedule

The global health content of the lecture series met my expectation

The lecturers were informative

The cross residency skill exchange improved my patient care at my global site

Questions 12 – 17 will ask you general information:

12. While participating in the certificate program did you complete any additional global health rotations?!!

Yes No

13. If yes, please indicate the program, location, and duration:

14. Who is your global health mentor:

EVALUATION PLAN 59

16. Please identify your intended professional practice:

Thank you for taking the time to complete the McGaw Graduate Medical Education Certificate

of Global Health follow-up survey!

EVALUATION PLAN 60

Appendix I: Pre-Test Resident Survey Email Reminder

Email Subject: McGaw: Congratulations & Next Step

Dear Resident First_Name (merge field),

Once again congratulations on your acceptance to the McGaw Graduate Medical Education

Certificate of Global Health!

The next step in the completion of your certificate is to take a few moments to complete a brief online survey (link below) as a means of continuing to improve the student experience. The pre-test is the first component of an evaluation procedure. Upon completion of the certificate, you will be required to take a post-test survey. All information you provide will remain confidential and will have no impact on approval for certificate, but please indicate your Feinberg School of Medicine ID number so we can effectively analyze results. Please complete the survey within two business days.

Survey Link: linkplaceholder.com

Due: within one week

We look forward to working and learning with you as you complete your certificate of global

health.

Should you have any questions, concerns, or requests please feel free to contact:

Dr. Ashti Doobay-Persaud, Director of the McGaw Graduate Medical Education Certificate of

Global Health program

adoobayp@nmh.org

EVALUATION PLAN 61

Appendix J: Post-Test Resident Survey Email Reminder

Email Subject: McGaw: Congratulations & Final Step

Dear Resident_First_Name (merge field),

Once again congratulations on your completion of the McGaw Graduate Medical Education

Certificate of Global Health!

The final step in the completion of your certificate is to take a few moments to complete a brief online survey (link below) as a means of continuing to improve the student experience. All information you provide will remain confidential and will have no impact on approval for certificate, but please indicate your Feinberg School of Medicine ID number so we can effectively analyze results. Please complete the survey within two business days.

Survey Link: linkplaceholder.com

Due: within two business days

We look forward to your feedback and to working with you in the future!

Should you have any questions, concerns, or requests please feel free to contact:

Dr. Ashti Doobay-Persaud, Director of the McGaw Graduate Medical Education Certificate of

Global Health program

adoobayp@nmh.org

EVALUATION PLAN 62

Appendix K: General Survey Reminder Email

Email Subject: McGaw: Survey Reminder

Dear Resident First_Name (merge field),

The next step in the completion of your certificate is to take a few moments to complete a brief online survey (link below) as a means of continuing to improve the student experience. Please complete the survey immediately.

Survey Link: linkplaceholder.com Due: Immediately All information you provide will remain confidential and will have no impact on approval for certificate, but please indicate your Feinberg School of Medicine ID number so we can effectively analyze results. Should you have any questions, concerns, or requests please feel free to contact:

Dr. Ashti Doobay-Persaud, Director of the McGaw Graduate Medical Education Certificate of Global Health program

adoobayp@nmh.org

EVALUATION PLAN 63

Appendix L: Resident Semi-Structured Interview Protocol

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Resident Semi-Structured Interview

Protocol Housekeeping & Consent Obtainment

Thank you for agreeing to participate in this interview today. My name is ___________ and I

will be your interviewer. In advance of our starting our discussion, there are a few housekeeping

items to attend to.

• We have planned this interview to last no longer than thirty minutes. • To facilitate our note taking and use of time, we would like to audio tape our

conversation. • Only the program director and evaluators on the project will be privy to the tapes, which

will be destroyed after transcription. Further, no identifying information will be shared in our final evaluation report.

• Before we start, we ask that you complete two forms. First, a questionnaire that asks for some demographic information; second, consent form, which is required to be included in our research.

• The consent form states: o All information will be held in confidence o Participation is voluntary and you may opt out at any time o We do not intend to inflict any harm

Introduction

You have been selected to speak with us today because of your participation in the McGaw

Global Health Resident Certificate (MGHRC) Program through the Fienberg School of Medicine.

Our evaluation focuses on both the outcomes achieved by the MGHRC and processes used to

accomplish these outcomes. We expect through this interview that we will also capture

improvement opportunities for MGHRC program.

EVALUATION PLAN 64

As I mentioned earlier, I have planned this interview to last no longer than one hour. I would like

to discuss several aspects of the program with you. At times, it may be necessary for me to

interrupt you in order to ensure we are able to discuss all key points. Please understand if this

occurs. Also, you are free to skip any questions you do not wish to answer or stop this interview

at any time. Do you have any questions before we begin?

A. I would like to begin by asking you some questions about your background and global health connection. A1. Tell me about your decision to enroll in the certificate program?

Probes:

• How did you learn about the opportunity? • Who contacted you or did you reach out to you Program Director? • What impact, if at all, did the global site locations have in terms of

your decision? • What effect, if at all, did your fellow residents have in terms of your

decision? B. Main Program Components

I would like to get a better understanding of your experience in completing the certificate

program.

B1. Has your decision to complete this certificate added to your practice?

Probes:

• What has been your favorite or most rewarding experience? • What part of the program most challenged you?

B2. Tell me about your Global Site experience.

Probes:

• For what reasons did you choose that specific site? • In what ways, if at all, has your global site experience affected your

practice?

B3. Tell me a bit more about your mentor/mentee experience

Probes:

• What, if any, impact did this relationship have on your professional development?

EVALUATION PLAN 65

• What is it like being able to spend time with an instructor outside of the hospital environment?

• In what ways, if at all, has your mentor influenced your practice?

C. Effectiveness and Needs

C1. Earlier, you reflected on the certificate program as it relates to your current practice. I

want to better understand this so we can improve the program for future residents. Can

you give me ideas of what, if anything should change and continue in the future?

Probes:

• Do you have specific ideas about what should change or remain in the program going forward? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.

• Do you have specific ideas about what should change or remain in how the program is organized? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.

• Do you have any specific areas that should be added to the certificate?

C2. How, if at all, has your experience differed from your expectations prior to the completion of this certificate? C3.Is there anything about the program that I did not ask you that you would like to share?

Thank you for your participation. Your feedback will be enable us to continue to

improve and grow our program. Please feel free to reach out should you like to share any

additional feedback.

EVALUATION PLAN 66

Appendix M: Semi-Structured Interview Resident Demographic Questionnaire

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Resident Demographic Questionnaire

Please complete the following questions in advance of our interview.

Interviewee Background & Resident Program

1. What is your residence program _______________________________

2. Did you complete a Global Rotation prior to this

program_______________________

a. If so where, ____________________________________________

3. Do you currently volunteer in the community ___________________________

a. If so where, ________________________________________________

Thank you for your participation.

EVALUATION PLAN 67

Appendix N: Semi-Structured Resident Interview Protocol Summary Notes

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Resident Interview Protocol

Resident Interview Protocol Summary Notes

Date:_________________________

Interviewee:___________________________________________________

Interviewer:___________________________________________________

Other Topics

Discussed:_____________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

___________________________

Documents

Obtained:______________________________________________________________________

______________________________________________________________________________

Post Interview Comments:

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

EVALUATION PLAN 68

Appendix O: Resident Semi-Structured Student Interview E-Mail

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Resident Interview Email

Thank you for volunteering your time to participate in an interview regarding your experience as

an enrolled student in the McGaw Graduate Medical Education Certificate of Global Health.

This interview will last no longer than 30 minutes, and be held at The Center for Global Health

offices (address below). In order to attend to your schedule we have blocked out several

interview dates and times, hopefully one will work with your schedule. If not, please contact Dr.

Doobay-Persaud to set up a more convenient time.

Please follow the link below to schedule your interview date and time.

(Doodle Link)

Please follow the link below to complete a brief pre-interview demographic form.

(Link Here)

All feedback and information you provide will remain private.

Should you have any questions please feel free to contact:��

Ashti Doobay-Persaud, MD, Faculty Development Chair

adoobayp@nmh.org

EVALUATION PLAN 69

Appendix P: Mentor Semi-Structured Interview Protocol

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Mentor Semi-Structured Interview

Protocol Housekeeping & Consent Obtainment

Thank you for agreeing to participate in this interview today. My name is ___________ and I

will be your interviewer. In advance of our starting our discussion, there are a few housekeeping

items to attend to.

• We have planned this interview to last no longer than thirty minutes. • To facilitate our note taking and use of time, we would like to audio tape our

conversation. • Only the program director and evaluators on the project will be privy to the tapes, which

will be destroyed after transcription. Further, no identifying information will be shared in our final evaluation report.

• Before we start, we ask that you complete two forms. First, a questionnaire that asks for some demographic information; second, consent form, which is required to be included in our research.

• The consent form states: o All information will be held in confidence o Participation is voluntary and you may opt out at any time o We do not intend to inflict any harm

Introduction

You have been selected to speak with us today because of your participation as a Mentor in the

McGaw Global Health Resident Certificate (MGHRC) Program through the Fienberg School of

Medicine. Our evaluation focuses on both the outcomes achieved by the MGHRC and processes

used to accomplish these outcomes. We expect through this interview that we will also capture

improvement opportunities for MGHRC program.

As I mentioned earlier, I have planned this interview to last no longer than one hour. I would like

to discuss several aspects of the program with you. At times, it may be necessary for me to

interrupt you in order to ensure we are able to discuss all key points. Please understand if this

EVALUATION PLAN 70

occurs. Also, you are free to skip any questions you do not wish to answer or stop this interview

at any time. Do you have any questions before we begin?

A. Interviewee Background and Global Health Connection

A1. Tell me a little about your decision to become a Mentor for the certificate program?

Probes:

• How did you learn about the opportunity? • Who contacted you or did you reach out to you Program Director? • What impact, if at all, did your mentee have on your Global Health

practice? • What effect, if at all, did your colleagues have in terms of your

decision? B. Main Program Components

I would like to get a better understanding of your experience as a Mentor with the certificate

program.

B1. In what ways, if any, has your decision to be a Mentor added to your practice?

Probes:

• What has been your favorite or most rewarding experience? • What part of the program most challenged you?

B2. Tell me about your communication with your mentee while on their Global Site rotation.

Probes:

• How often did you communicate? • How did you utilize the Mentor/Mentee contract? • Would you recommend a particular communication protocol in the future?

If so what?

B3. In what ways, if any, has your mentee influenced your practice?

Probes:

• What, if any, impact did this relationship have on your professional development?

• What is it like being able to spend time with a resident outside of the hospital environment?

C. Effectiveness and Needs

EVALUATION PLAN 71

C1. Earlier, you reflected on the certificate program as this relates to your Global Health

practice. I want to better understand this so we can improve the program for future

Mentors. Can you give me ideas of what should change and continue in the future?

Probes:

• Do you have specific ideas about what should change or remain in the program going forward? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.

• Do you have specific ideas about what should change or remain in how the program is organized? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.

• Do you have any specific areas that should be added to the certificate?

C2. How, if at all, has your experience differed from your expectations prior to your mentee’s completion of this certificate?

C3. Is there anything about the program that I did not ask you that you would like to

share?

Well, thank you very much for taking the time to do this interview with us and sharing

your story. Your participation and understanding of the program is critical in our

evaluation. Please feel free to reach out should you like to share any additional feedback.

EVALUATION PLAN 72

Appendix Q: Semi-Structured Interview Mentor Demographic Questionnaire

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Mentor Demographic Questionnaire

Please complete the following questions in advance of our interview.

Interviewee Background & Resident Program

1. What is your Title _____________________________

2. What is your Department:_______________________________

3. Have you formally Mentored a Resident prior to this

program_______________________

a. If so where, ____________________________________________

4. Do you currently volunteer in the community ___________________________

a. If so where, ________________________________________________

Thank you for your participation.

EVALUATION PLAN 73

Appendix R: Mentor Semi-Structured Interview Protocol Notes

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Mentor Interview Protocol

Student Interview Protocol Summary Notes

Date:_________________________

Interviewee:___________________________________________________

Interviewer:___________________________________________________

Other Topics

Discussed:_____________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

___________________________

Documents

Obtained:______________________________________________________________________

______________________________________________________________________________

Post Interview Comments:

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

EVALUATION PLAN 74

Appendix S: Mentor Semi-Structured Student Interview E-Mail

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Mentor Interview Email

Thank you for volunteering your time to participate in an interview regarding your experience as

a Mentor in the McGaw Graduate Medical Education Certificate of Global Health. This

interview will last no longer than 30 minutes, and be held at The Center for Global Health offices

(address below). In order to attend to your schedule we have blocked out several interview dates

and times, hopefully one will work with your schedule. If not, please contact Dr. Doobay-

Persaud to set up a more convenient time.

Please follow the link below to schedule your interview date and time.

(Doodle Link)

Please follow the link below to complete a brief pre-interview demographic form.

(Link Here)

All feedback and information you provide will remain private.

Thank you again for your time and participation, have a wonderful day!

Should you have any questions please feel free to contact:�

Ashti Doobay-Persaud, MD, Faculty Development Chair

adoobayp@nmh.org

EVALUATION PLAN 75

Appendix T: Lecturer Semi-Structured Interview Protocol

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Lecturer Semi-Structured Interview

Protocol Housekeeping & Consent Obtainment

Thank you for agreeing to participate in this interview today. My name is ___________ and I

will be your interviewer. In advance of our starting our discussion, there are a few housekeeping

items to attend to.

• We have planned this interview to last no longer than thirty minutes. • To facilitate our note taking and use of time, we would like to audio tape our

conversation. • Only the program director and evaluators on the project will be privy to the tapes, which

will be destroyed after transcription. Further, no identifying information will be shared in our final evaluation report.

• Before we start, we ask that you complete two forms. First, a questionnaire that asks for some demographic information; second, consent form, which is required to be included in our research.

• The consent form states: o All information will be held in confidence o Participation is voluntary and you may opt out at any time o We do not intend to inflict any harm

Introduction

You have been selected to speak with us today because of your participation in the McGaw

Global Health Resident Certificate (MGHRC) Program through the Fienberg School of Medicine.

Our evaluation focuses on both the outcomes achieved by the MGHRC and processes used to

accomplish these outcomes. We expect through this interview that we will also capture

improvement opportunities for MGHRC program.

As I mentioned earlier, I have planned this interview to last no longer than one hour. I would like

to discuss several aspects of the program with you. At times, it may be necessary for me to

EVALUATION PLAN 76

interrupt you in order to ensure we are able to discuss all key points. Please understand if this

occurs. Also, you are free to skip any questions you do not wish to answer or stop this interview

at any time. Do you have any questions before we begin?

A. Interviewee Background and Global Health Connection

A1. Tell me a little about your decision to lecture for the certificate program?

Probes:

• How did you learn about the opportunity? • Who contacted you or did you reach out to Dr. Doobay-Persaud? • What impact, if at all, did a previous global health experience have in

terms of your decision? • What affect, if at all, did your colleagues have in terms of your

decision? • How, if at all, has your experience differed from your expectations

prior to your lecture? B. Main Program Components

I would like to get a better understanding of your lecture experience with the certificate

program.

B1. Tell me about your overall experience in lecturing for the program.

Probes:

• What has been your favorite or most rewarding experience? • What part of the program most challenged you? • How would you describe the students’ reception of the lecture? • In what ways did you structure your lectures to meet learning outcomes?

C. Effectiveness and Needs

C1. Earlier, you discussed your expectations of the lecture experience. I want to better

understand this so we can be sure to improve the program for future residents. Can you

give me ideas of what should change and continue in the future?

Probes:

EVALUATION PLAN 77

• Do you have specific ideas about what should change or remain in the program going forward? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.

• Do you have specific ideas about what should change or remain in how the program is organized? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.

• Do you have any specific areas that should be added to the certificate?

C2. Is there anything about the program that I did not ask you that you would like to

share?

Thank you for your participation. Your feedback will enable us to continue to improve

and grow our program. Please feel free to reach out should you like to share any

additional feedback.

EVALUATION PLAN 78

Appendix U: Semi-Structured Interview Lecturer Demographic Questionnaire

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Lecturer Demographic Form

Please complete the following questions in advance of our interview.

Interviewee Background & Resident Program

4. Please give your title __________________________________

5. Please give your program _______________________________

6. Did you give this lecture prior to this program_______________________

a. If so where, ____________________________________________

7. Do you currently volunteer in the community ___________________________

a. If so where, ________________________________________________

Thank you for your participation.

EVALUATION PLAN 79

Appendix V: Lecturer Semi-Structured Interview Protocol Notes

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Lecturer Interview Protocol Summary Notes

Date:_________________________

Interviewee:___________________________________________________

Interviewer:___________________________________________________

Other Topics

Discussed:_____________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

___________________________

Documents

Obtained:______________________________________________________________________

______________________________________________________________________________

Post Interview Comments:

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

EVALUATION PLAN 80

Appendix W: Lecturer Semi-Structured Lecturer Interview E-Mail

Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Lecturer Interview Email

Thank you for volunteering your time to participate in an interview regarding your experience as

a Lecturer for the McGaw Graduate Medical Education Certificate of Global Health. This

interview will last no longer than 30 minutes, and be held at The Center for Global Health offices

(address below). In order to attend to your schedule we have blocked out several interview dates

and times, hopefully one will work with your schedule. If not, please contact Dr. Doobay-

Persaud to set up a more convenient time.

Please follow the link below to schedule your interview date and time.

(Doodle Link)

Please follow the link below to complete a brief pre-interview demographic form.

(Link Here)

All feedback and information you provide will remain private.

Should you have any questions please feel free to contact:��

Ashti Doobay-Persaud, MD, Faculty Development Chair

adoobayp@nmh.org