Department of Graduate Medical Education (GME) Conflicts of Interest Kim Walker – No conflicts of...

Department of Graduate Medical Education (GME)

Conflicts of Interest

Kim Walker – No conflicts of interest to disclose Ann Dohn – No conflicts of interest to disclose

Kim Walker, PhD

Education Specialist

Ann Dohn, MA

DIO & GME Director

SES032Meaning beyond numbers:

The power of qualitative inquiry for program assessment


Session Outcomes

Describe the use and strengths of qualitative inquiry in program assessment.

Describe and perform the basic steps of qualitative inquiry including: deciding on a data collection tool, coding data for emergent themes, member checking and interviewer corroboration to establish validity.

Use emergent themes and supporting qualitative and quantitative data to provide constructive and actionable feedback to program leadership.

Compare and contrast the pros and cons of three software tools designed for qualitative inquiry.


Yes, Numbers Matter!Hypothetical Institutional Report Card


Qualitative Inquiry: Pros and Cons

• Qualitative data = “what” “why” & “how” of the resident and faculty perceptions and experiences that drive these ratings

• Explore topics in more depth and detail than quantitative research

• Qualitative data, while meaning-FULL, can also be unwieldy and difficult to come to any conclusion outside of “these residents nowadays are sure an entitled bunch of needy trophy kids!”

• Cannot generalize your findings to a broader population


Qualitative Data Mining


Data Source: Surveys

• Larger populations where opinions matter and research subjects are likely to respond to closed and open questions

• Important to minimize survey length and questions to only latent constructs of interest


Data Source: Observations


Data Source: Interviews

Types of interviews:

1. Structured (only pre-set questions)

2. Semi-structured

3. In-depth and unstructured

Uses:

4. To clarify meaning

5. Discuss progress/results

6. Support exploratory work

Interactive collection of participant perspectives.


Data Source: Focus Groups

• Combines interviewing and participant observation

• Uses group interaction to generate data and gain first hand insights


Focus Groups vs. In-depth Interviews

Use focus groups when…

• Group interaction may stimulate responses

• Topic amenable to less input from individual respondents

• Logistically feasible to assemble target respondents in one location

• Quick turn around needed, funds limited

Use in-depth interviews when…

• Group/peer pressure would inhibit responses

• Subject matter is too sensitive for group discussion

• Topic warrants greater depth of individual respondents


Recap: Sources of Qualitative Data

• Surveys (GME or Program Specific)• Program Assessments (faculty and residents)• Observations (resident shadowing; rounding)• Interviews (individual or small groups)• Focus groups (larger consensus groups)• Document reviews (Internal reviews)• Case studies (holistic, multiple data sources) • Internet? Message boards? Residency chat rooms?

What is the word on the “streets” about our programs?


Data Collected: Now What?


Data Coding: Approaching your data

General Specific

Specific General

Observation

Deductive

Inductive

AbductivePlausible explanation and recommendations


Coding Qualitative Data

Codes are:

Labels that assign symbolic meaning to the descriptive or inferential information compiled during the study

Prompts or triggers for deeper reflection

Developed using multiple approaches

Deductive – Provisional “start list” from conceptual framework, preliminary codebook

Inductive – Codes emerge progressively, no preliminary codebook

Abductive – Based on early plausible explanations, combination of deductive and inductive


Tools for coding: Manual


Data Coding: software supported


From Text to Themes

• Thorough, well documented analysis

• Increases opportunities for replication

• Enhances credibility• Clarify if broad categories (or

domains) are theoretically or logically categorized

• How findings relate to your theoretical framework

Transcribed text

Organized Codes

Identifying categories

Interpret findings

Formulate relevant themes

Highlight key findings


Mining the Data Activity:hypothetical data set


Mining the Data Activity:Step 1 – Read each excerpt and highlight key words that capture the essence of the response.


Mining the Data Activity:Step 2 – Summarize in few descriptive terms in “Individual Margin Coding Notes”

Interns disconnected

Needs: structure, organization, attention to details

Higher standards


Mining the Data Activity:Step 3 – Refine coding notes into themes identified across all responses/excerpts.

Interns disconnected

Needs: structure, organization, attention to details

Higher standards Structured,

detailed curriculum


Mining the Data Activity:Step 4 – Focus on key overall themes from the entire qualitative assessment.


Promoting Qualitative Research Validity

Strategy Description

Extended fieldwork To provide for both discovery and validation, collect data over extended period of time

Low-inference descriptors

Use of codes/categorizes as close to participants actual words

Triangulation Cross-checking information and conclusions through the use of multiple procedures or sources: Data, Methods, Investigator, Theory


Promoting Qualitative Research Validity

Strategy Description

Participant Feedback

Researcher’s interpretations and conclusions should be discussed with other participants

Peer Review Researcher’s interpretations and conclusions should be discussed with other people (skeptical, disinterested peers, devil’s advocate)

Negative case sampling

Locating and examining cases that disconfirm the researcher’s expectations and tentative explanations


What the numbers say: Not Good!


GME Plan

1. Qualitative Analysis of:Resident Focus GroupThree years Program and House Staff EvaluationsInternal Review Report

2. Summarize Findings – Focus and Prioritize on top emergent issues

3. Meeting to present to program leadership

4. Follow up: monitor subsequent year surveys; follow up focus group?


From Analysis to Recommendations

“Background Noise”

Areas of Concern

Program Case Study



Program Case Study:• Focused on top three emergent

themes/categories of resident dissatisfaction with learning experience, supported by quantitative and qualitative data

1. Lack of mentoring (research and career)2. Minimal faculty engagement/

camaraderie3. Service over education (lack of support

staff / need for additional residents)


Delivering the Golden Nuggets

Provide concise, data driven findings and recommendations

Who is receiving the information? Are they in a position to take action?

Define action plans, set goals and timelines

Check back in with leadership

Follow up with monitoring subsequent surveys; focus group/interviewees



Program Case Study:• Recommendations

1. Lack of mentoring (research and career): Implementation of Mentoring Program

2. Minimal faculty engagement/camaraderie Focus on increase faculty attendance at educational

sessions (attendance tracked) Planned offsite retreat for faculty and residents

3. Service over education (lack of support staff / need for additional residents) Hire additional support staff to handle “scut” work Program expansion for additional residents

• Next Step: Follow up focus group review and 2014 survey analysis


Concise, Data-Driven Summary

Applying quantitative metrics to qualitative data


Qualitative Software Tools

Pkg 1 Pkg 2 Pkg 3Current Version 10 7 Web-based

Supported OS Win XP SP2 or later

Win XP SP2 or later; Mac OS*

WinXP or Mac OS X 10.5 Modern Browser

Text Coding YES YES YESPDF Coding YES YES NOVideo Coding YES YES YESAudio Coding YES YES YESImage Coding YES YES NO

Solid program all round

Good for multimedia; Spanish version; *Mac OS in 2014

Web-based flexibility for collaboration; Mac OS


Resources

http://fieldresearch.msf.org/msf/bitstream/10144/84230/1/Qualitative%20research%20methodology.pdf


Questions

Please feel free to contact us with any questions @ [email protected] or [email protected]

Department of Graduate Medical Education (GME) Conflicts of Interest Kim Walker – No conflicts of...

Documents

Transcript of Department of Graduate Medical Education (GME) Conflicts of Interest Kim Walker – No conflicts of...