Facilitating Assessment for Learning:Practical and Evidence-based Strategies for Learners, Faculty and Programs

APPD Spring Conference

Duncan Henry, MDErin King, MDErin McGonagle, MSIVBruce Herman, MD

Education in Pediatrics Across the Continuum (EPAC)§AAMC supported consortium of 4 schools

• U Minnesota, U Colorado, U Utah, UCSF

§Test competency-based, time variable model of advancement from UME-GME-practice§Pediatrics used as a model§EPA-based Competency Framework

Objectives

§Develop a practical exercise incorporating reflection to promote feedback-seeking and iterative practice for learners

§ Implement strategies to enhance trust and feedback provision between faculty and trainees

§ Identify programmatic changes that can support assessment for learning and develop a plan for implementation

Structure of the Workshop§ Introductions and Background (5 minutes)

• Defining key concepts

§Session #1 (25 minutes)

• Focus on the Learner – Reflection

§Session #2 (25 minutes)

• Focus on the Supervisor – Trust

§Session #3 (25 minutes)

• Focus on the Program – Programmatic Assessment

§Questions and Wrap-up (10 minutes)

Background and Key Concepts§Feedback (content, delivery, integration, promotion) is an

essential component of learner development

§Assessment systems should provide useful feedback to multiple stakeholders

• Assessment for learning AND assessment of learning

§Self-regulated learning occurs when learners self-identify goals, seek out feedback related to those goals, and monitor their progress – assessment is a key component of this process

§ In an ideal world, we create systems that support learners and supervisors in engaging and sustaining this process

Mini-Session #1:A PRIMER ON REFLECTION

Erin McGonagle

University of Colorado School of Medicine

EPAC Cohort 3

Figure: A participatory design feedback loop: co-creation of learning opportunities by learners andteachers. Borrowed from “Twelve tips to promote afeedback culture with a growth mind-set: Swinging thefeedback pendulum from recipes to relationships.”

Approaches to Reflection

§Personal Reflection• Written – diary/journal

• Thought-based or otherwise unformatted

§Shared Reflection• 50-word written reflection

• Digital Storytelling: one-image story reflection

• PechaKucha (Japanese:������): a presentation style in which 20 slides are shown for 20 seconds each (6 minutes and 40 seconds in total)

• Simple structured reflection

Several components of a good critical reflection (from “Twelve tips for teaching reflection at all levels of medical education”):

(1) Linking past, present, and future experience

(2) Integrating cognitive and emotional experience

(3) Considering the experience from multiple perspectives

(4) Reframing

(5) Stating the lessons learned

(6) Planning for future learning or behavior.

Reflective Process

Introduction to the ActionPartner up! We will be drawing portraits of our partners, reflecting on the process, and then giving and receiving feedback on our reflections.

Take 1 minute to interview your partner. Then, spend 4 minutes drawing a portrait of your partner using any of the supplied materials. Be creative! After 5 minutes spent on one partner, switch.

While you are completing this exercise, keep in mind that you will be asked to reflect on each stage of the exercise when you have finished. Note moments of discomfort, excitement, etc.

Worksheet #1 - Learner§Take 5 min to generate written reflections using the prompts for

each reflection – before the action, during the action, and on completion of the action.• Linking past, present, and future experience

• Integrating cognitive and emotional experience

• Considering the experience from multiple perspectives

• Stating the lessons learned

• Planning for future learning or behavior.

§Once completed – take 5 min to share with your partner. • How did they approach reflection in this setting?

• How do they reflect in their practice?

Mini-Session #2:PROMOTING TRUST IN

FACULTY-RESIDENT INTERACTIONS

Bruce Herman, MD

Program Director

University of Utah, Primary Children’s

Faculty-Trainee Assessment Relationships§Four players in a feedback-dyad

• Trainee‒ Mastery versus performance orientation

‒ Perception of the learning environment*

• Supervisor‒ Vary with time in training

§ Data, Approach, Perspective, Confidence

• Learning Environment‒ Workplace affordances

• Relationship between provider-receiver‒ Development of trust

Chen et al, Medical Education, 2016Sheu et al, Acad Med, 2017Telio et al, Acad Med, 2015

Trust Formation

Hauer et al, Medical Education 2015

Model for Changing the Feedback/Assessment Culture

Ramani et al, Med Teach 2018

Worksheet #2 - Supervisor§Choose either your clinical learning environment (shift-based,

service-based, continuity) or the one you find most challenging

§Take 5 min to generate your own ideas on how to accomplish these

goals in the clinical environment you chose

• Establish a positive learning climate

• Be a professional role model

• Observe learner performance

• Facilitate reflection and informed self-assessment

§Once completed – share with your table. Find somebody from a different learning environment.

‒ What’s easiest for them?

‒ What’s hardest for you?

Mini-Session #3:PROGRAMMATIC

INFRASTRUCTURE TO PROMOTE SELF-REGULATED

LEARNINGDuncan Henry, MD

Associate Program Director

University of California San Francisco

Erin King, MD

Associate Program Director

University of Minnesota

Programmatic Assessment – A Theoretical Model

van der Vleuten et al, Med Teach, 2012

Example #1: UCSF

§Developed workplace-based assessments using an EPA-based framework • Rolled out a mobile-capable platform

• Resident requested

• Co-completed with faculty

• Generate milestone data behind the interface

§Provided faculty and resident development on assessment content and system• Low stakes, high frequency, contextualized

observations

• Asked for supervision-based judgment

§Paired assessments with an advisor system

Example #2: Stanford

Rassbach et al, Acad Med 2018

Faculty members fill out a narrative description based on their direct observation

Core Evaluator interprets data and completes

Milestone-based evaluation

Example #3: UMN

Faculty members fill out a narrative description

Core Evaluator interprets data and completes milestone

evaluationCore Evaluator also gives feedback at the end of the rotation

Same core evaluator gives feedback 5 to 8 times over the course of residency

Example #3: UMN

Worksheet #3 - Programmatic§ Identify a single learning experience (i.e. rotation) or assessment

focus (i.e. caring for the newborn) that you want to address.

§Take 5 min to generate your own ideas about what your current

state is in and what your ideal changes would be in the following key elements of programmatic assessment.

• Protected time

• Longitudinal relationships

• Shared mental model/learning goals

• Accountability/Ownership

§Take 10 min to share your ideas with the table. What’s your biggest

area for growth? What’s one element that’s working well for you?

Conclusions

§Feedback can and should be used as assessment for learning in trainees and assessment of learning for programs

§Promoting feedback-seeking in learners can be enhanced by using reflection as a part of the feedback encounter

§Enhancing trust between faculty and learners requires facilitation of reflection, direct observation, and modeling investment in the learner

§A culture of self-regulated learning and iterative practice in residency requires programmatic changes to assessment structure, a shared set of goals, longitudinal relationships, and bilateral accountability in faculty and learners

Questions/Comments§Duncan Henry (Duncan.Henry@ucsf.edu)

§Bruce Herman (Bruce.Herman@hsc.utah.edu

§Erin King (erin.king@childrensmn.org)

§Erin McGonagle (erin.mcgonagle@ucdenver.edu)

References§ Aronson, L. Twelve tips for teaching reflection at all levels of medical education, Medical Teacher. 2011;33:3,

200-205.

§ Chen, C et al. Students’ goal orientations, perceptions of early clinical experiences and learning outcomes.

Medical Education 2016;50:203–213.

§ Crommelinck, M and Anseel, F. Understanding and encouraging feedback-seeking behaviour: a literature

review. Medical Education. 2013;47:232–241

§ Gomez-Garibello, C. and Young, M. Emotions and assessment: considerations for rater-based judgements

of entrustment. Medical Education. 2018;52:254–262

§ Hauer, K et al, How clinical supervisors develop trust in their trainees: a qualitative study. Medical Education2015;49:783–795

§ Ramani, S et al. Twelve tips to promote a feedback culture with a growth mind-set: Swinging the feedback

pendulum from recipes to relationships. Medical Teacher. 2018 Feb 7:1-7

§ Rassbach, CE & Blankenburg, R. A Novel Pediatric Residency Coaching Program: Outcomes After One

Year. Acad Med. 2018;93:430–434.

§ Telio, S et al. The “Educational Alliance” as a Framework for Reconceptualizing Feedback in Medical

Education. Acad Med. 2015;90:609–614.

§ van der Leeuw, R & Slootweg, I. Twelve tips for making the best use of feedback, Medical Teacher.

2013;35:5, 348-351

§ Van der Vleuten, C et al. A model for programmatic assessment fit for purpose. Medical Teacher. 2012;34:205–214