Assessment Taskforce (formerly known as Evaluation Taskforce)

APPD National Meeting

March 27, 2015

Katy Bartlett and Mark Vining

Disclosure

• We have nothing to disclose.

Introductions

• Chair: Katy Bartlett (term 2011-2015)

• Vice-Chair: Mark Vining (term 2013-2017)

• Incoming Vice-Chair: Carrie Rassbach (2015-2019)

Agenda

• Large Group Discussion: 15 minutes

• Break out sessions: 45 minutes

• Report Back to Large Group: 30 minutes

• Plans for post-meeting: 15 minutes

Taskforce Charges

1. Understanding the needs of APPD membership in the domain of learner, faculty and program assessment

2. Communicating to APPD leadership proposed areas of development of assessment instruments and procedures.

3. As well, the Task Force will serve as a group of content experts (program directors and leaders) who assist pediatric residency programs in improving and implementing their assessment procedures. Such procedures may include assessment and feedback to individual trainees and faculty, as well as evaluating curricular or programmatic activities.

What have we been doing?

Past work of the Assessment Taskforce

Assessment Primer

• Project initiated by Patty Hicks and Carol Carraccio, edited by Alan Schwartz

• Part 1: assessment principles

• Part 2: assessment of competencies

• Written for program directors

What have we been doing?

• Faculty development modules on assessment in graduate medical education: – The Milestone Project (on MedEdPortal) – Medical Knowledge (On Share Warehouse) – Patient Care (on MedEdPortal) – Interpersonal and Communication skills (on MedEdPortal) – Professionalism (on Share Warehouse) – Practice-Based Learning and Improvement (in submission

to MedEdPortal) – Systems-Based Practice (ready to submit to MedEdPortal)

• Series entitled “Faculty Development Series on Assessment”

Faculty Development Series on Assessment

in Graduate Medical Education

Patient Care Skills: Assessment and Evaluation

Module created by: Heather L. Burrows M.D., Ph.D

David Rappaport M.D.

and Pam Dietz M.D.

Based on the chapter by Ann Burke, M.D. entitled “Patient Care” published in Assessment in Graduate Medical Education: A Primer for Pediatric Program Directors. Chapel Hill: ABP, 2011.

Faculty Development Series on

Assessment in Graduate Medical

Education:

The Milestone Project

Vasudha L. Bhavaraju, MD

Katy Bartlett, MD

Ann Guillot, MD

Daniel Schumacher, MD, MEd

Successes of Modules

• Small groups of taskforce members working collaboratively on shared academic project

• Each group led by a different taskforce member

• Scholarly activity for those who participated

• Resources available to members of our organization during a time of intensive faculty development

Small group discussions in Spring 2014

1. How to use the CCC to “interpret” traditional assessment data to inform milestone reporting –Suzette Caudle

2. How to incorporate Milestone anchors into assessment tools – Katy Bartlett

3. How to observe and assess the “challenging” subcompetencies (e.g., QI, Personal and Professional Development, etc.) – Dan Schumacher

4. How to assess important knowledge/skills/attitudes that are not captured sufficiently well within the 21 required subcompetencies – Mark Vining

What came out of that meeting?

• Session at fall meeting to share best practices for milestone-based assessment and CCCs

• Survey to quantify faculty time commitment for CCC members across programs

– survey questions have been developed

– IRB approval has been obtained

– submitted to the research taskforce for review prior to distribution to the APPD program director listserv

• Mini poster symposium on assessment strategies

Next Steps

• Develop new working groups around multi-source assessments. – Patients (Carrie Rassbach)

– Peers (Jennifer DiPace)

– Self-Assessment (MacKenzie Frost)

– the multi-disciplinary team (Bill Varade)

– “Personal clinical effectiveness” (Greg Gorman and Sara Multerer)

– Multi-source assessments for fellowship programs (Pnina Weiss and Charlene Rotandi)

Small group instructions

• Spend 45 minutes discussing challenges, sharing current ideas/strategies and brainstorming about opportunities

• Report back to large group for feedback on ideas • Return to small group to develop a concrete plan

for moving forward on developing tools or strategies after the meeting. – Identify leader and active members (including list of

contacts) – Assign roles – Set date for next “meeting”