EducationalEducationalScholars Scholars ProgramProgram

PAS: May 6, 2008PAS: May 6, 2008

Academic Pediatric

Association

2

CME Disclosures

The speakers at The speakers at

this session have this session have no relevant no relevant financial interests financial interests to disclose or to disclose or conflicts of conflicts of interest to report.interest to report.

Mid-course monitoring of Mid-course monitoring of an educational program: an educational program:

•What are the crucial What are the crucial sources of sources of information?information?•How should one How should one respond to these data?respond to these data?

4

How would you apply these guiding principles?

Planned curriculum: 14 learning goals Principles of adult learning in all interactions

• Curriculum reflects identified needs of learners• Hands-on practice of authentic tasks• Scholarly projects with mentoring and regular

feedback• Developmental evaluations of authentic

educator documents Professional networking and peer learning Dynamic, evolving learning program, shaped

by needs assessments, evaluation and feedback from participants

Serious research and evaluation of program to help build sustainable support

5

How would you apply How would you apply needs assessment needs assessment results? results?

6

Needs Data: See HandoutCohort I: Years 1 vs 2

• Items with greatest and least reported increase in competence

Cohort 2 Start of Year 1• Items with greatest and least

reported competence

7

Our Response to the Needs Assessment Results

Modified curriculum each year to address highest perceived needs:• Yr 1: Evaluation, qualitative research

(ISMs and PAS session)• Yr 2: Education technology and

problem learners (ISMs)• Yr 3: Peer feedback (scholarly writing

wasn’t included in the needs assessment)

How would you How would you apply program apply program evaluation results?evaluation results?

9

Program Evaluation by Scholars

ESP Activity Time Expended

(hrs)

Interest% >3

Value% >3

PAS sessions (mean of 2) 10.4 58 % 47%Intersession Modules (Mean of 4)

3.8 42 % 37%

PAS Workshop reviews 7.8 42 % 36 %

Developing EP 10.8 56 % 63 %

Scale 1=low 5=high

10

ESP Activity Mean Time Expended (hrs)

Interest>3

Value% >3

Conducting your project 63 88 % 84 %Writing project report 7 36 % 38 %Interactions with mentor (Not ESP)

16 48 % 50 %

Interactions with ESP facilitator re project

4 48 % 35 %

Interactions with ESP facil re EP/CV/career

3 54 % 46 %

Program Evaluation by Scholars

Scale 1=low 5=high

11

Overall Ratings of ESP by Scholars

Effort expended 5.9Value received 6.2

Scale 1=low 9= very high

12

Our Response to the Program Evaluation Results

Modified timeline of assignments

Reduced number of Intersession Modules

Continued strong emphasis on project reports and EPs

Considering reducing workshop review requirement

Hope to increase faculty interactions with scholars!

13

How can you use an How can you use an educational program educational program as a springboard for as a springboard for educational educational scholarshipscholarship??

14

Rationale for the EP Research Project

ESP Scholars needed EP for career development

We needed EPs for scholar and program evaluation

We needed research opportunities to reward our volunteer faculty

RESULT: The EP Project

15

Quick Project Overview Since Jan 2006

Developed EP template based on findings of AAMC national consensus conference

MedEdPortal review and approvalConstruction of EP analysis tool by

consensus development process, using Scholars’ EPs as raw material

Revision of template to match toolPresentation to AAMC-GEA Steering

CommArticle submitted to Academic MedicineMultiple workshops on EPs

EPTemplate

List of >100List of >100Quantitative Quantitative

ItemsItems

List of 52List of 52Qualitative Qualitative

ItemsItems

Tool 1.1:Tool 1.1:Selected and Selected and

Combined Combined 43 Items43 Items

Tool 1.2:Tool 1.2:Refined and Refined and Reconciled Reconciled

ItemsItems

Tool 1.3:Tool 1.3:Final contentFinal content

and and Format Format

MedEdMedEdPortal Portal

ApprovalApprovalMedEdMedEdPortal Portal ReviewReview

EP Test 4n=2

8 raters

Development of the Educator Portfolio Analysis Tool

EP Test 3n=3

5 raters

EP Test 5n=25-358 raters

Tool DevelopmentInter-rater Reliability Testing

EP TemplateRevision

EP Test 1n=27

6 raters

EP Test 2n=5

4 raters

EP TemplateRevision

TemplateDevelopment

See handout for item list

17

Future PlansInterrater reliability testing with

new EPs (with article)National consensus development

process through AAMCNational dissemination via

workshopsEP website with electronic data

entry/analysis ($$ to support ESP)National norming study

18

ESP BusinessESP Business

19

Review of Program Requirements: Cohort 1

2 workshop reviews/yr (due at ESP session) Intersession Module 1 by Oct 15 Final Educator Portfolio and updated

Curriculum Vitae by Jan 15, 2009 Peer review of another scholar’s EP by

Mar 15 (substitutes for Intersession Module 2) Evidence of peer reviewed acceptance of

a ESP project publication or national presentation by Dec 2009 or thereafter

Final project report at program completion We will ask for EPs and CVs periodically in

the future

20

Cohort 2: Simplified ESP Assignment Timeline

At PAS: 2 workshop reviewsJuly 15: Annual Project ReportsAug 30: Program evaluation surveyOct 15: Intersession Module 1

(technology in educ)Jan 15: Updated Educator Portfolios

and CVs Mar 15: Intersession Module 2 (EP peer

review)

21

Agenda for Today 8:00-8:30   Introduction/orientation 8:30-12:00 Turning Scholarly Projects Into Scholarly

Products • 8:30 - 10:00 Writing educational scholarship articles • 10:00 - 11:00 Planning and presenting workshops • 11:00 - 12:00 Preparing and giving platform presentations

12:00- 1:30  Lunch and discussion: • Cohort 1: How to take lessons learned from ESP to move

forward in one’s career? • Cohort 2: Discussion of projects

1:30-4:15 Developing Peer Feedback Skills • 1:30-1:45 Giving and receiving peer feedback • 1:45-2:45  Platform session with feedback • 2:45-3:45  Poster review with feedback • 3:45-4:15  Feedback discussion and ESP session evals

4:15-5:00 Feedback from Scholars on the ESP program

Break