Session Number: F111 Looking Inward: Improving Our...
Transcript of Session Number: F111 Looking Inward: Improving Our...
![Page 1: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/1.jpg)
Session Number: F111 Looking Inward: Improving Our Program’s Self Assessment Process
Matthew McQuillan, MS, PA-C Ryan White, MS, MPH, PA-C Rutgers, The State University of New Jersey Physician Assistant Program
![Page 2: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/2.jpg)
Educational Goals & Objectives
• Critically evaluate a program self assessment process
• Utilize core faculty to implement a program self assessment process through the development of an assessment committee
• Describe the opportunities and challenges associated with optimizing a program self assessment process
![Page 3: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/3.jpg)
![Page 4: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/4.jpg)
Session Outline
• The Problem – What we were doing and why a change was necessary
• Our Solution – Forming the Program Assessment Committee
• Our Process – Our current program evaluation process
• The Future – Where we go from here
![Page 5: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/5.jpg)
The Problem
![Page 6: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/6.jpg)
The Problem
• Data collection was not the primary issue • Systematic analysis was a concern • Aggregate analysis needed improvement
– Professional Evaluation Form – Post Rotation Exam performance
• Consider how data is used and reported – SCPE student evaluation of clinical site – Course/Instructor evaluation; Rotation/preceptor evaluation
• Relying on experience > data analysis
![Page 7: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/7.jpg)
The Problem
• Driven by our instruments • Responsive to changing demands for data driven analysis
– ARC-PA – University/School – The Market
• Manpower/Workload issues • External vs internal benchmarks
– PANCE data – Deceleration
![Page 8: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/8.jpg)
The Problem: former drivers
Program Assessment
Experience
Intuition
Crisis Management
Data
![Page 9: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/9.jpg)
The Problem: driving in reverse?
Outcomes Practice Specialty
Curriculum Specialty- Specific
Mission Specialty- Driven
![Page 10: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/10.jpg)
The Problem: Current Process
![Page 11: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/11.jpg)
Our Solution: Forming the Program Assessment Committee
![Page 12: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/12.jpg)
Our Solution:
The Program Assessment Committee (PAC) • Impetus • Charge • Composition • Responsibilities • Frequency
![Page 13: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/13.jpg)
Our Process
![Page 14: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/14.jpg)
Program Assessment
Mission
Learning outcomes
ARC-PA
Health care environment
Our Process
![Page 15: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/15.jpg)
Our Process
Revising our approach to program evaluation Step 1: Review current process Step 2: Categorize data Step 3: Revise instruments Step 4: Assign responsibilities Step 5: Fill in the details
![Page 16: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/16.jpg)
Our Process
Review current process – Develop a comprehensive spreadsheet detailing our current data
collection and analysis process – What data do we already collect – Who is responsible for collecting it
Categorize data – Administrative – Admissions – Didactic – Clinical – Post-grad/other
![Page 17: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/17.jpg)
Our Process
![Page 18: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/18.jpg)
Our Process
![Page 19: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/19.jpg)
Our Process
Revise instruments – Allow program mission, student outcomes, ARC standards and
healthcare environment to drive the process – Review each of the data collection instruments and revise as necessary.
Curriculum Humanities
Outcomes Alumni Survey
Mission Humanistic Care
![Page 20: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/20.jpg)
Our Process
![Page 21: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/21.jpg)
Our Process: Old SCPE Evaluation (Part 1)
![Page 22: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/22.jpg)
Our Process: Old SCPE Evaluation (Part 2)
![Page 23: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/23.jpg)
Our Process: Revised SCPE Evaluation
![Page 24: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/24.jpg)
Our Process
Assign responsibilities Identify the person(s) who would ideally be responsible for collecting and analyzing data • Program Director • Committee Chairs • ADDE • ADCE
![Page 25: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/25.jpg)
Our Process
![Page 26: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/26.jpg)
Our Process
Fill in the details: • Instrument/Data Source • Who compiles the data and how frequently • Benchmark • Data collected • Where is data stored • Who does analsysis • How are results used • Method of data collection
![Page 27: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/27.jpg)
Our Process
![Page 28: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/28.jpg)
Our Process
![Page 29: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/29.jpg)
The Future
![Page 30: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/30.jpg)
The Future
• Ensuring implementation of the process • Who is responsible for ensuring it gets done? • Continuous revision as needed
• Attend the PAEA Program Evaluation Pando Workshop!
![Page 31: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/31.jpg)
References Accreditation Review Commission on Education for the Physician Assistant (2010). Accreditation Standards for Physician Assistant Education, 4th edition (with clarifications through 09/13). Epstein M. Assessment in Medical Education. New England Journal of Medicine. 2007; 356: 387-96. Essary A, Statler M. Using a Curriculum Map to Link the Competencies for the PA Profession With Assessment Tools in PA Education. Journal of Physician Assistant Education. 2007;18(1):22-28. Holmboe E, Ward D, Reznick R, et al. Faculty Development in Assessment: The Missing Link in Competency-Based Medical Education. Academic Medicine. 2011;86(4):460-67. Schwartz A. Assessment in Graduate Medical Education: A Primer for Pediatric Program Directors. Chapel Hill, NC: American Board of Pediatrics; 2011.
![Page 32: Session Number: F111 Looking Inward: Improving Our …2016forum.paeaonline.org/2014/wp-content/uploads/proceedings2014/F111.pdf• Critically evaluate a program self assessment process](https://reader034.fdocuments.net/reader034/viewer/2022042110/5e8a3c905bd50b148142e7a0/html5/thumbnails/32.jpg)
QUESTIONS?