Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

26
Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery

Transcript of Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Page 1: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Bob WoodwardsSAC Chair, Oral and Maxillofacial Surgery

Page 2: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

What do we want?

Objectivity Reproducibility Fairness Ability to spot underachievement early

in training path

Page 3: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Competence-based curriculum Assessment framework Syllabus - explicit standards for common surgical

and specialty-specific knowledge, clinical judgement, technical and operative skills and generic professional skills.

Page 4: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Does

Shows how

Knows how

Knows

Assessment in simulated

settings

Assessment in the work

place

Miller GE. Acad Med 1990;65 (Suppl.):S63–S67.

Page 5: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Producing the “Surgical Expert”

Page 6: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Mini-PATF Mini-PATF 1 Mini-PATF 1

Mini-CEXF Mini-CEX and Case-based DiscussionF 2

DOPSF Surgical DOPS and Procedure-Based AssessmentsF 3

CBDF

ExaminationsS ExaminationsS

Assigned Educational Supervisors’ ReportsS

ARCPS

ISCP Assessment System

Foundation Core Surgical Training Specialty Training

F1 F2 CT1 CT2 ST3 CCT

Page 7: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Key roles• The Programme Director

• The Assigned Educational Supervisor

• Clinical Supervisor

• Trainers – in the multi-disciplinary team

• The trainee

Page 8: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Trainee-led learningGuided by the Assigned Educational Supervisor

• Making meetings with the AES

• Uploading assessments to portfolio

• Completing the Learning Agreement

• Triggering assessments, engaging assessors

• Reading guidance notes

• Taking up learning opportunities (planned or ad hoc)

• Being prepared in good time for reviews

Page 9: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Principles of WPBA

•Assessments for learning

•The main purpose is to provide feedback

•Start early in the placement and spread throughout year

•Fits into normal practice

•Repeated to show progress

•Different settings, patients, different assessor improvesreliabilty

•Trainees upload all assessments – good and less good

Page 10: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

FeedbackShould cover:

• How the trainee felt about the performance

• Trainee’s strengths

• Suggestions for development

• Action plan for improvement

Page 11: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Steps in workplace assessment• Observation fits into normal practice

• Provides verbal constructive feedback immediately after

• Completion of form (if paper-based keep a copy)

• Trainee uploads assessment into the portfolio

• Assessor validates the assessment

Page 12: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

ISCP WPBA methods• Mini-Clinical Evaluation Exercise (Mini-CEX)

• Case-based Discussion (CBD)

• Direct observation of procedural skills (Surgical DOPS)

• Procedure-based Assessment (PBA)

• Multi-source feedback (revised version of Mini-PAT coming soon)

Page 13: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Clinical Evaluation Exercise - Mini-CEX• Trainee-patient encounter e.g. A&E, clinic, ward

• Observation + 10 mins for the form + feedback

• History taking, physical exam, management, communication

• At least 6 per year

• Standard set is for the completion of that stage

• Assessors are consultants, senior trainees, staff grades, other healthcare professionals + one by the AES

Page 14: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Case Based Discussion - CBD• Focussed discussion about a challenging case managed

by the trainee

• Based on trainee’s entries in patient’s notes

• 30 minutes e.g. as part of mid-point appraisal

• Ideal for evaluating reflective practice

• 6 per year absolute minimum

• Standard set is for the completion of that stage

• Assessed by trainee’s AES or equivalent

Page 15: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Direct Observation of Procedural Skills- Surgical DOPS

• Covers all skills required to successfully perform a simple procedure

• Ideally whenever a procedure is carried out

• Observation + 10 mins for the form + feedback

• Assess at least monthly

• Assessors are consultants, senior trainees, staff grades, other healthcare professionals + one by the AES

Page 16: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Procedure Based Assessment - PBA• Covers all skills required to successfully perform a

specialty index procedure

• Ideally whenever a procedure is carried out

• Observation + 10 minutes for the form + feedback

• Assess at least monthly

• Set at the standard of CCT

• Assessor must be a consultant + one by AES

Page 17: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

11132 records from 2071 trainees

global rating by level of training ST1 2.28 ST2 2.61 ST3 2.81 ST4 3.06 ST5 3.34 ST6-8 3.69

After J Foulkes 2009

Page 18: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Multi-source Feedback - MSF• Identifies problems of professional behaviour against

GMP

• 1 per year, can be repeated if necessary

• Trainee self-assessment and 8-12 team ratings

• Health and probity issues can be raised anonymously

• Electronic feedback via the Assigned Educational Supervisor

• AES has a face to face meeting with the trainee to present feedback and sign it off

Page 19: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Feedback Chart for Dr X

2.5

3

3.5

4

4.5

5

5.5

Ra

tin

gYour Self Rating

Rating achieved

Group Rating

Number of Raters: 10

Page 20: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Utility• Blueprinted against GMP

• Studies - JCST evaluation, PBA study by Jonathan Beard

• Valid - PBAs developed and trialled by SACs

• Valid – Methods developed and validated in Foundation

• Educational impact – quality of feedback and action plan, must be followed by reflection

• Feasible - fits into normal practice

• Reliable – different settings, patients and assessors

• Acceptability – trainer/trainee engagement

• Cost effective – time needed

Page 21: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Portfolio evidence• A range of different types of assessments in different

settings

• Assessors’ written comments

• The mini-PAT report by the AES

• The surgical logbook

• Trainees’ presentations, audits, projects, reflections

• Records of discussion with the trainee in appraisals

• Learning Agreement, in particular AES’s report

Page 22: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

AES’s end of placement report Makes use of:

• Evidence in the portfolio, especially assessors’ comments

• Day to day observations of the trainee

• A debriefing session with the clinical supervisor

• Learning Agreement outcomes plus notes

• Standards of GMP

Page 23: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

Preparation for the ARCP• Deanery should provide at least 6 weeks’ notice to

trainees

• It is the trainee’s responsibility to provide a portfolio of complete evidence

• Trainees should be aware that incomplete portfolios will result in outcome 5

• Ensure ARCP panels are thoroughly briefed

Page 24: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

ARCP Evidence

•Learning agreement, signing off syllabus topics

•AES reports

•WPBA in portfolio showing progress

•Exams

•Surgical logbook

•Audit, research, projects

Page 25: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

ARCP Outcomes Satisfactory Progress 1. Achieving progress and competences at the expected rate (clinical) Achieving progress and competences at the expected rate (academic)   Unsatisfactory or insufficient evidence (trainee must meet with panel) 2. Development of specific competences required – additional training

time not required 3. Inadequate progress by the trainee – additional training time required 4. Released from training programme with or without specified

competences Released from academic programme 5. Incomplete evidence presented – additional training time may be

required   Recommendation for completion of training 6. Gained all required competences (clinical) Gained all required competences (academic)

Page 26: Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery.

www.iscp.ac.uk