Workplace-Based Assessment

Procedure Based (PBA)

These slides have been prepared to facilitate discussion on the use of the PBA. The suggested practical group exercise at the end is designed to be used with a training video.

Overview

• What is the PBA?• Purposes of the PBA• PBA Form• PBA Worksheet• PBA Domains• Completing the form• PBA Global Summary• PBA vs elogbook levels• How does the PBA work?• Who should use the PBA?

• PBA Blueprint• How many PBAs?• PBA validity and reliability• Preparation for the PBA• During the PBA• After the PBA• Group exercise• Further Reading• Acknowledgements

What is the PBA?

• Mandatory workplace assessment for learning

• An observation of the trainee performing a practical procedure usually in the operating theatre

• Available for advanced specialty index procedures

• Covers patient safety, consent, pre-operative planning and preparation, intraoperative technique, communication with the team and post-operative management

PBA vs DOPS

DOPS:

• The counterpart of the PBA

• For the simpler index procedures for Core Training

• Set in Outpatients or Treatment Room

• Adaptable as a portion of a more complex procedure

• Should be developed alongside the PBA

Purposes of the PBA

• To provide feedback to help the trainee improve their technical and procedural skills, decision making, leadership and professional skills

• To encourage a reflective approach to learning

• To help the trainee build evidence of their progress

• PBAs form one of the indicators that inform the AES’s end of placement report for the ARCP

• Individual PBAs are not designed to sign off independent practice.

PBA Form

PBA WorksheetProvides examples of positive and negative behaviours that are associated with each competency

Serves as an internal check of the validity of the competency

Provides a useful guide for assessors and trainees

PBA Domains

Six core domains:

1. Consent

2. Pre-operation planning

3. Pre operative preparation

4. Exposure and closure

5. Intra operative technique

6. Post operative management

Completing the form

• Rated to the standard of Certificate of Completion of Training (CCT)

• Duration of the procedure allows analysis of progression

• Duration of the assessment includes feedback element

• Operation more difficult highlights unusual problems

• Simulated setting allows PBAs to be used in skill labs etc.

• Competencies marked G for generic allow cross specialty analysis

• Competencies marked T are procedure-specific

• 3-point rating scale plus Not assessed for areas outside remit or not observed

• Area for written feedback

• Trainee can add reflection when PBA is uploaded

• Trainee can link to syllabus topics when PBA is uploaded

PBA Global Summary

• Four levels describing of the trainee’s overall competence on that occasion

• Maps to surgical logbook levels• Level 4 is equivalent to a specialist in NHS practice• Level 4 in any one procedure does not permit a trainee to

perform a procedure unsupervised

PBA vs elogbook levels

PBA LEVELS

0 Insufficient evidence observed

1 Unable to perform the procedure under supervision

2 Able to perform under supervision

3 Able to perform with minimum supervision

4 Competent to perform unsupervised

ELOGBOOK LEVELS https://www.elogbook.org/

Observed (O)

Assisting (A)

Supervised - trainer scrubbed (S-TS)

Supervised - trainer unscrubbed (S-TU)

Performed (P)

Training more junior trainee (T)

How does the PBA work?

• The trainee selects an advanced procedure, representative of their normal workload

• The setting is in the operative theatre, scrubbed

• The assessor observes the trainee carrying out the procedure OR parts of procedures

• The assessor and trainee discuss the procedure using the PBA form as a guide

• The assessor provides feedback on progress, covering strengths, exploring development needs and agreeing action plans within a feasible timescale.

• The assessor records their feedback on the PBA form and provides a rating against competencies where applicable

• Takes as long as the procedure with 10-15 minutes feedback

Who should use the PBA?

Trainee• The PBA is for higher specialty training - ST3 and above

Assessor• Consultants• Trainee’s current AES• Senior trainees depending on training level and complexity

of procedure• Senior SAS grade surgeons

Assessors must have received training in the tool and be expert in the clinical problem/task

PBA Blueprint

Good Medical PracticeCurriculum

Knowledge Judgement Technique Professional

Knowledge Skills and Performance

a) Maintain your professional performance * ** ** **b) Apply knowledge and experience to practice ** ** ** **c) Keep clear, accurate and legible records ** ** ** **

Safety and Quality

a) Put into effect systems to protect patients and improve care ** ** ** **b) Respond to risks to safety ** ** ** * *c) Protect patients and colleagues from any risk posed by your health

** ** ** **Communication Partnership & Teamwork

a) Communicate effectively ** ** ** **b) Work constructively with colleagues and delegate effectively

** ** ** **c) Establish and maintain partnerships with patients * * * **

Maintaining Trust

a) Show respect for patients * ** ** **b) Treat patients and colleagues fairly and without discrimination * ** ** **c) Act with honesty and integrity * * ** *

Assessment blueprints help to ensure that assessment methods are mapped to the domains of the curriculum and Good Medical Practice.

A single method will cover some domains and the range of assessments should cover all domains

** Directly assessed

* Indirectly assessed

How many PBAs?

• To be used throughout surgical training

• Ideally performed every time the trainee is supervised undertaking an index procedure

• Should be of a sufficient number at level 4 but will vary according to the complexity of the procedure and experience of the trainee

• Should be agreed with the trainee’s AES

• The JCST Quality Indicators state that training posts should offer at least 40 WBAs per year, of which the PBA should be of an equal proportion to other methods. This is checked at the ARCP

PBA validity and reliability

Validity and reliability is enhanced by:

• Assessor training and trainee induction• Linking to Learning Agreement objectives• Use as an assessment for learning• Using frequently throughout training• Using a range of assessors who are qualified and trained• Use across index procedures• Increasing the complexity according to the trainee’s

competence and progression • Feedback addressing trainee development needs• Triangulation with the surgical logbook• Trainee reflection on feedback

Preparation for the PBA

Trainee:• Arrange the case and PBA with the assessor well in

advance• Arrange to have the form to hand, either printed or online• Reflect on the procedure beforehand

Assessor:• Receive training in the tool• Ensure it is a procedure in which you have expertise

Both:• Ensure the patient has given consent• Make suitable time for the procedure and feedback• Agree the appropriate level of supervision

During the PBA

Trainee:• Carry out the procedure as you would normally• Explain what you intend to do throughout• Ensure the patient suffers no increased risk or discomfort

Trainer:• Observe the encounter• Intervene if patient safety is at risk• Immediately afterwards and in a suitable environment, discuss

with the trainee, giving the trainee the opportunity to give their view• Summarise the feedback including what was done well and what

needs to be improved• Expand on areas that were less than satisfactory, including

suggestions for remedial steps and actions• Ensure the form captures feedback and areas observed

After the PBA

Trainee:• Upload feedback to the portfolio accurately in good time• Reflect on performance and feedback received• Follow up action plans

Trainer:• Validate the assessment in good time• If necessary, follow up on any actions agreed• Discuss the trainee’s needs with the AES if necessary

Group exercise

Purpose: To practice rating PBA, discuss best practice and enhance consistency

Technique: With a group of new and experienced trainers:

1. Group discusses the meaning of the PBA competencies

2. Group views a PBA video (see ISCP PBA example videos) https://

www.iscp.ac.uk/surgical/assessment_observation.aspx (pause at point of feedback in order to discuss the trainee’s performance)

3. Each trainer completes the PBA Form ratings and writes feedback comments based on the video

4. Group compares ratings and discusses written feedback

5. Each trainer reflects on whether they would adjust their ratings/comments in light of discussion

Repeat the exercise with another video (alternatively a role play).

Further discussion:• Use of scenarios in a simulated setting to help trainees practice skills

Further reading

1. ISCP Guidance notes on using the PBA https://www.iscp.ac.uk/static/public/cex_guidance.pdf

2. ISCP Tips for using PBA

https://www.iscp.ac.uk/static/public/cex_tips.pdf

3. PBA Form

https://www.iscp.ac.uk/static/public/cex_form.pdf

Acknowledgements

The PBA was derived from OSATS by the Surgical Education Research Group, Department of Surgery, University of Toronto.

The use of the PBA in the UK was pioneered in Orthopaedics by the Orthopaedic Curriculum and Assessment Project (OCAP)

The OpComp Project in General Surgery led to further updates

The surgical Specialty Advisory Committees (SACs) developed PBAs for the surgical specialties within for the Intercollegiate Surgical Curriculum Programme (ISCP)

The ISCP PBA was validated by Professor Jonathan Beardhttp://www.journalslibrary.nihr.ac.uk/__data/assets/pdf_file/0003/64749/FullReport-hta15010.pdf