Keele University School of Medicine

Reliability of the Leicester Clinical procedure Assessment Tool (LCAT), a

tool to support holistic generic assessment of clinical procedure skills

RK McKinley, J Strand, L Schuwirth

T Gray, T Alun-Jones, H Miller

Keele University School of Medicine

Background

• Assessment– Repertoire of competence

• Micro-certification

– Changing roles• Skills extension• Skills cascade

– Volume– Implications for assessment

Keele University School of Medicine

Instrument specification• Generic• Holistic• Multi-professional • Multi-level• Multi-modal• Enable high quality assessment:

– reliable, valid, acceptable, feasible, educational impact

• Nationally accredited

Keele University School of Medicine

Methods

• Multi stage:– Systematic review– Qualitative development– National validation– Psychometrics– Implementation

Keele University School of Medicine

Stage 1: Systematic review• LCAT v1.7

Keele University School of Medicine

Stage 2: Qualitative

• Focus groups

• Observed assessments, debrief interviews

• Trial ‘hi-fi’ OSCE

• LCAT v2.5

Keele University School of Medicine

Stage 3: Delphi

• National sampling frame– Recruitment– Questionnaire

• Definition of consensus• Agreement

– Categories: All >95%– Components: All >89%

• LCAT v3.0

Keele University School of Medicine

LCAT v3.0

Category– Communication and working with

the patient– Safety – Infection Prevention– Procedural Competence– Team working

Components9

7

6

12

4

Keele University School of Medicine

Stage 4: Psychometrics

• Hi-fi OSCE– 8 stations (2X3 tracks)

• Prosthetics + simulator• Venepuncture, venous cannulation, IV drugs,

ABG, ♀&♂ urinary catheterisation, skin suture, ECG

• ‘Prep station’• 15 minutes + 5 feedback

Keele University School of Medicine

Stage 4: Psychometrics

• Analysis– Used data from 1 assessor per station– Assessors not systematically assigned to

stations– Variable group of assessors– Unbalanced design

Keele University School of Medicine

Stage 4: Psychometrics

• Analysis– P (candidate)– S (stations)– A:(PS) assessors within candidates x

stations– PS,e interaction candidates and stations +

error

Keele University School of Medicine

Stage 4: Psychometrics

• 46 candidates

• 50 assessors– 19 medical – 27 nurses– 1 midwife – 3 HCA

Keele University School of Medicine

Stage 4: Psychometrics

• Results: variance

Effect Df Variancecomponent

% of totalvariance

P (candidates) 41 2.28 24.1

S (stations) 7 0.98 10.3

PS,e (candidate x stations plus general error)

287 3.61 38.1

A:PS (assessors within candidates x stations)

149 2.59 27.3

Keele University School of Medicine

Stage 4: Psychometrics

• Results: D analysis

N stations

 N Assessors

1 2 3

6 0.66 0.70 0.71

7 0.69 0.73 0.75

8 0.72 0.76 0.77

9 0.74 0.78 0.79

10 0.76 0.79 0.81

12 0.79 0.82 0.83

15 0.83 0.85 0.86

0.76

0.790.79

Keele University School of Medicine

Summary• Generic• Holistic• Multi-professional • Multi-level• Multi-modal• Enable high quality assessment:

– reliable, valid, acceptable, feasible, educational impact

• Nationally accredited

Keele University School of Medicine

Summary• Generic• Holistic• Multi-professional • Multi-level• Multi-modal• Quality:

– reliable, valid, acceptable, feasible, educational impact

• Nationally accredited

Keele University School of Medicine

Acknowledgements• Analysis:

– Ron Hoogenboom, Cees van der Vleuten and Arno Muijtjens

• Colleagues: – Linda Ward, Clinical Librarian UHL – University Hospitals of Leicester – Eastern Leicester PCT – Students and staff

• University of Leicester • De Montfort University

• Funding: – Nation Health Service University – Leicestershire, Northamptonshire and Rutland WDC

Keele University School of Medicine

Keele University School of Medicine