Keele University School of Medicine Reliability of the L eicester C linical procedure A ssessment T...

Keele University School of Medicine

Reliability of the Leicester Clinical procedure Assessment Tool (LCAT), a

tool to support holistic generic assessment of clinical procedure skills

RK McKinley, J Strand, L Schuwirth

T Gray, T Alun-Jones, H Miller


Background

• Assessment– Repertoire of competence

• Micro-certification

– Changing roles• Skills extension• Skills cascade

– Volume– Implications for assessment


Instrument specification• Generic• Holistic• Multi-professional • Multi-level• Multi-modal• Enable high quality assessment:

– reliable, valid, acceptable, feasible, educational impact

• Nationally accredited


Methods

• Multi stage:– Systematic review– Qualitative development– National validation– Psychometrics– Implementation


Stage 1: Systematic review• LCAT v1.7


Stage 2: Qualitative

• Focus groups

• Observed assessments, debrief interviews

• Trial ‘hi-fi’ OSCE

• LCAT v2.5


Stage 3: Delphi

• National sampling frame– Recruitment– Questionnaire

• Definition of consensus• Agreement

– Categories: All >95%– Components: All >89%

• LCAT v3.0


LCAT v3.0

Category– Communication and working with

the patient– Safety – Infection Prevention– Procedural Competence– Team working

Components9

7

6

12

4


Stage 4: Psychometrics

• Hi-fi OSCE– 8 stations (2X3 tracks)

• Prosthetics + simulator• Venepuncture, venous cannulation, IV drugs,

ABG, ♀&♂ urinary catheterisation, skin suture, ECG

• ‘Prep station’• 15 minutes + 5 feedback



• Analysis– Used data from 1 assessor per station– Assessors not systematically assigned to

stations– Variable group of assessors– Unbalanced design



• Analysis– P (candidate)– S (stations)– A:(PS) assessors within candidates x

stations– PS,e interaction candidates and stations +

error



• 46 candidates

• 50 assessors– 19 medical – 27 nurses– 1 midwife – 3 HCA



• Results: variance

Effect Df Variancecomponent

% of totalvariance

P (candidates) 41 2.28 24.1

S (stations) 7 0.98 10.3

PS,e (candidate x stations plus general error)

287 3.61 38.1

A:PS (assessors within candidates x stations)

149 2.59 27.3



• Results: D analysis

N stations

N Assessors

1 2 3

6 0.66 0.70 0.71

7 0.69 0.73 0.75

8 0.72 0.76 0.77

9 0.74 0.78 0.79

10 0.76 0.79 0.81

12 0.79 0.82 0.83

15 0.83 0.85 0.86

0.76

0.790.79


Summary• Generic• Holistic• Multi-professional • Multi-level• Multi-modal• Enable high quality assessment:




Summary• Generic• Holistic• Multi-professional • Multi-level• Multi-modal• Quality:




Acknowledgements• Analysis:

– Ron Hoogenboom, Cees van der Vleuten and Arno Muijtjens

• Colleagues: – Linda Ward, Clinical Librarian UHL – University Hospitals of Leicester – Eastern Leicester PCT – Students and staff

• University of Leicester • De Montfort University

• Funding: – Nation Health Service University – Leicestershire, Northamptonshire and Rutland WDC

Keele University School of Medicine Reliability of the L eicester C linical procedure A ssessment T...

Documents

Transcript of Keele University School of Medicine Reliability of the L eicester C linical procedure A ssessment T...