Assessment of clinical skills

Joseph Cacciottolo Josanne Vassallo

UNIVERSITY OF MALTA

ANNUAL CONFERENCE - OSLO - MAY 2007

Assessing clinical skills

1. WHY do we need to assess ?

2. WHAT do we want to measure ?

WHY do we assess ?

• To ensure safety of patientsour responsibility to the public

• Achievement of a minimum standardresponsibility to the candidate and University

In principle…

WHY do we assess ?

• To ensure competence

• Often as a means of academic competition

In practice: the scope

WHY do we assess ?

• Formative: to give feedback and advice

• Summative: to grade

• Qualificative or licensing

In practice: the purpose

WHAT do we measure ?

To test not only presence of knowledge

…but also the application of knowledge

In principle…

Several types of clinical assessment

One model presented in some detail to

provide a framework for later discussion

One model of clinical assessment

• Certification of competence - pass / fail a state (and legal) requirement

• Grading in rank orderfor employment / placement purposes

• A competition for the award of a prize

In principle…a three-fold aim

One model of clinical assessment

Measurement of:

• adequacy of basic clinical skills • ability to interpret clinical findings • facility of communication in practical settings• ability to think analytically about diagnosis• ability to discuss management logically

In practice…

Practical steps for assessment

• 6 encounters with different clinical situations

• Two examiners at every encounter, each examiner giving an individual assessment

• Highly structured examination and detailed assessment of skills

• Examiners from other Universities for process evaluation and quality control

Practical steps for assessment

• Encounters with at least six real patients

• Ability to interpret and discuss clinical data

• Management of an emergency scenario

• Appraisal of communication skills and attitude

Set-points for testing

• Attitude to patient

• Actual examination skills

• Presentation of findings

• Clinical judgment

Clinical examination of patients

Set-points for testing

• Evaluation of data

• Significance of data

• Clinical reasoning

Interpretation of clinical data

Set-points for testing

• Ability to solve problems

• Ability to discuss logically

• Clinical judgment and prioritization

• General medical knowledge

Management of emergency situations

Set-points for testing

• Attitude to ‘patient’

• Ability to communicate well

• Clinical judgment

• General medical knowledge

Communication skills

Evaluation

• Subjective evaluation: looking in the mirror • Objective evaluation of outcome

Evaluating ourselves

“Examinations are formidable even to the best prepared…

…for the greatest fool may ask more than the wisest man can answer”

Charles Caleb Colton 1780 - 1832

Evaluating outcome

• Pattern of results

• Consistency of results

• Patterns of marking

• Process shortcomings

• Basis for improvement

Analysis of data to assess effectiveness

What happens to candidates who fail ?

• Review of performance – a formative exercise

• Counselling at a personal level

• Specific attention and individual training

• Repeat assessment after a period of time

Points for discussion

• Competence versus performance

• Exam based versus continuous assessment

• Methodology related issues

• Organisational issues

Competence versus Performance

• In relation to methods used

• Use of core curriculum with additional modules as required

• Eventual medical practice in different environment

• Impact on mobility

Competence versus Performance

• In relation to methods used

• Use of core curriculum with additional modules as required

• Eventual medical practice in different environment

• Impact on mobility

Competence versus Performance

• In relation to methods used

• Use of core curriculum with additional modules as required

• Eventual medical practice in different environment

• Impact on mobility

Competence versus Performance

• In relation to methods used

• Use of core curriculum with additional modules as required

• Eventual medical practice in different environment

• Impact on mobility

Exam based vs continuous assessment

• Assessment of a modular curriculum

• Written exams: MCQs, SAQ, Essays

• Real-time exams

• Orals: clinical / table viva, OSCEs

• Clinical assessment on site: mini-CEX, DOPS, multi-source feedback

• Log books, portofolios, CATs

Exam based vs continuous assessment

• Assessment of a modular curriculum

• Written exams: MCQs, SAQ, Essays

• Real-time exams

• Orals: clinical / table viva, OSCEs

• Clinical assessment on site: mini-CEX, DOPS, multi-source feedback

• Log-books, portfolios, CATs

Methodology-related issues

• Reliability

• Validity

• Practicality

• Transparency

Organisational issues

• Resources available

• Student numbers

• Patient availability

• Invasive procedures

• Scheduling

Role of grading

• Is there need for grading ?

• Selection process for employment

• Selection for postgraduate training

• Quality of assessment method and performance of candidates

Conclusions

There is wide diversity among European medical schoolsregarding methods of assessment of clinical skills:

• some schools aim at pass/fail outcomes, others use systems that lead to grading

• several quality assurance mechanisms are used to varying degrees

There is place for widespread application of agreedstandard methods to:

• assess clinical competences in core curricula • assess additional competences essential to individual practice

UNIVERSITY OF MALTA MEDICAL SCHOOL

1676 - 2007

EU UNIVERSITY PARTNER

● joseph.cacciottolo@um.edu.mt ● josanne.vassallo@um.edu.mt

Thank you

ANNUAL CONFERENCE - OSLO - MAY 2007