Best practice in

assessment

Emyr Wyn Benbow

University of Manchester

Traditional medical courses, circa 1974

Second MB general pathology and microbiology

lots of lectures and formal tutorials

assessed by essays, short notes and MCQs

Third MB systemic pathology and microbiology

lots of lectures and formal tutorials

assessed by essays, short notes and MCQs

Final MB

Final MB

Tomorrow’s Doctors 2009

Tomorrow’s Doctors 2009

Tomorrow’s Doctors 2009

So what are we really assessing?

Knowledge

Skills

Attitudes

So why are we really assessing?

Ranking of students

Measurement of improvement in a student

To diagnose student difficulties

Evaluation of teaching method

Motivating student to study

From Dr Kamrun Nahar, Mymensingh Medical College

Current medical courses

huge divergence of teaching and learning

methods

traditional lecture courses

enquiry-based learning

problem-based learning

case-based learning

e-learning

personal digital assistant

The Manchester Curriculum

Phase 1 (Y1+2) Laying foundations

Essential Skills – Sem 1

Life cycle - Sem 1

Cardio fit. - Sem 2

Mind and Movement – Sem 3

Nutrition & Metabolism - Sem 4

Early experience

Phase 2 (Yr 3+4) Developing clinical

competence

Yr 4

Mind and Movement

Families & Children

PPD Portfolio; Project Option

Yr 3

Intro to Clinical Learning (ICL)

Heart, Lungs & Blood

Nutrition, Metabolism & Excretion

Phase 3 (Y5) Consolidation - preparation

for practice; Teaching Hospitals & DGH;

Community; Electives; PPD Portfolio

Foundation years - Starting a career

Doct

or a

s Sc

ient

ist

and

Scho

lar

Doct

or a

s Pr

actit

ione

r

Doct

or a

s Pr

ofes

sion

al

Current medical courses

huge divergence of assessment methods

SBAs (MCQs), EMQs

modified essay questions

short answer questions

OSCEs

workplace-based assessments

high-tech simulations

Current medical courses

requirements of assessment methods

validity

reliability / generalisability

feasibility

fairness

educational impact

cost-effectiveness

acceptability

defensibility

Doctor as scientist and scholar

SBAs (MCQs), EMQs

context-rich scenarios

MSCAA question bank

modified essay questions

sequential presentation of information

open-ended question

short answer questions

SBA example

You are asked to perform an autopsy on a 54 year old alcoholic man who

was found dead in sheltered accommodation. Post mortem examination reveals a large quantity of blood in the upper gastrointestinal tract together with hepatic cirrhosis. The gastric and duodenal mucosa is

normal.

Identify the most likely source of the bleeding.

Option list

A Angiodysplasia

B Crohn’s disease

C Jejunal diverticulum

D Oesophageal varices

E Trauma to the mouth

Answer: Oesophageal varices (80%)

EMQ and SBA components

EMQ option list - list of possible answers - all realistic members of

set - 10 to 26 options

“traditionally”; some evidence that these lists should be shortened

- single words or short

phrases

MCQ option list

- list of possible answers

- limited members of set

- 4 to 6 options

“traditionally”; MSC-AA

uses 5

- single words, phrases or

even short sentences

• What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function?

A B *C* D E Overall P-value 1 0 99 0 0 94 8 1 90 1 0

• A 2-year-old boy has a 1-week history of oedema. Blood pressure is

100/60 mm Hg, and there is generalized oedema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. 0 0 98 2 0 88

5 2 82 8 1

• A 2-year-old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distension with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood.

0 1 98 1 0 84 10 9 66 10 5 From Susan M. Case & David B. Swanson: Constructing Written Test

Questions, 2002 http://www.nbme.org/about/itemwriting.asp.

21

MODIFIED ESSAY QUESTION MEQ

Please answer all the questions in sequence. Do not look through the book before you start.

Mrs. Halima, a 50 years old lady complains of lethargy, weight loss and excessive vaginal discharge, has been admitted to the gynae ward on which you work.

Q.1. What are the three most likely diagnosis. a.

b.

c.

Q.2. List three specific questions which would help you distinguish between these possibilities. a.

b.

c.

Q.3. A pap smear ordered by her general practitioner 2 month back revealed CIN II. Write down the clinical examinations you want to do. a.

b.

c.

Doctor as practitioner

OSCEs

workplace-based assessments

high-tech simulations

Our OSCEs

1 2 3

4

5 6 7

8

Resources needed

Healthy female volunteer, sat on a chair

Facilities for hand-washing

Jug of water and cup/glass (for the volunteer)

Examiner

Setting up the station

The chair should be positioned in such a way to

allow the candidate to examine the ‘patient’ both from

the patient’s right side and from behind

Approach to the examination

Key elements to include: Student introduces himself/herself (full name) Student checks the patient’s name Explains purpose of examination and obtains permission

Treats the patient with sensitivity and respect throughout the examination

What’s an UPSA?

• Undergraduate Practical Skills

Assessment

– should be done in the clinical setting if at

all possible

– may have to be done in the skills labs

– monitored via the portfolio process

What’s an UPSA?

• Undergraduate Practical Skills Assessment

– GMC list 32 procedures

– you have to do 27 UPSAs: some cover more than

one “GMC” procedure

– each has a guide and assessment form on Medlea

– assessed by professionals who do that task

routinely

– validated on Medlea

Doctor as Professional

clinical attachments

workplace-based assessments

OSCEs

portfolios

yellow card systems