‘An objective evaluation of student selection’ · Alexander Todd Baron Todd of Trumpington...

‘An objectiveevaluation of

student selection’Has the time yet come?

Chris McManusDivision of Medical Education

UCL

UCL Medical Education Conference28th June 2011

Alexander ToddBaron Todd of Trumpington

1907-1997

Nobel Prize in Chemistry, 1957“"for his work on nucleotidesand nucleotide co-enzymes"

First to isolate Vitamin B12

• “We hope that research will in thecourse of time lead to the establishmentof reasonably objective criteria ofprofessional performance, valid in themany different fields and kinds ofmedical practice …

• … and thereby make possible anobjective evaluation of studentselection.

• This is a long-term aspiration however.”

Medical school selection

• Academic factors– A-levels

– Aptitude tests (BMAT, UKCAT, GAMSAT, etc)

• Non-academic factors– Personality

– Motivation

– Stress / Resilience / Burnout / Satisfaction

• Widening access– Complex issue

Selection for medical schools

• “Aptitude” tests– BMAT (UK)

– UKCAT (UK)

– GAMSAT (Australia and UK)

– MSAT (Australia)

– MCAT (US)

Selection for medical schools

• What is meant by ‘aptitude’?

• Tests are of two fundamental types:

– Achievement: Knowledge/content– A-levels

– BMAT: Section 2: Scientific knowledge and applications;

– MCAT biological sciences / physical sciences subtests

– GAMSAT: III: Reasoning in Biological and Physical Science

– Ability/Aptitude: Essentially IQ tests, although that term islittle used:

– UKCAT: Verbal reasoning/ Quantitative reasoning / Abstract reasoning/ Decision analysis

– BMAT: Section 1: Aptitudes and skills

– MCAT: Verbal reasoning subtest

– GAMSAT: I - Reasoning in the Humanities and Social Sciences

(1973)

Commissioned byCommittee of Vice-Chancellorsand Principals

The ISPIUA project: Investigationinto Supplementary PredictiveInformation for University Admission

Test of Academic Aptitude (TAA):Mathematics (M) and Verbal (V)

October 1967: 27,315 students inthe 4th term of the Sixth Form

The situation today

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

8 10 12 14 16 18 20 22 24 26 28 30

I

II.i

II.ii

III

Left

A-level points (A=10, B=8, C=6 etc)

Per

cen

tage

A-levels and university achievement, 2002(1999 entrants)

www.hefce.ac.uk/pubs/hefce/2003/03_32.htm

Sutton Trust SAT Study

• 9000 students took SAT in Autumn 2005

• 8000 matched to A-levels in 2006

• 2754

– Entered university 2006

– Graduated 2009

• 3800 still studying within HE


• Correlations with degree class (N=2754):

– Average A-level points score 0.38

– Average KS4 (GCSE) score 0.36

– Total A-level points score 0.34

– Mean SAT score 0.26

– SAT Writing score 0.26

– SAT Reading score 0.24

– SAT Maths score 0.18

ISPIUA


• Correlations with degree class (N=2754):

– Average A-level points score 0.38

– Average KS4 (GCSE) score 0.36

– Total A-level points score 0.34

– Mean SAT score 0.26

– SAT Writing score 0.26

– SAT Reading score 0.24

– SAT Maths score 0.18

Other aptitude tests….

• UKCAT

– Published evidence ‘mixed’

• BMAT

– Published evidence sparse and equivocal

• GAMSAT

– Published evidence from Australia suggestsno predictive value

Other medical schools

• Cambridge

• High A-level grades require:

• Intellectual ability

•Motivation / drive / study habits /learning styles / personality(conscientiousness)

•Content knowledge (e.g. biology /chemistry)

A-levels as predictors

Westminster Medical School Follow-up

• Dr Peter Fleming– Clinical entrants to Westminster, 1975-1982

– Age, about 21 (born 1954-1961)

– Timed IQ test (AH5): Verbal & Spatial scores

– N=511

• Followed up 1989 (McManus & Tunnicliffe)

– Qualified 1978-1985 (4-11 years previously;aged 28-35))

• Followed up 2002 (McManus, Smithers & Partridge)

– Qualified for 17-24 years

– Age, about 41-48

ALEVELS

5.33

5.00

4.67

4.33

4.00

3.67

3.33

3.00

2.67

2.33

2.00

1.67

1.33

100

80

60

40

20

0

Std. Dev = .71

Mean = 4.00

N = 508.00

TOTALJT

70605040302010

AL

EV

EL

JT

5.00

4.67

4.33

4.00

3.67

3.33

3.00

2.67

2.33

2.00

1.67

1.33

1.00

Westminster Hospital Follow-up• IQ and A-levels

62.5

60.0

57.5

55.0

52.5

50.0

47.5

45.0

42.5

40.0

37.5

35.0

32.5

30.0

27.5

25.0

22.5

20.0

100

80

60

40

20

0

Std. Dev = 7.74

Mean = 40.4

N = 511.00

Mean A-level grade

TotalAH5score

1716151413121110987654321

100

90

80

70

60

50

40

30

20

10

0

1716151413121110987654321

100

90

80

70

60

50

40

30

20

10

0

PRHO PRHOSHO SHO

Registrar

SR

Consultant GP Principal

Trainee/SpR

GP AssistantOther

Years after qualification Years after qualification

Hospital Doctors General Practitioners

Career progression

1716151413121110987654321

100

80

60

40

20

0

1716151413121110987654321

100

80

60

40

20

0

Memberships

Diplomas

Academicdegrees

Years after qualification Years after qualification

Hospital Doctors General Practitioners

Memberships

Academicdegrees

Diplomas

Post-graduate qualifications

17161514131211109876543210

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

Years after qualification

AAA, AAB

ABB, BBB, BBC

BCC, CCC or less

p<.001

A-levels, IQ and Memberships

Intelligence: p=.012 (after taking A-levels into account, p=.423)

The 1991 Cohort Study

EntrantsOct 1991

EntrantsOct 1992

N ~ 3300N ~ 590092%

ApplicantsOct-Dec 1990

Questionnaire

Leverhulme/DoH

N ~ 270060%

Final Year1996

Questionnaire

Final Year1997

Questionnaire

DoH

N ~ 270055%

PRHOsAug 1996-Jul 1997

Questionnaire


Questionnaire

NTPMDE

N ~ 240060%

SHO/SpRGPs

Dec 2002

Questionnaire

LondonDeanery

MRCP~1997-99

N = 92531%

March2009

N ~ 270061%

LondonDeanery

Questionnaire

Part 1mark at first attempt

80.075.0

70.065.0

60.055.0

50.045.0

40.035.0

30.025.0

20.015.0

10.0

100

80

60

40

20

0

Std. Dev = 11.34

Mean = 47.7

N = 925.00

MRCPmark

MRCP(UK) markat first attempt

5.55.04.54.03.53.02.52.01.51.0

Pa

rt1

ma

rka

tfirs

ta

tte

mp

t

75

70

65

60

55

50

45

40

35

30

25

20


EEE DDD CCC BBB AAA

A-level grade

A-levels

BasicMed Sci

BSc Class

MRCPmark

BSc Taken

GCSE

AH5


Non-cognitive aspects

Can selectors predict whowill be a happy

or an unhappy doctor?


EntrantsOct 1991

EntrantsOct 1992

N ~ 3300N ~ 590092%


Questionnaire

Leverhulme/DoH

N ~ 270060%

Final Year1996

Questionnaire

Final Year1997

Questionnaire

DoH

N ~ 270055%


Questionnaire


Questionnaire

NTPMDE

N ~ 240060%

SHO/SpRGPs

Dec 2002

Questionnaire

LondonDeanery

MRCP~1997-99

N = 92531%

Satisfaction with a medical career

0 1 2 3 4 5 6

0 1 2 3 4 5 60123456

181614121086420

60

50

40

30

20

10

0

Unhappy/Dissatisfied

Happy/Satisfied


EntrantsOct 1991

EntrantsOct 1992

N ~ 3300N ~ 590092%


Questionnaire

Leverhulme/DoH

N ~ 270060%

Final Year1996

Questionnaire

Final Year1997

Questionnaire

DoH

N ~ 270055%


Questionnaire


Questionnaire

NTPMDE

N ~ 240060%

SHO/SpRGPs

Dec 2002

Questionnaire

LondonDeanery

MRCP~1997-99

N = 92531%


EntrantsOct 1991

EntrantsOct 1992

N ~ 3300N ~ 590092%


Questionnaire

Leverhulme/DoH

N ~ 270060%

Final Year1996

Questionnaire

Final Year1997

Questionnaire

DoH

N ~ 270055%


Questionnaire


Questionnaire

NTPMDE

N ~ 240060%

SHO/SpRGPs

Dec 2002

Questionnaire

LondonDeanery

MRCP~1997-99

N = 92531%

March2009

N ~ 270061%

LondonDeanery

Questionnaire

Set A: 20 pairsone happy/ one unhappy

Set B: 20 pairsone happy/ one unhappy

All pairs aged under 21 at entry

All pairs matched for sex

96

19

35

22

20

Total

10.03

10.37

9.60

10.27

10.20

Mean nocorrect/20

4848Total

910Doctors

1916Selectors

1012Medical Students

1010Psychology Students

Set BSet A

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Correct

0

5

10

15

20

25

0

12

24

36

48

60

72

84

96

Performance of individual judges

So are the judges perhaps justresponding at random?

• Maybe the task is simply too difficult?

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48

Correct

0

1

2

3

4

5

0

4

8

12

16

20

24

28

32

36

40

Performance on particular pairs of doctors

95% CI

Significantlybetter than

chance

Significantlyworse than

chance

Consensus of judges

• +1 if answer consensus question insame way as other judges

• -1 if answer consensus question inopposite way to other judges

Time (minutes)

2402101801501209060300

Co

nse

nsu

sm

ea

su

re

100

80

60

40

20

Correlation of consensus measure with time taken

What differentiates successfullydiscriminated pairs fromunsuccessfully discriminatedpairs?

• No relationship to personality of doctors

– Big Five

– Empathy

• Correlation with educationalachievement

– Particularly with predicted A-levels

• (r=.321, p=.011, N=62)

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48

Correct

0

1

2

3

4

5

0

4

8

12

16

20

24

28

32

36

40

95% CI

Significantlybetter than

chance

Significantlyworse than

chance

‘An objectiveevaluation of

student selection’Has the time yet come?

‘An objective evaluation of student selection’ · Alexander Todd Baron Todd of Trumpington...

Documents

Transcript of ‘An objective evaluation of student selection’ · Alexander Todd Baron Todd of Trumpington...