Specialty*Selec+on*&*Future* Na+onal*Evalua+on*of ... ·
Transcript of Specialty*Selec+on*&*Future* Na+onal*Evalua+on*of ... ·
24/02/2012
1
Specialty*Selec+on*&*Future*Developments*
Dr*Alison*S*Carr*Senior*Clinical*Advisor*
Medical*Educa+on*&*Training*Programme*
What’s'in'this'presenta-on'
• Context*• Na+onal*Evalua+on*of*Specialty*Selec+on*(NESS)*report*
• Shortlis+ng*• Communica+on*skills*
• Reliability*• What*areas*of*selec+on*require*further*work*and*what*are*the*priori+es?*
Medical Education & Training Programme
Context'
Improvements'to'Specialty'Selec-on
To*pilot*new*ways*of*selec+ng*doctors*thoroughly*
Stakeholder*engagement*
To*develop*a*clear*evidence*base*before*introducing*new*methods*of*selec+on*
Small*incremental*changes*made*that*are*carefully*evaluated*
Medical Education & Training Programme
What'are'we'aiming'to'achieve'in'
Specialty'Selec-on?'
Cost- Effectiveness
Validity Acceptability Reliability
What'are'we'aiming'to'achieve?'
Recruitment*&*Selec+on*being:*• *Fair:*improved*transparency*&*equity*• *Valid*• *Reliable*• *CostNeffec+ve*with*limited*resources*• *Feasible*• *Defensible/*equal*opportuni+es*compliant*• *Collabora+ve*between*Colleges,*Deaneries*&*DH*
Recruitment*&*selec+on*based*on*bestNprac+ce*and*sound*governance*resul+ng*in*improved*quality*&*safety*for*pa+ents*through*selec+on*of*trainees*more*likely*to*be*effec+ve*doctors,*high*fill*rates*and*minimal*risk.*
Medical education & training programme
24/02/2012
2
The'Na-onal'Evalua-on'of''
Specialty'Selec-on'Project'
An*overview*and*presenta+on*of*selected*sensi+vi+es*within*the*NESS*report*
The'NESS'Project'
• Scope*o Selec+on*into*14*special+es*(mainly*at*ST1/CT1)*
o In*depth*analysis*in*5/13*deaneries*
o Evalua+on*mainly*in*2009*
www.mmc.nhs.uk
• Commissioned*in*2008*• Report*in*2010*
o Evaluate*Specialty*Selec+on*Pilots*(Wave*1*&*2)*in*2009*
o Addi+onal*brief:*report*on*na+onally*coordinated*recruitment*
Medical Education & Training Programme
NESS*covers*the*selec+on*processes*for*71%*of*4,060*hospital*training*posts*(England*only)*and*2,751*General*Prac+ce*(GP)*training*posts*(UK)*available*at*ST1/CT1*level*in*2009**
Aimed*to*answer:**
*• Acceptability:*Do*candidates*and*assessors*have*
confidence*in*the*selec+on*process?**• Reliability:*Would*the*outcomes*of*the*selec+on*
process*be*the*same*if*it*were*repeated*on*a*different*day,*with*different*ques+ons*and*different*assessors?**
• Validity:*Do*selec+on*scores*predict*performance*in*speciality*training?**
• CostHeffec-veness:'Does*the*selec+on*process*provide*value*for*money*as*determined*by*an*economic*evalua+on?**
Pilots'for'NESS'evalua-on'
'Wave'' Organisa-on' Specialty'and'level' Methodology'
1*(extended)* South*West*Peninsula*Deanery*
Anaesthesia,*ACCS*and*CMT,*CT1*
Machine*marked*test,*selec+on*centre*
1* RCP/KSS*Deanery* CMT,*CT1* Clinical*problem*solving*test*
1* Trauma*and*Orthopaedic*SAC*
Trauma*and*Orthopaedic,*CT1*
10*sta+on*selec+on*centre,*clinical*problem*solving*tests*
1*(extended)* RCS* Core*Surgical*Training,*CT1/2*
Selec+on*centre*
2** Mul+Ndeanery* Neurosurgery,*ST1*–*4* New*sta+ons*for*selec+on*centre*
2** West*Midlands* GP,*ST1*–*3* Cogni+ve*ability*test*
2** KSS*Deanery* GP,*ST1*–*3* Regional*selec+on*centres*
2** Mul+Ndeanery* General*Surgery,*ST3* Regional*selec+on*centre*
2** Mul+Ndeanery* Paediatrics,*ST1* Targeted*assessment*of*porfolio*
2** London* Paediatrics,*ST1* Knowledge*test*
NESS:'SpecialtyHDeanery'Matrix'
Specialty
Eas
t M
idla
nds
Eas
t of
Eng
land
K
SS
Lond
on
Mer
sey
Nor
th
Wes
tern
N
orth
ern
Oxf
ord
Pen
insu
la
Sev
ern
Wes
sex
Wes
t M
dlan
ds
York
s &
H
umbe
r
ACCS/Anaesthesia
�*
Cardiothoracic Surgery
�* �* �* �* �* �* �* �* �* �* �* �* �*
Core Medical Training
�* �* �* �* �* �* �* �* �* �* �* �* �*
Core Surgical Training
�* �* �* �* �* �* �* �* �*
Emergency Medicine
�* �* �* �* �* �* �* �* �* �* �* �* �*
General Practice �* �* �* �* �* �* �* �* �* �* �* �* �*General Surgery �* �* �*Histopathology �* �* �* �* �* �* �* �* �* �* �* �* �*Neurosurgery �* �* �* �* �* �* �* �* �* �* �* �* �*Obs & Gynae �* �* �* �* �* �* �* �* �* �* �* �* �*Paediatrics �* �* �* �* �* �* �* �* �* �* �* �* �*Psychiatry �* �* �* �* �* �* �* �* �* �* �* �* �*
24/02/2012
3
Data*sources*considered:*
! Documents*(academic*literatures,*specialty/deanery/College/MMC/DH*documents*and*websites*and*pilot*reports);**
! Surveys*(online*applicant*ques+onnaire,*selec+on*day*ques+onnaires*completed*by*candidates*and*assessors);**
! Interviews*(with*candidates,*assessors*and*specialty/deanery*representa+ves);**
! Score*data.**
Acceptability:*
Overall,*high*degree*of*trainee*sa+sfac+on*in*all*3*domains*surveyed:*
o Fairness*o FitnessNforNpurpose*o Organisa+on**
Reliability'H'Internal'consistency:'
! Ness*comment**Cronbach�s*alpha*is*widely*reported*as*a*measure*of*reliability,*despite*concerns*of*many*sources*of*variance,*e.g.*between*assessors,*not*considered*in*calcula+on*
! NESS*regard*an*alpha*>*0.7*as*a*necessary*but*not*sufficient*condi+on*for*high*reliability.**
! Cronbach�s*alpha*was*calculated*for*all*the*selec+on*systems*with*obtainable*data.**
! 7/*10*special+es*(centres*and*special+es*named)*had*a*mean*overall*alpha*lower*than*0.7,*which*is*a*cause*for*concern.*
How*reliable*is*the*decision*on*whether*to*offer*a*candidate*a*post?**
! The*fundamental*purpose*of*selec+on*is*to*have*confidence*that*the*candidates*offered*posts*are*beker*than*those*who*are*not.**
! Across*the*eight*special+es*where*this*was*es+mated,*there*was*considerable*uncertainty*as*whether*posts*should*or*should*not*have*been*offered*to*between*28*and*54%*of*candidates.**
Validity:*
! In*terms*of*the*predic+ve*validity*of*shortlis+ng,*aler*correc+ng*for*restric+on*of*range,*the*Pearson*correla+on*coefficients*for*the*specialty/*deanery*combina+ons*for*which*data*was*available*range*from*0.33*(CST*in*the*West*Midlands)*to*0.82*(O&G*in*North*Western),*with*a*mean*of*0.55.**
! None*of*these*assessments*of*validity*address*long*term*predic+ve*validity*and*the*issue*of*whether*the*selec+on*systems*iden+fy*those*who,*in*the*long*term,*are*bestNsuited*to*be*consultants*in*the*specific*specialty.**
Cost*effec+veness*
• Per*applicant*costs*ranged*from*£160*to*£1,100.**
24/02/2012
4
ST1/CT1'Shortlis-ng'in'2009'
Additional)Qualifications
Additional)Achievements
Achievements)Outside)Medicine
Activities)and)Achievements)showing)commitment)to)specialty
Commitment)to)Specialty
Clinical)Skills
Research)Skills/Management
Presentations
Publications
Clinical)Audit/Clinical)gov
Teaching)experience
Training)courses)attended
Team)working
Information/Technology
Career)progression
Skills)in)written)English
Personal)statement
Overall)excellence)of)application
Additional)Qualifications
Additional)Achievements
Achievements)Outside)Medicine
Activities)and)Achievements)showing)commitment)to)specialty
Commitment)to)Specialty
Clinical)Skills
Research)Skills/Management
Presentations
Publications
Clinical)Audit/Clinical)gov
Teaching)experience
Training)courses)attended
Team)working
Information/Technology
Career)progression
Skills)in)written)English
Personal)statement
Overall)excellence)of)application
419
2.511
3.528
3.518
4.514
2.517
ACCS
Anaesthesia
Microbiology
Ophthalmology
Radiology
Surgery
Additional2Qualifications
Additional9QualificationsMAXAdditional9QualificationsMIN
ST1'Regional'Recruitment'2009'
Scores*are*%*Total**Applica+on*Form*Score*
411
37
57
5&&&&&&&12.5
39
514
ACCS
Anaesthesia
Microbiology
Ophthalmology
Radiology
Surgery
Additional&Achievements
Additional9AchievementsMAXAdditional9AchievementsMIN
ST1'Regional'Recruitment'2009'
Scores*are*%*Total**Applica+on*Form*Score*
Scores*are*%*Total**Applica+on*Form*Score*
ST1'Regional'Recruitment'2009'
What'developments'have'been'made?'
24/02/2012
5
30
44
70
85
95
2007 2008 2009 2010 2011
Nationally2co4ordinatedrecruitment2(%)
125,000
15,540
85,000
10,009
33,202
7,073
24474
6799
16,044
6,697
2007 2008 2009 2010 2011
CT1/ST10Applications0per0year
CT1/0ST10Posts0per0year
Effects'of'Na-onal'Specialty'Recruitment' What'developments'have'been'made?'
• Collabora-on'with'Colleges,'Deaneries'&'
BMA/JDC'
• Electronic'applica-ons'via'College'or'Deanery'IT'
systems'
• Offers'period'• Evalua-on'of'selec-on'
processes'&'further'
refinements'
www.mmc.nhs.uk
Na-onal'recruitment'
• Standardisa+on*o Na+onal*Person*Specifica+ons*o Applica+on*form*o Scoring*matrices*o Selec+on*centre*
• Cost*effec+veness*" Applica+ons*" Interviews*per*specialty*
Medical Education & Training Programme
Pilots'of'selec-on'into''
Specialty'Training'(2008H10)'
Na+onal*Evalua+on*of*Specialty*Selec+on*(NESS),*University*of*Birmingham*– Wave*1*&*2*pilots*– Na+onal*processes*
South'West'Peninsula'Pilot'ST1:'
CPS/'SJT'
• *Anaesthesia*• *ACCS*Selec-on'Centre*• Anaesthesia*
CPS*Test**
What'are'we'trying'to'achieve?'
Predic-ve'Validity'
Significant'predictors'of:' GP' Anaes'
CT1'
CMT'
CT1'
CST'
CT1'
CPS/SJT* Selec+on*performance* �* �* �* �**
Exam*results* �* �* �*
InNtraining*performance* �*
EndNof*training*competence* �**
SC* Selec+on*performance* �* �*
Exam*results* �* �*
InNtraining*performance* �* �*
EndNof*training*competence* �**
Medical Education & Training Programme CPS*Clinical*Problem*Solving*Test*SJT*Situa+onal*Judgment*Test*SC*Selec+on*Centre
Predic-ve'validity'and*incremental'validity'of*selec+on*tools*in*predic+ng*performance*in*MRCP'
Exam' Predictor' R'values' P'value'
MRCP Part 1*
GP CPS SJT CMT shortlisting, CMT interview CMT final total*
r=.73; range across diets r=.53 to r=.79, r=.41 mean r=.50 mean r=.48 mean r=.53 *
All <.001**
MRCP Part 2*
GP CPS SJT CMT shortlisting, CMT interview and CMT final total*
r=.58; range across diets r=.52 to r=.74, r=.40 mean r=.32 mean r=.37 mean r=.38 *
< 0.05*All*<.001*
PACES* the CMT interview SJT CPS CMT shortlisting*
mean r=.44 r=.39, r=.38, mean correlation of r=.35*
<.001 <.001 <.001 <.001
Validation study used data from 2009 CMT selection process, incl. GP machine-markable tests, and MRCP outcome data (Part 1, Part 2 and PACES) since 2008
Patterson F, Carr V, Burr WA, Berkin L, Dacre J March 2011
What'further'developments'are'needed?'
24/02/2012
6
Interdeanery CMT Raw Interview Score Correlations 2009 Deanery sites
1 2 3 4 5 6 7 8 10
2 .74** N= 41
5 .76** N= 163
.59** N= 175
7 .72** N= 30
.68** N= 134
8 .69** N= 34
9 .50** N= 93
10 .46* N= 26
11 .41* N= 26
12 .62** N= 24
.74** N= 28
13 .77** N= 20
.76** N= 40
.73** N= 39
.57** N= 28
14 .72** N= 82
.89**
* Correlation significant at 0.05 level (2-tailed), ** Correlation significant at 0.01 level (2-tailed).
Pat
ters
on e
t al
Anaesthesia selection scores differ by deanery (Kruskal Wallis test, p<0.001)
16151413121110987654321
120
110
100
90
80
70
60
50
40
30
Deanery
Sele
ctio
n sc
ore
Scores are particularly high in deaneries 4 & 13 and particularly low in 5 & 9.
Interview scores CT1 Anaesthesia 2011
• For*applicants*interviewed*in*2*UoAs,*r=0.39*between*1st*and*2nd*na+onal*score*totals.**
• Corresponding*correla+ons*between*1st*and*2nd*structured*interview,*presenta+on*and*porfolio*scores*are*0.16,*0.18*and*0.41.***
There*is*a*slight*learning*effect*amongst*twiceNinterviewed*candidates.**
Ronny*Cheung1*Toni*Ardolino1*Alison*Carr2*
1Clinical*Advisor,*DH***2Senior*Clinical*Advisor,*METP*
Communica+on*skills**in*level*1*specialty*recruitment*in*2010'
Context*&*Methods*
• Disparity*of*assessment*methods*and*endpoints*
• Ques+onnaire*survey*to*12*special+es*
Current*prac+ces*• Scoring**• Assessment*methodology*
• Ongoing*evalua+on*• Examples*of*best*prac+ce*
• Assessment*stage*– Applica+on*stage:*3/12*– Interview*stage:*all*special+es*
• Separate*score?*– 8/12;*Range*10%*(CMT)*N*35%*(Paediatrics)*
• Separate*sta+on?*– 2/12*use*a*communica+onNonly*sta+on*
• 10*/12*special+es*would*not*reject*an*applicant*solely*based*on*an*unsa+sfactory*score*in*communica+on*skills**
Key*Findings:*Scoring*
24/02/2012
7
How*are*communica+on*skills*assessed?*
Anaesthetics/
AC
CS
CM
T
General P
ractice
Histopathology
Microbiology
Neurosurgery
Ophthalm
ology
Paediatrics
Psychiatry
Public H
ealth
Radiology
Surgery
Academic summary
# #
Handwriting #
Written exercise
# # #
Portfolio interview
# # #
Panel interview
# # #
Presentation # # # #
Clinical interview
# # # #
Simulated clinical scenario
# # # # #
Team exercise
#
Anaesthetics/
AC
CS
CM
T
General P
ractice
Histopathology
Microbiology
Neurosurgery
Ophthalm
ology
Paediatrics
Psychiatry
Public H
ealth
Radiology
Surgery
Academic summary
# #
Handwriting #
Written exercise
# # #
Portfolio interview
# # #
Panel interview
# # #
Presentation # # # #
Clinical interview
# # # #
Simulated clinical scenario
# # # # #
Team exercise
#
• Design*– 7/12*outlined*the*basis*for*design*– Expert*consensus*most*common*(6/7)**
• Evalua+on*– Informal,*itera+ve*appraisals*– 5/12*conduc+ng*formal*evalua+on*
Key*Findings:*Evidence*base*
• Explicit scoring for communication skills
• Minimum standard for recruitment • Formal evaluation processes • Best practices shared
– E.g. simulated clinical scenarios, use of lay interviewers
Recommendations for ST1/CT1 in 2013
Likely'areas'of'future'development'
• Need*to*address:'• Fairness*• Consistency*• Standardisa+on*• Content*development*• U+lity*
• Focusing*on*costN*effec+veness*
• Future*improvements*• Implement*standards*for*recruitment*&*selec+on*across*all*special+es,*levels*and*locali+es*
• Develop*consistency*• Validate*processes*• Train*raters*• Benchmarking*• Evaluate*processes*effec+vely*
Medical Education & Training Programme
UK'MaDRaS'
• Quality*SubNcommikee*• 3*areas*for*explora+on*in*2012/3:*
o Promote selection pilots to improve validity & reliability of specialty selection.
• include studies of predictive validity • further developing a multi-specialty MMT for selection into
ST1.*
o Analysis*of*training*and*benchNmarking*of*selectors*and*determina+on*of*minimum*requirements*
o Determining*the*minimum*evalua+on*of*na+onal*specialty*selec+on*
Medical Education & Training Programme
Summary'
Medical Education & Training Programme
Any'Ques-ons?'
• Context*• Na+onal*Evalua+on*of*Specialty*Selec+on*(NESS)*report*
• Shortlis+ng*• Communica+on*skills*
• Reliability*• What*areas*of*selec+on*require*further*work*and*what*are*the*priori+es?*