Recruitment to Specialist Training

Recruitment tospecialist training:a resource guide for selectors

1. Objectives, Timetable & Workshop Slides

2. Resource Pack: Section AKey Concepts & Best Practice

3. Resource Pack: Section BNational Standards & Upskilling

4. Example Person Specification

5. Reference Report

6. Application Form Draft

7. Shortlisting Indicators & Example Responses

8. Example Interview Report Form

9. Upskilling in Selection Workshop: Selector Skills, Trainer Skills, Trainer Slides, FAQs

10. Evaluation & Audit Documentation

Contents

Nat

ion

alW

ork

sho

ps

National Workshops

Selection into Specialty Training: Upskilling selectors

National Specialty Selection Workshop: Methods & best practice for up-skilling selectors

9.30am Registration & Coffee 10am-4pm Workshop

Workshop facilitators Dr Sarah Thomas, Professor Fiona Patterson & colleagues.

Aims & objectives

To review the overarching principles and processes that have been agreed for national specialty selection

To review key concepts underpinning best practice selection and recruitment

To introduce the national documentation for specialty selection (person specification, reference report, application form)

To review the key skills required of selectors involved in selection (shortlisting & interviews)

To provide an opportunity to experience these skills using practical exercises

To provide supporting training materials for use by trainers in deaneries/ unit of application (UoA) in upskilling selectors for 2007.

To establish next steps in cascading training of selector skills in deaneries/ UoA’s.

Target audience & outcomes This workshop is aimed at those responsible for training selectors within their deanery/UoA in best practice selection principles and methods. The outcomes of the workshop are to 1) raise awareness and upskill participants in key concepts, principles and methods in best practice selection, such that they are able to deliver training effectively within their UoA (using a cascade approach); 2) to provide UoA representatives with relevant documentation and materials required for specialty selection; 3) to establish next steps in cascading training within UoA, including the role of the selection methodology team in the individual deanery visits.

Format

This is a participative workshop with a variety of practical exercises related to selector skills and will include plenary discussions, presentations and small group work.

Suggested pre-reading

Suggested pre-reading includes the documents attached. Those who have attended the planning and development meeting on 19th September will already have received these:

Document 1: Overarching framework for selection into specialty training Document 2: Overview of the development of the selection methodology presented to

PMETB for national selection

National Selection for Specialty Training

Methods & Best Practice for Upskilling Selectors

Timetable

10-10.20

Introduction

• Objectives & framework for National Specialty Selection

10.20-10.35

Selection Process Overview

10.35-11.15

Key Concepts in Best Practice Selection

• Research evidence • What’s different?

11.15

Tea/Coffee Break

11.30-11.50

National Documentation for Specialty Selection

• Person Specifications • Application form • Reference report & reference process

11.50-1pm

Selector Skills: Application Form & Shortlisting

• Applying indicators & rating scale to application form questions

1-2pm

Lunch

2-3pm

Selector Skills: Interviews

• Sources of error & bias, questioning styles • Applying indicators & rating scale to interview questions

3-3.15pm

Tea/Coffee Break

3.15-3.45pm

Implementation & Practical Considerations

• FAQs • Next steps

3.45-4pm

Review & Questions

4 pm

Finish

1

National Selection into Specialty Training:

Launch of Training Materials

17th October 2006

Key TopicsSelection Process Overview

− The plan for delivery− The Specialist Training programmes− The timeframe− How information is being cascaded

Key concepts in best practice selectionThe National Documentation for Specialty Selection

− Person Specification, Application Form, Structured Reference

Selector Skills− Short-listing, The interview

Implementation & practical considerations

Timetable:4 Sessions

Finish4 pm Review & Questions3.45-4pm

Implementation & Practical Considerations 3.15-3.45pmTea/Coffee Break3-3.15pm

Selector Skills: Interviews2-3pmLunch1-2pm

Selector Skills: Application Form & Shortlisting11.50-1pm

National Documentation for Specialty Selection11.30-11.50 Tea/Coffee Break11.15

Key Concepts in Best Practice Selection 10.35-11.15 Selection Process Overview10.20-10.35 Introduction 10.00-10.20

Framework for National Selection

Key Points http://www.mmc.nhs.uk/

UK Strategy Group agreed framework for national selection July 2006MTAS will be used by all 4 UK countries Nationally agreed person specs for each specialty & for each entry level (ST1, ST2, ST3) Nationally agreed application form, reference form, minimum standards for interview PMETB approval September 2006

Stakeholder & expert consultationStakeholder & expert consultation

Chairs of STCs

Chairs of STCs

Program Directors/

College Tutors

Program Directors/

College Tutors

COPMED/ COGPED Deaneries

COPMED/ COGPED Deaneries

Patient GroupsPatient Groups

BMABMANHS

EmployersNHS

EmployersDepartment

of HealthDepartment

of Health

GMCGMC

AoMRC, JACSTAGAoMRC,

JACSTAG

MMCMMC

Medical Directors/ HR/Trusts

Medical Directors/ HR/Trusts

Current Trainees/

Applicants

Current Trainees/

Applicants

PMETBPMETB

Stakeholders & Experts

Stakeholders & Experts

Ownership: Ownership: Are Are COPMeDCOPMeD/ Deans / Deans collectively signed up to the process?collectively signed up to the process?

COPMeDCOPMeD Steering Group for Recruitment & Steering Group for Recruitment & Selection into Specialty Training responsible for Selection into Specialty Training responsible for the work (reporting to UK Strategy Group)the work (reporting to UK Strategy Group)Close involvement of stakeholders in development, Close involvement of stakeholders in development, implementation & evaluation implementation & evaluation Deans have nominated Deans have nominated YOU to come here todayYOU to come here todayto to deliverdeliver implementation in your localityimplementation in your localityDeans will oversee evaluation of selection panelsDeans will oversee evaluation of selection panels

2

What Specialist Training Programmes will be available?

All recruited to afterF2 and during transition from thecurrent SHO pool

Medicine in General

Basic Neuroscience Training

Acute Care Common Stem

Histopathology Chemical Pathology Medical Microbiology

Obstetrics & Gynaecology

Paediatrics

Anaesthesia

Psychiatry

Radiology

Public Health

General Practice

Surgery in General OFMS Ophthalmology

Core Run Through Training

Medicine in General



Psychiatry

Surgery in General

Core


Paediatrics

Anaesthesia

Ophthalmology

Existing SHO programmes offer core training �

Recruit in 2007 to:

ST1

ST2

ST3

ST4+

Non-Core Run Through Training from ST1

Histopathology

Chemical Pathology

Medical Microbiology

Radiology

Public Health

General Practice

OFMS

Non-CoreCurrently do not have core training at SHO gradeIn 2007 will recruit to ST1 from F2 and SHO gradesGeneral Practice will recruit at ST1-3 in 2007

Specialist Training

Histopathology

Chemical Pathology

Medical Microbiology

Radiology

Public Health

General Practice

OFMS

Non-CoreMedicine in General



Psychiatry

Surgery in General

Core


Paediatrics

Anaesthesia

Ophthalmology

What levels will be selected into Specialist Training in 2007?

Specialty-specific − Levels recruited to depend on comparison

between competencies detailed in the new & old curricula

Recruitment will occur at ST1- ST4 in some Specialties

− Distinct application forms− Different recruitment rounds

How do trainees apply for How do trainees apply for Specialist Training?Specialist Training?

Advance information in Medical PressAdvance information in Medical PressSingle alert in Medical Press & electronic Single alert in Medical Press & electronic informationinformationSingle electronic application per recruitment Single electronic application per recruitment roundroundLongLong--listed nationally against person listed nationally against person specification specification Apply for up to 2 preferred Specialty groups in 2 Apply for up to 2 preferred Specialty groups in 2 preferred Units of Application* preferred Units of Application*

−− GP 1 GP 1 UoAUoA−− May no longer express ‘no geographical preference’May no longer express ‘no geographical preference’

*Unit of application- a Deanery or cluster of Deaneries working together

3

Recruitment & SelectionNational Advert

Central Co-ordination of Offers of Posts:0 4321

Successful

Round 1

Run-Through Grade Posts:2 Specialties,2 UOA &FTSTAs

E- Applications co-ordinated & sent to chosen UOAs*

Applications co-ordinated &sent to chosen UOAs*

Assessment RankingSelection

Shortlisting

Central Co-ordination of Offer of Post

Unsuccessful

Vacant Run-Through Grade Posts& FTSTAs

Round 2

UnsuccessfulRe-enter

Shortlisting Shortlisting ShortlistingShortlisting

AssessmentRankingSelection




Appointed toTraining

programme

Offers to Applicants

Training Programme

10 VacanciesAppointable

TraineesRanked 1- 45

1

45

Applicants will receive a single e-letter listing their offers and will have a limited period in which to reply

o Acceptance is a binding commitment to the UOA and the employer

o Applicants breaking the agreement are disqualified from any further participation in the recruitment and selection process except in very exceptional and unforeseen circumstances

Ranking of Applicants

Training Programme

10 Vacancies45 Trainees Appointable

Ranked 1- 45

1

45

There will be a continuous process of issuing more offers to appointable candidates in Round 1 of Selection

TimeframeAugust 2006-7

Aug Identify No of Programmes/ Posts Sept Agree person specification for ST1, 2 & 3

Jan Finalise complete selection/ interview schedule 6 Jan Alert in Medical Press- Application start

22 Jan- 2 Feb Submit applications2 Feb Application form to selection panels

By 23 Feb Shortlist applicants24 Feb One email confirming interviews or not

27 Feb Trainees confirm attendance or not28 Feb- 13 Apr Complete selection activities:Identify appointable applicants

2007

06

19-26 Apr Round 1 Offers made & accepted27 Apr Trainees with no offers asked to reapply

28 Apr Advertise unfilled postsBy 27 May Shortlist applicants

23 June One email of offers/ rejections4- 22 June Complete Selection

26 June Acceptance/rejection of offer

Round 1

Round 2Indicative

Implications of TimeframeSelectors will need to have the following availability:

Provisional DatesShortlisting

− 2- 23 Feb− 14- 27 May

Selection− 28 Feb- 13 April− 4- 22 June

Selection Panel should be available to complete short-listing within the timeframe as well as to attend the selection process for candidates

Feasibility: Feasibility: Can the Selection process be rolled out uniformly?

Mechanisms for 2007Targeting of agreed selection criteria within specialtiesNationally agreed documentation, appeals system UoA leads for selection identified using train-the-trainer cascade. Today is Phase 2 – Launch of training materials.Auditing & evaluation procedures, annual QA report

4

Feasibility: Feasibility: Are there barriers to implementation in particular areas?

Potential barriersUnits of Application with differing needs (→training tailored accordingly)Timeframe & resourceResistance to change?

Selection Methodology Cascade

Phase 1:Phase 1: Concepts & Planning (19 Sept)Concepts & Planning (19 Sept)Phase 2Phase 2:: Launch of Training Materials (3, Launch of Training Materials (3,

10, 17 Oct)10, 17 Oct)Phase 3:Phase 3: Individual Deanery visits Individual Deanery visits -- support support

for roll out as appropriate (Oct & for roll out as appropriate (Oct & Nov)Nov)

Scope of Our WorkScope of Our Work

Phase 1Phase 1 Person Specs for National Selection 2007Person Specs for National Selection 2007Phase 2Phase 2 PrePre--application information for candidatesapplication information for candidatesPhase 3Phase 3 Methods design: References, Application form & Methods design: References, Application form &

InterviewInterviewPhase 4Phase 4 UpskillingUpskilling assessors/adminassessors/adminPhase 5Phase 5 EvaluationEvaluation

Consultation (especially trainees)Consultation (especially trainees)

Our role: To advise on Selection methods for entering Specialty training

BackgroundBackgroundIn In ““Proposals for Reform of the SHO gradeProposals for Reform of the SHO grade””Sir Liam DonaldsonSir Liam Donaldson argues that: argues that:

““Reform must take account ofReform must take account of……weak selection weak selection && appointment procedures: appointment procedures: these are not standardised these are not standardised & & are frequently are frequently not informed by core competenciesnot informed by core competencies’’. .

UK Strategy Group agreement to national selectionPMETB Panel review 25th August 2006

Why Change?Why Change?

Historically we have done an effective job.Historically we have done an effective job.Little formal evaluationLittle formal evaluationOur aim is to Our aim is to improveimprove efficiency & effectiveness efficiency & effectiveness for for allall stakeholdersstakeholders (trainees, admin, interviewers, HR)(trainees, admin, interviewers, HR)−− cost, fairness, reliability & validity (standards)cost, fairness, reliability & validity (standards)

ST1 is new: We ST1 is new: We mustmust deliver a process that is ‘fit deliver a process that is ‘fit for purpose’for purpose’

WhatWhat’’s new?s new?

This is the This is the firstfirst step. For 2007, there will be step. For 2007, there will be uniform targeting of agreed selection uniform targeting of agreed selection criteria (via the criteria (via the Person SpecificationsPerson Specifications). ). Minimum requirement of a Minimum requirement of a 30 minute 30 minute structured interviewstructured interviewIdentifying Identifying best practicebest practice (no re(no re--invention)invention)Resource pack Resource pack & opportunities to further & opportunities to further upskillupskill selectorsselectors

5

Research on Selection Systems

Research & implementation projects to Research & implementation projects to develop selection tools that are develop selection tools that are standardisedstandardised, , fair, defensible, reliable, valid, costfair, defensible, reliable, valid, cost--effective & effective & feasible. feasible. 1. Develop & validate selection criteria for various

specialties2. Design & validate selection tools that assess core

criteria

Growing literature in medicineGrowing literature in medicinePatterson et al. (Patterson et al. (BMJBMJ, 2005; , 2005; BMJBMJ 2001; 2001; BJGPBJGP, 2000; , 2000; Arch. Dis. Child. Arch. Dis. Child. 2006; TOG in press, 2006, RCS)2006; TOG in press, 2006, RCS)

Research on selection methodsResearch on selection methodsStructured interviews can be reliable & valid selection tools. Structured interviews can be reliable & valid selection tools. Reliability of Reliability of assessmentassessment & of & of assessorsassessors using using standardisedstandardised rating scales & documentation. Rigorous rating scales & documentation. Rigorous trainingtraining is critical. is critical.

Research shows Selection Centres (designed Research shows Selection Centres (designed appropriately) are best predictor of future job performanceappropriately) are best predictor of future job performance

RReliabilityeliability && validity validity gains gains -- combining different selection combining different selection tools, tools, standardised scoring systemsstandardised scoring systems to measure to measure key key competenciescompetencies egeg. 3 x 10 minute stations (= 30 minutes, with 2 . 3 x 10 minute stations (= 30 minutes, with 2 assessors per panel)assessors per panel)

Research Research consistently demonstrateconsistently demonstratess that application form that application form datadata & references& references hahaveve limited validitylimited validityValidityValidity can be can be significantly significantly improvedimproved by providing by providing standardised standardised competencycompetency--based based rating scales rating scales

Defining reliability & validity

Reliability: “The instrument measures consistently under varying conditions”

Validity: “The instrument measures what it claims to measure”

Evaluating validity in selection1.1. Faith validityFaith validity

““The person who The person who sold me the sold me the selection tool/ test selection tool/ test was very plausible”was very plausible”

2.2. Face validityFace validity““The test looks The test looks plausible”plausible”

3.3. Content validityContent validity“The test looks “The test looks plausible to experts”plausible to experts”

4.4. Criterion validityCriterion validityPredictive validityPredictive validity

“The tool/test predicts who “The tool/test predicts who will be competent doctors”will be competent doctors”

5.5. Incremental validityIncremental validity“How much additional value “How much additional value does using another test/tool does using another test/tool provide?”provide?”

6.6. Construct validityConstruct validity““The tool measures The tool measures something meaningfulsomething meaningful””

Research on selection in medicineResearch on selection in medicinePractically, for some specialties we can’t interview all applicants.We need to shortlist. Options?– Psychometric testing, MCQs?

Academic success predicts very little of the variance in Academic success predicts very little of the variance in subsequent clinical performance subsequent clinical performance (e.g. Ferguson et al, BMJ; 2002)(e.g. Ferguson et al, BMJ; 2002)

For ST2, ST3 exams/research more relevant to indicate For ST2, ST3 exams/research more relevant to indicate competence. The is not possible for differentiating at ST1competence. The is not possible for differentiating at ST1

We recommend a combination of questions to add We recommend a combination of questions to add predictive power predictive power –– this has been pilotedthis has been pilotedMTAS can search for plagiarismMTAS can search for plagiarismFor 2007, this methodology is fit for purposeFor 2007, this methodology is fit for purpose..For the future, evaluation is necessary, on a national basis. For the future, evaluation is necessary, on a national basis. Future innovations could be a Machine marked test for Future innovations could be a Machine marked test for shortlistingshortlisting

Candidate reactionsFactors affecting reactions use organisational justice theories:

− Distributive justice: Perceived fairness regarding equity (outcome consistent with expectations) & equality (opportunities).

− Procedural justice: Information, feedback, job-relatedness, recruiter effectiveness.

Applicants prefer multiple opportunities to demonstrate skills

6

National Documentation-The Person SpecificationTab 4

Person SpecificationsPurpose

− Usability (especially candidate perspective)− Standardisation within specialties− Operationalisation of assessment (e.g. AF,

interview questions, scoring framework)

Two sections:−Eligibility Criteria:

Minimum entry requirements, all essential−Selection Criteria:

Used to rank candidates Essential/ desirable criteriaSpecialty-specific criteria (some common domains)Commitment to the specialty

National Documentation-The Structured ReferenceTab 5

Reference Reports

Standardised, structured report form nationally agreedMinimum of 2 refereesGuidelines in how reference report used in the selection process

National Documentation-The Application FormTab 6

Application FormsApplication Forms

EligibiltyEligibilty−− Personal details, GMC Registration, Professional Personal details, GMC Registration, Professional

Qualifications, Eligibility for entry into Specialist Qualifications, Eligibility for entry into Specialist training, Previous posts, Refereestraining, Previous posts, Referees

Selection CriteriaSelection Criteria−− Clinical, Academic & Research skills, Probity, Clinical, Academic & Research skills, Probity,

Evidence of commitment to the Specialty, Personal Evidence of commitment to the Specialty, Personal skills relevant to the Specialtyskills relevant to the Specialty

Currently undergoing substantial development

Questions directly linked to person specificationQuestions directly linked to person specification

7

Application ProcessApplication Process

Specialty specific formsSpecialty specific formsLong listing via MTAS (electronic portal)Long listing via MTAS (electronic portal)ShortlistingShortlisting in in UoAUoA by trained by trained shortlistersshortlisters(minimum of 2)(minimum of 2)Lay involvement, determined by the Lay involvement, determined by the UoAUoAScoring frames providedScoring frames provided

Shortlisting Indicators Tab 7Reference Framework for Target Selection Criteria

Framework for Target Selection CriteriaClinical, Academic & Research Skills

Short-listing Process All interview panel members (or a sub-set) carry out shortlisting (Orange Guide)

− Panel members do not have to mark ALL of a single application form.

− Deaneries may choose to split questions between panel members.

Benefits in terms of speed & calibrationDifferent methods/options:

− All in one room – marking/ decisions− Sent out to short-listers & returned− electronic

GMC reporting for falsification added to Application Form

Rating Scale

Applicant responses scored on 4-point scale:4 = good to excellent3 = satisfactory 2 = areas of concern 1 = poor [0] = No evidence

Evaluation & rating process

Standardised system Standardised system of documentation & of documentation & scoring framework scoring framework allows for more allows for more validvalid & & fair assessmentsfair assessmentsReduces potential for error & biasReduces potential for error & biasIncreases transparencyIncreases transparency

8

Selector Skills Exercise 1

Evaluating & scoring the application form

Practical considerations in Shortlisting

Calibration and reliabilityCalibration and reliabilityEach question marked by 2 different short Each question marked by 2 different short listerslistersReturning scoresReturning scores

Selector Skills - The Interview

Interviews in SelectionInterviews in Selection

2 main types of interview in selection− Behavioural (past behaviour = best predictor of future behaviour)

− Situational (scenarios)

− Both useful, dependent on interviewer skills

Skills: Peeling, probing, familiarisation, observerecord, classify, evaluate

Example Situational Interview Question Negative Indicators

• insist on operating• threaten a critical incident• consults senior anaesthetist

without discussing with SpR• tell the patient anaesthetist

“refusing” to anaesthetise• just accepts it

Positive Indicators• consider the reasons (eg

overnight, co-morbidity)• review the decision to operate

and assess priority• discuss it with the anaesthetist• involve the surgical senior cover• deal with the patient

appropriately

EVALUATION 1 2 3 4

Management Scenario:Anaesthetic SpR refuses to anaesthetise a patient for appendicectomy

9

Negative Indicators• Failing to take a history of the nature

of the accident• Failure to take adequate history

including possibility of asthma• Failure to examine patient including

trachea • Sending patient for CXR without

excluding tension pneumothorax• Failing to assess circulation in arm

Positive Indicators• Structured approach to the question• Awareness of possible vascular

injury to arm • Awareness of possible

pneumothorax• Of risk of associate intra-abdominal

injury – ruptured spleen • Awareness of risk of head injury

EVALUATION 1 2 3 4

Clinical Scenario: 12 yr old boy – who has come off his BMX –he has a broken arm (supracondylar fracture) and is complaining of breathlessness.

Familiarise

Interviewers need to know: Interviewers need to know: −− the questions they are askingthe questions they are asking−− how they relate to the person specification how they relate to the person specification −− what probes are appropriatewhat probes are appropriate

Be familiar with the relevant indicators for Be familiar with the relevant indicators for the questionthe questionBe familiar with the rating scale to be usedBe familiar with the rating scale to be used

What’s the difference between a skilled interviewer & novice?Questioning styles…..Questioning styles…..open questions, one at a timeopen questions, one at a timeManaging silences….Managing silences….Making the question feel authentic! FollowMaking the question feel authentic! Follow--up up essential to uncover what “you” (the candidate) essential to uncover what “you” (the candidate) actually didactually did

PeelPeel the layers the layers 1.facts 2. meaning 3. attitude1.facts 2. meaning 3. attitude

ProbeProbe for evidence for evidence who, what, why, when…etcwho, what, why, when…etc

Verify what the trainee actually did, or would do Verify what the trainee actually did, or would do in scenariosin scenarios

321

Example Interview Report FormTab 8


Questioning StylesQuestioning Styles

Observation: Sources of error & bias

Haloes & Horns effectsHaloes & Horns effectsPrejudice (race, gender, etc.)Prejudice (race, gender, etc.)Stereotyping & ‘Red Rags’Stereotyping & ‘Red Rags’ConcentrationConcentration

−− ‘Primacy & recency’ effectseffects

Observation biasObservation bias−− ‘Similar-to-me effect’/ Personal Liking

Temporal extensionTemporal extension− Decision made within 3 seconds

First impressions last First impressions last − “Can I suspend judgement?”

10

‘Real World’ Selector Stereotypes

white stilettoswhite stilettoslimp handshakeslimp handshakeswhite sockswhite socksmale earringsmale earringstattoostattoosMillwallMillwall fansfansmale long hairmale long hairsweaty palmssweaty palms

designer stubbledesigner stubblecloseclose--set eyesset eyesdoubledouble--barrel barrel surnamessurnamesVolvo driversVolvo driversperoxide hairperoxide hairheavy makeheavy make--upupbelly showing!belly showing!

RecordingLegible!AccurateComplete

− If in doubt, include itNon-judgmental

− Record appropriately, do not judge or evaluate while recording

Specific – need to be able to provide explanation for judgment later on

Selector skills Exercise 3

Effective recording & evaluatingEffective recording & evaluating

Evaluation & Rating Process

4 = good to excellent3 = satisfactory 2 = areas of concern 1 = poor [0] = No evidence

Standardised system of markers allows formore valid & fair assessments & increases transparency

Interviews/Selection Centres

MethodsMinimum of 30 minute structured interviewQuestions/exercises must be;− Based on the relevant Person Spec. − Consistent across interviewers/ interviewees− Previously agreed scoring framework that links to the

Person SpecExample questions & calibration exercises provided

Interviews/Selection CentresLogistics

Chairperson/ facilitator appointedVenue, admin support, materials planned in advance

ContentMinimum of 30 minutes contact− Panel interview; Split-panel (two 15 minutes); Three

10 minute exercisesMultiple selection tools encouragedSelection centres (focus also on demonstrating, not reporting)

SelectorsAll trained, lay involvement essential

11

National Documentation-Candidate evaluationTab 10

FeedbackFeedbackSelection decision uploaded to MTAS; offers Selection decision uploaded to MTAS; offers ONLYONLYby MTAS at the end of the selection process, or the by MTAS at the end of the selection process, or the national system will not worknational system will not workFeedback offered Feedback offered only afteronly after the whole of the the whole of the selection process. selection process. This is This is essentialessential so there is fairness to all so there is fairness to all candidatescandidatesWe must ensure fairness to all candidates We must ensure fairness to all candidates UoAUoA s will be asked to s will be asked to auditaudit to ensure national to ensure national adherence adherence


Practical considerations

TimetableLogisticsCandidate experience

ReviewNational specialty selection in 2007 includes national National specialty selection in 2007 includes national person specifications, application forms & referencesperson specifications, application forms & referencesKey conceptsKey concepts in best practice selection inform new in best practice selection inform new practicepracticeShortShort--listing & interview questions listing & interview questions MUSTMUST be directly be directly linked to person specslinked to person specsSelector skills training vital to contribute to Selector skills training vital to contribute to standardised, fair, defensible, reliable, valid, coststandardised, fair, defensible, reliable, valid, cost--effective & feasibleeffective & feasible process.process.Selection system not static Selection system not static –– continuous continuous improvementimprovement

Final questions?

Thank you……

Res

ou

rce

Pac

k;S

ectio

n A

Resource Pack: Section A

1

Section A. Key Concepts and Best Practice

1. Background

2. Why Change?

3. What’s New?

4. Person Specifications

5. The Selection Process

6. Research on Selection Methods

7. Application Process 8. Interviews/Selection Centres

9. Reference Reports

10. Feedback Attachment 1: Orange Guide Excerpt. Attachment 2: PMETB Generic Standards for Training Excerpt.

2

Across all four UK countries, specialties currently apply selection criteria inconsistently. For 2007, there will be uniform targeting of agreed selection criteria, nationally agreed documentation and a nationally agreed appeals system. This is a first step towards achieving national selection for all specialties. This document provides guidance for staff involved in delivering the selection process for specialty training. Minimum standards are presented and a guide to best practice at each stage of the selection process is outlined. By following this guidance, the aim is to deliver a selection processes for specialty training is efficient, valid and fair. The Resource Pack is designed to allow Units of Application [UoAs]/Deaneries to evaluate their current selection processes against best practice in selection. 1. Background Postgraduate medical education and training has undergone a period of rapid change, partly to ensure standards for safety, efficacy, and accountability of health care. In 2002, the Chief Medical Officer for England, Sir Liam Donaldson, highlighted, in his report called “Proposals for Reform of the SHO grade,” the need for best practice selection:

“Reform must take account of… weak selection and appointment procedures: these are not standardised and are frequently not informed by core competencies”.

Since Sir Donaldson’s report, the four UK Health Departments published a policy statement on “Modernising Medical Careers” (MMC), which set out principles for a major reform in postgraduate medical education. The MMC programme places emphasis on delivering robust methods for assessing and developing doctors throughout their career life-cycle. Implementation of MMC has led to trainees making career decisions earlier, so the selection systems must be robust and transparent. The main regulatory body behind these planned changes is the Postgraduate Medical Education and Training Board (PMETB). The PMETB, established in 2003, is independent of government. It bears responsibility for the standards and quality assurance of all postgraduate education, training and assessment in medicine in the UK. In August 2006, PMETB reviewed the principles for national selection and the key concepts are presented here. 2. Why Change? Obviously, selection has been effective in the past. However, there is scope for improvement. Specifically, the objective for 2007 is to improve the efficiency and the effectiveness of the selection system and methodologies used therein. Although considerable effort will be spent on delivering these changes, achieving national selection will have significant advantages for all stakeholders (trainees, admin, interviewers, HR). For example, for many years there have been ‘multiple applications’, where trainees complete many applications to different training programmes for the same specialty throughout the UK. This potentially duplicates effort for applicants, administration teams, HR and various staff conducting shortlisting and interviews. By having a national portal for applications, valuable resource will be saved. 3. What’s New? The intention is to build on good practice and current expertise - not to re-invent! For 2007, there will be a train-the-trainer cascade of the new national documentation for speciality selection, taking place in the various Units of Application (usually Deaneries). Specifically, there will be nationally agreed Person Specifications, speciality specific shortlisting procedures and a minimum standard for the interview process. In addition, auditing and evaluation procedures will be introduced. The auditing and evaluation process is overseen by the Postgraduate Deans and used to identify areas for modification and improvement. 4. Person Specifications A Person Specification describes the qualifications, skills, experience, knowledge and other attributes a candidate needs in order to perform effectively on the training programme. Person Specifications are the cornerstone of the selection process and should be derived from a thorough analysis of the post in question. The Person Specification outlines the selection

3

criteria that will form the basis of selection decisions at shortlisting and interview/selection centre. Selection criteria must be specific, justifiable, measurable, fair and pitched at a level appropriate for the training programme. The content of interviews/selection centre exercises must derive from the person specification and be designed to elicit evidence from candidates in relation to the criteria. All selection tools used (whether at shortlisting or interview) must be directly relevant to the criteria listed on the Person Specification. 5. The Selection Process The four UK countries have agreed to a common process and further detail is available via the MMC website at http://www.mmc.nhs.uk/. The recruitment process for specialty training is summarised in Figure 1. Key features to note are that applications and longlisting process will be delivered via a centralised electronic portal (called the Medical Training Application Service; MTAS). There is a minimum standard of a 30 minute structured interview. However, other options are encouraged such as split panel interviews, and selection centres1. Note that additional, specific training is essential before selectors can use a selection centre approach.

Figure 1. The Selection Process

1 Selection ‘centres’ are a process, not a place, involving multiple selection activities such as simulations, group challenges and structured interviews.

Advert & Pre-application Information

Pre-application information for each specialty accessible via MTAS, including; Brief description of the specialty, attraction & challenges, future of the specialty A trainee’s view Person specification: Eligibility and Selection criteria Information on selection methods, the selection process Links to relevant websites, references, etc

Application Form Submitted Long listing against eligibility criteria via MTAS

Short Listing Person specification criteria must be used so that applications can be measured against

the published selection criteria

Interview/Selection centre

Minimum requirement: 30 minute, structured interview Options: 2 split-level panel interviews (15 minutes each, 3 selectors per panel, lay

involvement) OR 3 X 10 minutes interviews, 2 selectors per panel. For a specialty, the same competencies measured across all UoA Selectors use national documentation with validated scoring framework

References used

Allocation

References collected Structured, competency-based rating scale, used a pre-allocation check

4

6. Research on Selection Methods There has been a considerable amount of research conducted (over many years) on developing robust selection criteria and methodologies in medicine (e.g. Patterson, 20052). The ongoing research and development aims to develop tools and techniques for selection that are standardised, fair, defensible, reliable, valid, cost-effective and feasible. This body of research has been used to develop selection criteria across all specialties for 2007. 7. Application Process

LongListing & Eligibility Criteria • The first step is to ensure candidates meet all the eligibility criteria for the training

post at the appropriate level. For example, for entry to ST1, applicants must have F2 competencies. Candidates who do not meet all eligibility criteria will be rejected at this stage (and do not proceed to shortlisting).

• This process will be conducted by the electronic system via MTAS. Certain criteria might be flagged so they can be assessed at interview stage (e.g. evidence of F2 competencies for some candidates might be assessed at interview stage).

• Candidates who meet all eligibility criteria will go through to shortlisting and assessed against the relevant selection criteria.

Shortlisters • All staff involved in the shortlisting process must be trained in the selection principles

and the shortlisting process. • There should be a minimum of 2 shortlisters per application form. To improve

reliability, shortlisters could chose to divide the application form questions (where a shortlister focuses on one or more questions across an applicant pool). Ideally, responses to individual questions should be separated and marked by different shortlisters.

• Ensure that all applications are treated confidentially and that applications are only circulated to those involved in the shortlisting process.

Shortlisting & Shortlisting Criteria • Applications will be sent to the relevant UoA for shortlisting. • When shortlisting, each candidate must be assessed objectively and consistently

against the selection criteria identified in the relevant person specification. Candidates should only be assessed against these criteria.

• Scoring frameworks are provided for each criterion and must be applied consistently across shortlisters. Training and calibration is essential in this respect.

• Selectors must avoid making assumptions about candidates’ skills, experience or qualifications without demonstrable evidence.

• Candidates are scored on a scale of 1-4 for each criterion (1=poor, 2=areas of concern, 3=satisfactory, 4=good to excellent, and 0 if no evidence is presented).

• Shortlisting criteria must be consistently applied. This is the responsibility of the UoA. • Marks for all selection criteria should be combined to create a final total that will be

used to rank candidates. (Eligibility criteria are not included in the ranking process). • A cut-off score for inviting candidates to interview/selection centre should be agreed

by the panel and applied consistently in the UoA. 8. Interviews/Selection Centres

Methods • The aim of the interview/selection centre is to determine candidates’ suitability for the

post by obtaining detailed evidence to judge them against the relevant criteria. • Interviews/selection centres also give candidates an opportunity to find out more

about the specialty/programme/deanery and thus enable self-selection. 2 Patterson et al (2005) A new selection system to recruit GP registrars: Preliminary findings from a validation study. British Medical Journal.

5

• Research clearly demonstrates that interviews can represent an important and valid means of selecting candidates provided it is structured and interviewers are trained appropriately.

• For best practice, interviewer questions must be: based on the relevant Person Specification, consistent across interviewers and interviewees based on the interviewer panel using a consistent set of indicators and

scoring criteria to evaluate interviewee responses. • The two most common forms of structured interview used in selection are:

Behavioural Interviewing and Situational Interviewing. Behavioural interviewing involves asking interviewees to describe previous

behaviour in past situations that are job relevant.

Situational interviews present interviewees with hypothetical job-related situations/scenarios and ask them to indicate how he or she would respond.

• A combination of both types of questions could be used alongside questions about

commitment to the specialty. In all cases interviewers must use appropriate scoring frames, detailing a good, average and poor response (as appropriate) in advance of the interview taking place. Using indicators to rate interviewee responses, ensures greater reliability and consistency across interviews and interviewers. Example scoring keys and rating scale using indicators are provided in the resource pack.

• All interviewers must be trained in interview skills and equal opportunities.

Logistics & Panel Composition • Advance planning is essential to ensure the necessary staff, administrative support,

venue, equipment and materials are available for the interviews/selection centres. • The Orange Guide provides specific guidance on the appointment panel composition.

A chairperson must be appointed to ensure the interview/selection centre is conducted professionally and fairly and that questions/exercises are within best practice guidelines (see Attachment 1 below).

• Lay involvement is essential as a requirement of the PMETB Generic Standards for Training (see Attachment 2 below).

• The interview panel (or facilitators) are responsible for ensuring the interview/exercise runs to time.

• Sufficient time should be allowed between interviews/exercises for selectors to complete scoring sheets/report forms. Similarly, sufficient time for rest breaks for selectors and candidates should also be built into the schedule (as appropriate).

• Each interview/exercise should take place in a different room where possible. Where this is not possible, soundproof screens must be provided to ensure fairness.

• A selector or facilitator should be responsible for welcoming candidates and explaining the process for the interview/exercise. Candidates must be made aware of the (maximum) length of each interview/exercise at the start and of the timing of any subcomponents if appropriate (e.g. 5 minutes’ briefing time then 15 minutes’ simulation exercise).

• All candidates must be asked in their invitation letter whether they require any special arrangements in order to attend the interview/selection centre. All reasonable/necessary adjustments should be made (in compliance with the Orange Guide).

Interview Questions/Selection Centre Content • Content of the interviews/selection centre must clearly relate to the criteria identified

in the Person Specification. • Each candidate must receive a minimum of 30 minutes assessment time (interviews

or selection centre exercises). The purpose of this is to ensure candidates have sufficient opportunity to provide evidence of the required competencies.

• Second stage assessments should have face validity (i.e. be relevant/reasonable to the candidate).

• Interviews must be competency-based and structured. The same question areas should be asked of all candidates applying of the same post. A range of acceptable probing questions should be developed that can be used to elicit further information

6

from candidates as necessary. Structured interviews that use the same questions for every candidate based on the selection criteria will help ensure candidates are assessed fairly and only against the selection criteria, in addition to help defending discrimination claims from unsuccessful candidates.

• Using multiple assessments is strongly encouraged such as using a split-panel interviewer with three 10 minute interviews conducted in three separate rooms. The purpose of this is to increase reliability and validity by ensuring candidates receives multiple assessments by multiple selectors.

Selectors • All selectors involved in interview/selection centres must be trained in selection

principles and the interview/selection centre process. Selectors must be trained in recording appropriate information at interview/selection centre. Lay involvement is essential where appropriate (see Attachments 1 and 2 below).

• Each selector on a panel should assign marks to candidates independently which can be aggregated to give a final mark for the interview/exercise where appropriate.

• Standardised, competency-based scoring frameworks must be developed and used for each interview/exercise. This should include a proforma for collecting observation notes made during the interview/exercise and evaluations/justifications for marks completed after the interview/exercise. Selectors must ensure that sufficient detail is recorded to justify a selection decision (and to reference feedback at a later date, as appropriate). Example interview report forms are provided in the trainers resource pack.

9. Reference Reports

• References should be requested at application stage and will be used to inform the selection process. References will be used as a pre-allocation check once selection decisions have been made. Guidelines for UoA on use of reference reports will be provided.

• The nationally agreed standard, structured reference form must be used. • Candidates will be asked to provide two references, one of which must be from their

current employer. 10. Feedback

• A standardised procedure for providing feedback to successful and unsuccessful candidates should be agreed in the UoA, in line with the national process and implemented consistently.

• Informal feedback, delivered immediately after the interview, by the interview panel, is not permitted.

• Feedback can only be given after the selection decision has been formally communicated to the candidate, in line with the national system.

• Staff providing feedback to candidates must be appropriately qualified. • Feedback must directly relate to the selection criteria and be based on evidence

gathered during the selection process. Feedback should be descriptive, accurate and as objective as possible and must be substantiated by documentation completed during the selection process, as appropriate.

• Under data protection legislation, candidates have the right to access any data held on their application, including shortlisting/interview/selection centre documentation and references.

7

Attachment 1: OORRAANNGGEE GGUUIIDDEE PPaaggee 1100.. WWhhoo ssiittss oonn tthhee aappppooiinnttmmeenntt ccoommmmiitttteeee?? 9. In England, Wales and Northern Ireland, the membership of an appointment committee should be: i. a lay chairman who has the confidence of the participating hospitals and locations. The chair ensures that their interests are fully considered in the appointment process. Chairmen will be appointed from a list compiled by the postgraduate dean and endorsed by trusts, colleagues and where appropriate, universities. A lay chairman may be an executive or non executive member of a trust board or other senior non-medical member of management. A senior doctor, for example, a trust medical director may act in a lay capacity as chairman. In this case the vital distinction between the lay role of the chairman and the roles of the clinical and academic members of the committee should be born in mind. A lay chairman cannot undertake the roles of any other members of the committee while acting as chairman; ii. the regional college adviser or a nominated deputy ; iii. the relevant postgraduate dean or a nominated deputy ; iv. representatives of the consultant staff in the training location(s) involved in the (rotational) training programme. The composition will depend on local circumstances but will be a minimum of two and normally a maximum of four consultants. Where more than four trusts are involved in the training programme representatives should sit on appointment committees on a rotational basis ; v. a nominee from the appropriate university in the deanery ; vi. the programme director or chairman of the deanery specialty training committee; vii. a representative of senior management in an employing trust in the training rotation. The appointment committee for a dual certification programme must reflect the interests of each of the specialties concerned. Postgraduate deans, in convening appointment committees, should also take in to account the diverse nature of the modern medical work force which includes significant numbers of women doctors and doctors from ethnic minorities. Committees should, therefore, reflect this diversity and include, where possible, members who are women and/or from ethnic minorities. 10. In Scotland the membership of the Committee will comprise at least five members including: i. a chairman selected from a panel drawn up by the regional postgraduate dean in consultation with the trusts in the region ; ii. a member from the appropriate section of the National Panel of Specialists ; iii. a member of the regional Postgraduate Medical Education Committee (usually the regional postgraduate dean or a deputy) ; iv. a senior medical representative of the services principally involved in the training programme for the post in question (e.g. clinical director or consultant); and v. a consultant appointed by the relevant university. 11. If extra or alternate membership is proposed to take account of a particular discipline, placement or rotation, the postgraduate dean should be consulted and will be responsible for arranging this where it is necessary. A proper balance of membership should be care fully preserved.

8

Attachment 2: PMETB GENERIC STANDARDS FOR TRAINING April 2006.

Domain 4. Recruitment, selection and appointment. The purpose of this domain is to ensure that the processes for entry into postgraduate training programmes are fair and transparent. Processes must be consistent with PMETB Principles for Entry to Specialist Training. Responsibility: Deaneries. Evidence: Deanery data, trainee surveys. Standard: Processes for recruitment, selection and appointment must be open,

fair, and effective. Recruitment and selection Mandatory 4.1 Candidates will be eligible for consideration for entry into a specialist training programme if they:

a. are a fully registered medical practitioner or hold limited registration with the General Medical Council or are eligible for any such registration;

b. are fit to practise. 4.2 The selection process (which may be conducted by interview or by other process)

must: • ensure that information about places on training programmes, eligibility and

selection criteria and the application process is made widely available in sufficient time to doctors who may be eligible to apply;

• use criteria and processes which treat eligible candidates fairly;

• select candidates on the basis of open competition;

• have an appeals system against non-selection on the grounds that the criteria

were not applied correctly, or were unfairly discriminatory;

• seek from candidates only such information (apart from information sought for equalities monitoring purposes) as is relevant to the published criteria and which potential candidates have been told will be required.

4.3 Selection panels must consist of persons who have been trained in selection

principles and processes. Developmental 4.4 In addition to 4.1, to be eligible for consideration for entry into a specialist training

programme, candidates must be able to demonstrate the competences required to complete Foundation Training. (This covers candidates who have completed Foundation Training, candidates who apply before completion and those who have not undertaken Foundation training, but can demonstrate the competences in another way.)

4.5 Selection panels must include a lay person.

Res

ou

rce

Pac

k;S

ectio

n B

Resource Pack: Section B

Tab

=w

here

to fi

nd d

ocum

enta

tion

and

mat

eria

ls in

the

reso

urce

pac

k

Sect

ion

B.

N

atio

nal

Sta

nda

rds

& R

esou

rce

Pac

k fo

r U

pski

llin

g M

eth

odol

ogy

Nat

ion

al S

tan

dard

D

ocu

men

tati

on

Trai

ner

Res

ourc

e

Per

son

Spe

cifi

cati

on

Per

son

Spe

cific

atio

ns

deliv

ered

by

CO

PM

eD S

teer

ing

Gro

up (A

utum

n 20

06)

The

re w

ill be

nat

iona

lly a

gree

d pe

rson

spe

cific

atio

ns fo

r eac

h sp

ecia

lty a

nd fo

r eac

h en

try le

vel o

f ea

ch s

peci

alty

(e.g

. ST1

, ST2

, ST3

). A

gree

men

t will

be v

ia th

e A

cade

my

of M

edic

al R

oyal

C

olle

ges

Spe

cial

ty T

rain

ing

Com

mitt

ee a

nd C

OP

MeD

.

Per

son

Spe

cific

atio

ns w

ere

requ

este

d fro

m th

e 16

Spe

cial

ties

for S

T1, S

T2 a

nd S

T3 e

ntry

(as

appr

opria

te).

Thes

e w

ill fo

llow

a g

ener

ic fo

rmat

.

The

Per

son

Spe

cific

atio

n id

entif

ies

elig

ibilit

y an

d se

lect

ion

crite

ria re

leva

nt to

eac

h S

peci

alty

.

Sel

ectio

n cr

iteria

incl

ude

com

pete

ncie

s co

mm

on a

cros

s al

l spe

cial

ties

and

com

pete

ncie

s id

entif

ied

from

rese

arch

as

impo

rtant

to p

artic

ular

spe

cial

ties.

The

Per

son

Spe

cific

atio

ns d

escr

ibe

how

and

whe

n th

e se

lect

ion

crite

ria a

re e

valu

ated

. Th

ese

will

be a

gree

d vi

a A

oMR

C a

nd C

OP

MeD

.

Tab

Ex

ampl

e Pe

rson

Sp

ecifi

catio

n

for A

naes

thet

ics

at S

T1

Exa

mpl

e P

erso

n S

peci

ficat

ion

Ref

eren

ce

Rep

ort

Form

del

iver

ed b

y C

OP

MeD

Ste

erin

g G

roup

(Aut

umn

2006

)

The

re w

ill be

a n

atio

nally

agr

eed

stru

ctur

ed re

fere

nce

form

des

igne

d by

the

sele

ctio

n m

etho

dolo

gy

team

and

agr

eed

with

the

AoM

RC

, CO

PM

eD a

nd o

ther

rele

vant

sta

keho

lder

s.

App

lican

ts w

ill be

requ

ired

to s

ubm

it th

e na

mes

of t

wo

refe

rees

, bot

h of

who

m m

ust h

ave

supe

rvis

ed th

e ap

plic

ant i

n th

e pa

st 2

yea

rs. R

efer

ence

s w

ill be

requ

este

d fo

r sho

rt lis

ted

cand

idat

es o

nly.

Ref

eren

ce R

epor

ts w

ill be

stru

ctur

ed a

nd re

quire

refe

rees

to p

rovi

de in

form

atio

n re

leva

nt to

the

Per

son

Spe

cific

atio

n.

Tab

N

atio

nal

Ref

eren

ce F

orm

(D

raft)

Nat

iona

l Ref

eren

ce F

orm

App

licat

ion

Form

Fo

rm d

eliv

ered

by

CO

PM

eD S

teer

ing

Gro

up (A

utum

n 20

06)

U

pski

lling

&

Sho

rtlis

ting

deliv

ered

by

UoA

/Dea

nery

The

re w

ill be

a n

atio

nally

agr

eed

appl

icat

ion

form

, with

nat

iona

lly a

gree

d sp

ecia

lty-s

peci

fic

sect

ions

, des

igne

d by

the

sele

ctio

n m

etho

dolo

gy te

am a

nd a

gree

d w

ith th

e A

oMR

C, C

OP

MeD

and

ot

her r

elev

ant s

take

hold

ers.

All

appl

icat

ions

will

be s

ubm

itted

ele

ctro

nica

lly v

ia M

TAS

.

App

lican

ts w

ill be

invi

ted

to a

pply

to tw

o sp

ecia

lties

and

to tw

o un

its o

f app

licat

ion

(UoA

) for

eac

h (u

nles

s on

e of

the

spec

ialti

es s

elec

ted

is g

ener

al p

ract

ice,

in w

hich

cas

e th

ey w

ill be

rest

ricte

d to

a

sing

le g

eogr

aphy

as

is c

urre

ntly

the

case

for G

P re

crui

tmen

t).

A

pplic

ants

will

be a

sked

to s

ubm

it a

sing

le a

pplic

atio

n fo

r vac

anci

es a

t the

mos

t app

ropr

iate

leve

l of

train

ing

with

in a

spe

cial

ty (S

T1 o

r ST2

or S

T3, e

tc.)

Tab

N

atio

nal

App

licat

ion

Form

(D

raft)

Ta

b

Exam

ple

Indi

cato

rs fo

r Sh

orlis

ting

Ta

b

Rat

ing

Scal

e

Tab

Trai

ner S

lides

&

Exa

mpl

e Ti

met

able

for a

½

day

wor

ksho

p

Tab

Exa

mpl

e A

pplic

ant

Res

pons

es to

App

licat

ion

Que

stio

ns (f

or u

se in

ca

libra

tion)

Tab

=w

here

to fi

nd d

ocum

enta

tion

and

mat

eria

ls in

the

reso

urce

pac

k

App

licat

ion

form

que

stio

ns w

ill re

flect

thos

e co

mpe

tenc

ies

iden

tifie

d as

impo

rtant

to e

ach

spec

ialty

(a

nd li

nked

to th

e P

erso

n S

peci

ficat

ion)

.

Sho

rtlis

ting

will

be s

core

d lo

cally

by

sele

ctio

n pa

nels

incl

udin

g tra

ined

sel

ecto

rs in

the

spec

ialty

ag

ains

t nat

iona

l, sp

ecia

lty s

peci

fic c

riter

ia.

Sho

rtlis

ting

resu

lts w

ill be

com

mun

icat

ed to

app

lican

ts v

ia e

-mai

l onl

y in

acc

orda

nce

with

the

agre

ed n

atio

nal t

imet

able

.

Inte

rvie

w/

Sel

ecti

on

Cen

tre

Ups

killi

ng &

P

roce

ss d

eliv

ered

by

UoA

/Dea

nery

As

an a

bsol

ute

min

imum

, sel

ectio

n w

ill co

nsis

t of a

30

min

ute

com

pete

ncy

base

d st

ruct

ured

in

terv

iew

.

The

pan

el c

ompo

sitio

n m

ust c

onfo

rm to

the

guid

ance

pro

vide

d in

the

Ora

nge

Gui

de (a

nd m

ust

have

lay

invo

lvem

ent).

Sel

ectio

n w

ill be

sco

red

loca

lly b

y a

sele

ctio

n pa

nel i

nclu

ding

trai

ned

sele

ctor

s in

the

spec

ialty

ag

ains

t nat

iona

l, sp

ecia

lty s

peci

fic c

riter

ia d

escr

ibed

on

the

rele

vant

Per

son

Spe

cific

atio

n.

Sel

ectio

n re

sults

will

be c

omm

unic

ated

to a

pplic

ants

via

e-m

ail o

nly

in a

ccor

danc

e w

ith th

e ag

reed

na

tiona

l tim

etab

le.

Spe

cial

ties/

UoA

may

opt

to g

o be

yond

the

min

imum

requ

irem

ent a

nd u

se a

dditi

onal

sel

ectio

n m

etho

ds to

enh

ance

val

idity

(e.g

. 3x1

0min

s in

terv

iew

s, s

elec

tion

cent

res

usin

g m

ultip

le s

elec

tion

tool

s su

ch a

s si

mul

atio

ns, s

cena

rios,

etc

).

A n

atio

nal a

ppea

ls s

yste

m w

ill en

sure

con

sist

ency

of a

ppea

l pro

cess

es fo

r Uni

ts o

f App

licat

ion

(UoA

). Th

is w

ill be

agr

eed

via

CO

PM

eD, A

oMR

C a

nd o

ther

sta

keho

lder

s as

app

ropr

iate

.

Tab

Ex

ampl

e In

terv

iew

Fo

rmat

with

In

dica

tors

Ta

b

Exam

ple

Bla

nk

Inte

rvie

w F

orm

Ta

b

Exam

ple

Inte

rvie

w

Que

stio

ns

Tab

Trai

ner S

lides

&

Exa

mpl

e Ti

met

able

for a

½

day

wor

ksho

p

Wor

ksho

p E

xerc

ises

: E

xerc

ise:

Que

stio

ning

S

tyle

s (O

nion

Exe

rcis

e)

Exe

rcis

e: H

ow a

nd W

hat

to R

ecor

d Fr

eque

ntly

Ask

ed

Que

stio

ns (F

AQ

s)

Eva

luat

ion

&

Aud

it

Form

(s) d

eliv

ered

by

CO

PM

eD

Ste

erin

g G

roup

(A

utum

n 20

06)

Ups

killi

ng &

P

roce

ss d

eliv

ered

by

UoA

/Dea

nery

Eva

luat

ion

is c

ritic

al to

iden

tify

area

s fo

r im

prov

emen

t and

ens

urin

g ad

here

nce

to P

ME

TB

prin

cipl

es a

nd s

tand

ards

.

Eva

luat

ion

prof

orm

a m

ust b

e co

mpl

eted

and

sig

ned

by th

e ch

airp

erso

n of

a s

elec

tion

pane

l. S

peci

fical

ly, t

he p

anel

mus

t aud

it th

e pr

oces

s an

d m

etho

dolo

gy to

mon

itor c

ompl

ianc

e w

ith n

atio

nal

stan

dard

s.

Afte

r int

ervi

ew, a

ll ca

ndid

ates

will

be a

sked

to c

ompl

ete

an a

nony

mis

ed e

valu

atio

n pr

ofor

ma

rega

rdin

g re

crui

ter e

ffect

iven

ess.

Eva

luat

ion

info

rmat

ion

will

be c

olla

ted

cent

rally

by

the

UoA

. R

esul

ts w

ill be

repo

rted

to re

leva

nt

stak

ehol

der g

roup

s fo

r mod

ifica

tion

of th

e ex

istin

g pr

oces

s as

app

ropr

iate

.

Tab

Ev

alua

tion

&

Aud

it Fo

rm (D

raft

for

App

rova

l) Ta

b

Can

dida

te

Eval

uatio

n Fo

rm

(Dra

ft fo

r App

rova

l)

Exa

mp

leP

erso

nS

pec

ific

atio

n

Example Person Specification

PERSON SPECIFICATION APPLICATION TO ENTER SPECIALTY TRAINING at ST1: ANAESTHESIA

ENTRY CRITERIA ESSENTIAL DESIRABLE WHEN EVALUATED1

QUALIFICATIONS • MBBS or equivalent medical qualification

Application form

ELIGIBILITY • Eligible for full or limited registration with the GMC at time of appointment

• Evidence of achievement of Foundation competencies by August 2007 in line with GMC standards/Good Medical Practice including:

• Good clinical care • Maintaining good medical

practice • Good relationships and

communication with patients • Good working relationships with

colleagues • Good teaching and training • Professional behaviour and

probity • Delivery of good acute clinical

care • Eligibility to work in the UK

Application form

Application form Interview / Selection centre2

Application form

FITNESS TO PRACTISE

• Is up to date and fit to practise safely

Application form References

LANGUAGE SKILLS

• All applicants to have demonstrable skills in written and spoken English that are adequate to enable effective communication about medical topics with patients and colleagues which could be demonstrated by one of the following:

• a) that applicants have undertaken undergraduate medical training in English; or

• b) have the following scores in the academic lnternational English Language Testing System (IELTS) – Overall 7, Speaking 7, Listening 6, Reading 6, Writing 6.

• However, if applicants believe that they have adequate communication skills but do not fit into one of the examples they need to provide evidence

Application form Interview / Selection centre

1 ‘when evaluated’ is indicative, but may be carried out at any time throughout the selection process 2 A selection centre is a process not a place. It involves a number of selection activities that may be delivered within the Unit of Application.

HEALTH • Meets professional health requirements (in line with GMC standards/Good Medical Practice)

Application form Pre-employment health screening

CAREER PROGRESSION

• No unexplained career gaps • Less than 12 months’

experience3 (at SHO level) in this specialty (not including Foundation modules) by August 2007

Application form

APPLICATION COMPLETION

• ALL sections of application form FULLY completed according to written guidelines

Application form

SELECTION CRITERIA

CLINICAL SKILLS • Clinical Knowledge & Expertise: Capacity to apply sound clinical knowledge & judgement. Able to prioritise clinical need. Works to maximise safety & minimise risk

• Personal Attributes: Shows aptitude for practical skills, e.g. manual dexterity

Application form Interview / Selection centre References

ACADEMIC / RESEARCH SKILLS

• Research Skills: Demonstrates understanding of the principles of audit & research

• Evidence of relevant academic & research achievements , e.g. degrees, prizes, awards, distinctions, publications, presentations, other achievements

• Evidence of active participation in audit

• Teaching: Evidence of interest and experience in teaching

Application form Interview / Selection centre

3 Any time periods specified in this person specification refer to full time equivalent

PERSONAL SKILLS • Vigilance & Situational Awareness: Capacity to be alert to dangers or problems, particularly in relation to clinical governance. Demonstrates awareness of developing situations

• Coping with Pressure: Capacity to operate under pressure. Demonstrates initiative & resilience to cope with setbacks & adapt to rapidly changing circumstances. Awareness of own limitations & when to ask for help

• Managing Others & Team Involvement: Capacity to work cooperatively with others and demonstrate leadership when appropriate. Capacity to work effectively in multi-professional teams

• Problem Solving & Decision Making: Capacity to use logical/lateral thinking to solve problems & make decisions

• Empathy & Sensitivity: Capacity to take in others’ perspectives and see patients as people

• Communication Skills: Demonstrates clarity in written/spoken communication and capacity to adapt language as appropriate to the situation

• Organisation & Planning: Capacity to organise oneself & prioritise own work. Demonstrates punctuality, preparation & self-discipline. Understands importance of information technology


PROBITY • Professional Integrity & Respect for Others: Capacity to take responsibility for own actions and demonstrate a non-judgemental approach towards others. Displays honesty, integrity, awareness of confidentiality & ethical issues


COMMITMENT TO SPECIALTY

• Learning & Personal Development: Demonstrates interest and realistic insight into anaesthesia, intensive care & acute care. Demonstrates self-awareness & ability to accept feedback.

• Extracurricular activities / achievements relevant to anaesthesia, intensive care and/or acute care


Ref

eren

ceR

epo

rt

Reference Report

of 4:

REF 1

Structured Reference

Form REF 1 Issued by (organisation)

(website)

AUG 2007

The candidate to whom this reference refers has applied for Specialty training post(s) and has given your name as a referee. We would be grateful if you could provide us with information requested below. Please note that we can only accept references on this structured reference form. This professional reference should verify factual information and comment on the strengths and weaknesses of the candidate as an indicator of his/her suitability for appointment. This is not a personal testimonial but your objective assessment, from personal observation, of competencies based on the relevant person specification(s). If the candidate is successful, we may also use the information you provide to help inform the formative assessment process and identify potential learning needs. In such cases the comments you make may be discussed with the candidate named below and/or his/her trainer. Your reference may also be made available to other departments within the NHS. This reference form has been developed with the General Medical Council publication “Good Medical Practice” in mind. Your attention is drawn to the following paragraph: “When providing references for colleagues, your comments must be honest and justifiable; you must include all relevant information which has a bearing on the colleague’s competence, performance, reliability and conduct” (GMC Good Medical Practice, Second Edition, July 1998 – The duties of a doctor registered with the General Medical Council, Item 11 – References.)

Candidate Name: (MTAS to complete)

Candidate GMC No: (MTAS to complete) Candidate Ref No: (MTAS to complete)

(1st specialty applied for - MTAS to complete) Post(s)/Specialties Applied For: (2nd specialty applied for - MTAS to complete)

Relationship to candidate:

Educational Supervisor Clinical Supervisor/Employer Other (please specify)

Please state the dates the candidate was supervised by/worked with you:

Date started: Date finished:

Position held / Grade:

Location:

Are you aware of any formal disciplinary procedure(s) the candidate was subject to during their time with you? YES NO If Yes, please give details:

of 4:

REF 1



(website)

AUG 2007

Please give your opinion regarding the candidate’s present knowledge, skills and personal attributes by ticking the appropriate boxes on the next three pages. Statements are provided to give examples of behaviours that would constitute different levels of performance, although this is not intended to be an exhaustive list. Please use the space provided to give examples of the candidate’s behaviour that support the rating you have given them in each area; this is essential if you have given a rating of D or C.

[Medical graduates only] Clinical Knowledge & Expertise: Appropriate knowledge base and capacity to apply sound clinical judgement to problems. Capacity to prioritise clinical need.

A B C D

Good to excellent Satisfactory Weak Cause for concern

Comments/evidence:

[Non-medical graduates for public health only] Technical Knowledge & Expertise: Appropriate knowledge base relevant to public health and capacity to apply sound judgement to problems.

A B C D

Good to excellent Satisfactory Weak Cause for concern

Comments/evidence:

Communication Skills: Capacity to communicate clearly and effectively with others, adjusting behaviour and language (written/spoken) as appropriate to needs of differing situations.

A B C D

Always speaks clearly, gives others time to speak and checks understanding

Usually communicates clearly, tends to use appropriate language

Can be lacking in clarity and coherence when communicating with others

Uses technical language that others do not understand, ignores what they have to say

Comments/evidence:

Empathy & Sensitivity: Capacity and motivation to take in patients’/colleagues’ perspectives and treat others with understanding.

A B C D

Always shows empathy and sensitivity, gives reassurance to others

Usually demonstrates empathy towards others

Shows some interest in the individual and occasionally reassures others

Is not sensitive to the feelings of others and treats them in an impersonal manner

Comments/evidence:

Professional Integrity: Capacity and motivation to take responsibility for own actions. Respect for position, patients and protocol.

A B C D

Takes full responsibility for their own actions, demonstrates respect for all

Often shows respect to others, is generally aware of ethical issues

Sometimes seeks to blame others for their actions

Does not take responsibility for their actions, does not demonstrate respect for others

Comments/evidence:

of 4:

REF 1



(website)

AUG 2007

Problem Solving & Decision Making: Capacity to think beyond the obvious, with analytical but flexible mind. Capacity to bring a range of approaches to problem solving and decision making.

A B C D

Thinks beyond surface information, uses a range of problem solving strategies

Usually thinks beyond surface information, able to make decisions

Often relies on surface information, can be hesitant when making decisions

Lets assumptions guide diagnosis/decision making, does not think around issues

Comments/evidence:

Organisation & Planning: Capacity to organise time/information in a structured and planned manner. Capacity to prioritise conflicting demands and deliver on time.

A B C D

Excellent at managing own time and prioritising workload

Usually able to prioritise tasks and meet deadlines

Is often late for meetings and deadlines and disorganised with paperwork etc.

Is always late for meetings/deadlines and unable to prioritise tasks

Comments/evidence:

Learning & Development: Capacity and motivation to learn from experience, commits time and resources to appropriate personal and professional development activities.

A B C D

Actively seeks out constructive criticism/feedback and development opportunities

Often learns from experience, generally reacts well to constructive criticism/feedback

Needs assistance in identifying own development needs/targets

Reacts badly to constructive criticism or feedback, not interested in own development

Comments/evidence:

Managing Others & Team Involvement: Capacity to work effectively in partnership with others and demonstrate leadership when appropriate.

A B C D

Is excellent at supporting and motivating others, able to lead when appropriate

Recognises contribution of others, usually able to compromise

Tends to take a ‘back seat’ rather than participating, reluctant to lead

Sticks rigidly to their own agenda, critical of others’ ideas

Comments/evidence:

Working under Pressure: Capacity to work effectively under pressure, remaining calm and objective. Demonstrates initiative and resilience to cope with setbacks and rapidly changing circumstances.

A B C D

Always remains calm under pressure, rapidly adapts to changing situations

Often recognises when to share workload, able to cope with changing circumstances

Finds it difficult to remain calm under pressure or to switch off after work

Loses temper easily, refuses to share workload, unable to adapt to change

Comments/evidence:

of 4:

REF 1



(website)

AUG 2007

Recommendation of candidate for training in (1st specialty - MTAS to insert) Strongly without reservation A Would have some reservations C Could recommend as competent B Could not recommend for training D If you have any other comments regarding this candidate’s application for this specialty, please give details here:

Recommendation of candidate for training in (2nd specialty - MTAS to insert) Strongly without reservation A Would have some reservations C Could recommend as competent B Could not recommend for training D If you have any other comments regarding this candidate’s application for this specialty, please give details here:

This reference is based upon (tick all that apply): General impression A Collective opinion of tutors/consultants/supervisors C Close observation B Employer’s views D SIGNATURE

NAME (print in block capitals)

POSITION HELD CONTACT TELEPHONE NO. Name of deanery/ hospital/practice/organisation DATE (dd/mm/yyyy)

It is essential that this form is stamped with an official deanery, hospital, training practice or organisation stamp. If no stamp is available, please attach a compliment slip signed by the referee. Forms received without a stamp or a signed compliment slip will be returned. Official stamp (deanery/hospital/training practice/organisation)

Thank you for completing this reference. This form should be returned to the address

given on the accompanying e-mail. Please ensure that a signed paper copy is also given to the candidate.

Was the candidate’s attendance/timekeeping satisfactory?

YES NO If No, please give details

Would you be happy to work with this candidate again? YES NO

RECOMMENDED PROCESS

FOR

THE USE OF REFERENCES DURING RECRUITMENT AND SELECTIONINTO SPECIALIST TRAINING DURING 2007

Introduction

1. The principle of applicants being required to provide details of tworeferees, both of whom must have supervised the applicant in the past 2years, has been agreed by the four UK health departments.

2. It has also been agreed that structured references will be requested forshort-listed candidates only.

3. Equally, it has been agreed that the Medical Training Application Service(MTAS) will be used by all four UK countries for recruitment into specialisttraining programmes and FTSTAs.

4. The four UK health departments have stated that there will be nationallyagreed structured reference forms for each specialty or specialty group.

5. This paper sets out the recommended process for how references will fitinto the overall process and timeline for recruitment to specialist trainingduring the national recruitment episode planned for 2007.

The issues

6. Anecdotal evidence suggests references are not received for up to 40% ofapplicants by the time of interview. Deaneries and trusts have to chase upthe missing references, which is a time consuming exercise. When wemove to national recruitment within a very tight timescale, it will not bepossible to chase references in this way. .

7. The process and supporting MTAS system must provide a mechanism forensuring references are available at the time of the interview for everyapplicant.

8. There is considerable concern across deaneries and trusts regardingfalsified or altered references.

9. Every applicant has a legal right to see their references.

10. It is felt that some referees will not give an accurate reference if they knowit will be seen by the applicant (i.e. they may be reluctant to raise issues ofpoor performance). However, applicants are currently allowed to see theirreferences under the freedom of information act.

11. Where and how references are used within the selection process iscurrently variable across deaneries and trusts.

Recommended process

12. The national, structured reference form will be the only referenceaccepted.

13. Applicants must provide valid e-mail addresses for two referees. They willhave the option to nominate different referees for each specialty they applyto.

14. Once an application is submitted, the MTAS system will generate anautomatic e-mail message to nominated referees to inform them that theyhave been nominated (and by whom) and to request that they respond tothe message to prove that the communication channel is open.

15. Referees will also be able to submit the completed reference form at thisstage if they wish to do so. They will also be able to view a copy of theapplicant’s submitted application form.

16. If the referee does not respond we can use the MTAS system to sendreminders and/or to send notes to the applicant telling them that there is aproblem with the referee’s e-mail address. There is the three week short-listing period to do this. Precisely how this will be developed will bedetermined during the MTAS requirements workshops being held inSeptember/October 2006.

17. Once short-listing is complete, the MTAS system will automatically send amessage to the nominated referees of short-listed applicants who have notalready submitted a completed reference requesting they complete thestructured reference form and telling them by when it must be submitted.They will also be able to view a copy of the applicant’s submittedapplication form.

18. Referees will need to complete and submit the form in a single session.There will be no facility to save the form and return to it at a later date.

19. There will be a facility to print copies of the completed form. Copiesshould be printed and then a signed copy provided to the applicant.

20. Short-listed applicants will be instructed to bring a copy of all signed/certified references with them to the selection interview.

21. References will only be seen by the selection panel chair prior to theinterview. The chair may choose to explore an area with the candidate asappropriate during the interview.

22. Following the selection interview and scoring, the reference may be seenby the selection panel members as supplementary support for decisionsmade, under the guidance of the chair.

23. Where there is cause for concern between the outcome of the selectioninterview and the content of the reference(s), or major differences betweenthe references provided by the two referees, recruitment staff will be ableto cross-check the content of the signed/certified reference with theelectronic copy held on the MTAS system to ensure that the printedreferences have not been altered in any way.

Ap

plic

atio

nFo

rm D

raft

Application Form Draft

ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU

ATTEND THE INTERVIEW STAGE OF THE SELECTION PROCESS

SAMPLE APPLICATION FORM FOR SPECIALTY TRAINING

PROGRAMME ENTRY AT ST1

PLEASE NOTE:

• THIS DOCUMENT IS FOR ILLUSTRATION ONLY TO DEMONSTRATE THE SELECTION METHODOLOGY.

• THE ONLINE APPLICATION FORM WILL HAVE THE SAME SECTIONS BUT WILL BE PRESENTED DIFFERENTLY AS APPROPRIATE FOR THE ONLINE APPLICATION SYSTEM.

• THE ONLINE APPLICATION FORM WILL INCLUDE DECLARATIONS, FITNESS TO PRACTISE, CRB DECLARATION, EQUAL OPPORTUNITIES MONITORING FORM AND OTHER RELEVANT EMPLOYMENT CHECKS.


YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU ATTEND

THE INTERVIEW STAGE OF THE SELECTION PROCESS

STANDARDISED APPLICATION FORM FOR SPECIALTY TRAINING PROGRAMME: Entry at ST1

Declaration: I understand that any candidate falsifying evidence on this application form or in supporting evidence will be automatically referred to the General Medical Council (GMC)

Advertisement Reference Number

Please indicate which Unit(s) of Application and Specialty/Specialties you wish to apply to by completing the boxes below

Which Specialty/Specialties are you applying to work in.

Unit of Application/ Deanery

A 1

2

B 1

2

Section 1 - Personal Details Surname Forenames

Title

If previously known by another name, please specify

Home Address

Address for correspondence

Home telephone number Home fax number Home email address Mobile phone number

Work telephone number Work fax number Work email address Preferred email address

Section 2 – Entry Qualifications Medical School/Dental School/University (NB Only include here the qualification(s) that meet(s) the eligibility requirements listed in the relevant person specification(s). Additional qualifications are covered in Section 7)

Dates




Evidence of IELTS (International English Language Testing System) or equivalent (for candidates who have not graduated from an English medium university) – state level achieved

Date awarded

SHORTLISTED CANDIDATES WILL BE REQUIRED TO BRING ORIGINAL CERTIFICATES AND DOCUMENTATION AS EVIDENCE TO THE INTERVIEWS

Section 3 - General Medical/Dental Council Registration Type of registration GMC/GDC Number Renewal date Name in which you are

registered

Section 4 – Eligibility for Entry into Specialty Training Have you been issued with a Certificate of Completion of Foundation training? You will need to bring the original certificate if you are called for interview

YES NO

Will you be issued with a Certificate of Completion of Foundation training by August 2007? YES NO

If not applying directly from Foundation, have you gained Foundation competencies? YES NO

If not applying directly from Foundation, please describe how you have gained Foundation competencies

Are you a United Kingdom (UK), European Community (EC) or European Economic Area (EEA) national? (Please circle as appropriate) YES NO

If not, do you have evidence of entitlement to enter and work permanently in the United Kingdom ie settled status? (Please circle as appropriate) YES NO

If you have circled no to both of the above questions please tick those boxes that relate to your current immigration status:

Tick Status Expiry date

Highly Skilled Migrant Programme

Permit Free Training

Refugee

Work Permit

Any other – please specify ……………………………………………………………

Residence Permit valid until: Please attach a copy of your residence permit to the application form

Date available to start if offered the post




Section 5 – Current and Previous Employment Current employer’s name

Address

Telephone number

Current position please state if locum position

Grade Date appointed

Length of contract

Length of notice required

Do you currently hold an NTN or VTN? (please give number)

YES/NO

Please list previous employment below:

Specialty, Hospital, Consultant Grade From To Months in post

Will you have completed less than 12 months’ experience in the specialty you are applying for by August 2007? [note: not for all specialties] YES NO

Do you have any unexplained career gaps (of more than 4 weeks’ duration?) since qualification? If YES, please specify

YES NO




Section 6 – Commitment to the Specialty

6.1 Please provide a brief statement describing the reasons why you have chosen to train for a career in this specialty. Please include any information about yourself that you feel is important or relevant in this regard. (max. 75 words)

6.2 What steps have you taken to develop your understanding of this specialty? (max. 150 words)

6.3 Please provide evidence of extracurricular activities and achievements relevant to this specialty (these could be work or non-work related). Indicate date and place relating to the evidence (max. 150 words)




Section 7 – Clinical, Academic & Research Skills 7.1 Give an example of a time when you personally made a significant difference to clinical outcome for a

patient by applying your judgement.. What did you do and how do you think the outcome was improved by your judgement? (max. 150 words).

7.2 What do you think is the relevance of medical research to a trainee doctor? If you have been involved in research, use examples from your own research to illustrate this. (max. 150 words).

7.3 Please provide evidence of your undergraduate and/or postgraduate academic and research achievements under the following headings (if appropriate).

a) Additional qualifications, e.g. degrees/diplomas – state class of degree awarded

Awarding Body Date of Qualification

OMFS only: MFDS

b) Relevant courses/training completed, e.g. ALS etc. [note: not for all specialties]

Place of training/name of training provider

Date of training




c) Prizes and other academic distinctions

Awarding Body Date of Award

d) Presentations/posters at conferences/ Publications (shortlisted candidates will be asked to bring copies of all abstracts and publications to the interview)

e) Describe your experience of clinical audit. Indicate clearly your level of involvement and what you learned about the process.

f) Experience of delivering teaching. What experience of delivering teaching do you have (including outside medicine)?




Section 8 – Personal Skills - EXAMPLE A range of personal skills have been identified as important for this specialty. For each of the skills indicated below, please give an example (preferably recent) from your own experience to illustrate how you dealt with particular situations. You may draw your experiences from work or other activities (unless otherwise specified).

8.1 [Vigilance & Situational Awareness] Describe a time when you identified a clinical situation that was changing rapidly. How did you identify this and what did you do? (max. 150 words)

8.2 [Coping with Pressure] Describe a time when you had a particularly difficult day at work due to workload pressures. What strategies did you use to cope with this situation both during and after this time? What other strategies do you regularly use to cope with stress? (max. 150 words)

8.3 [Managing Others & Team Involvement] Describe a time when you had to negotiate something to achieve something important with a colleague(s) or multiprofessional team. What approach did you take? What was the outcome? (max. 150 words)




Section 9 – Probity 9.1 [Professional Integrity] Please outline a time when you had to be open and honest with a patient, even

though you found it hard to do so. Why was this? What was the outcome? (max. 150 words)

Section 10 - Referees 1 Telephone Number Fax Number Email address

2 Telephone Number Fax Number Email address

Name, job title and address of two Consultant referees, covering the last two years of employment. One of the referees must be your present or most recent Consultant/Educational supervisor. The Unit of Application must obtain information from two referees who are able to comment, from experience, on your ability and suitability for the post.

Sh

ort

listi

ng

Ind

icat

ors

Shortlisting Indicators & Example Responses

Example Responses Only

THANK YOU FOR COMPLETING THIS FORM

Please indicate the level of entry (eg ST1, ST2 etc): ST1

Commitment to the Specialty

• What steps have you taken to develop your understanding of this specialty? (max. 150 words)

During my medical course I spent 3 weeks in the Cambridge clinical genetics department as part of an “SSC (student selected component)” including a range of adult and paediatric clinics covering the spectrum from adult cancer to pre-natal diagnosis. I also attended numerous genetics clinics apart from the SSC, which together with the SSC confirmed my attraction to the specialty. I have already been in contact with some of the Exeter department, and hope to spend some time in clinics during my F2 year. My obstetrics & gynaecology audit will look at the interface between obstetrics and genetics, and I hope to be involved in the links with genetics in my paediatrics F2 job as well.

• Please provide evidence of extracurricular activities and achievements relevant to this specialty. Indicate date and place relating to the evidence (max. 150 words)

During my medical course I was the BMA representative (2003-4) and the CMF (Christian Medical Fellowship) (2003-4) representative for Cambridge University. At BMA student conferences I proposed motions on various ethical issues, including informed consent for vaccination (2002, London), palliative care and euthanasia (2003, London), the law on driving offences and homicide (2004, Cambridge), and physician assisted suicide (2005, Keele). In 2003 I proposed a motion on discrimation in medicine at the BMA Annual Members’ Meeting (Torquay). All this indicates a strong interest in medical ethics which will be applicable to the ethical questions involved in genetics (e.g. the question of whether someone should have a genetic test, or whether ignorance is better). Genetics has a strong research focus. During my spare time from 2002-4 on my medical course in Cambridge I worked in clinical research, which should help in applying research results (both clinical and laboratory) to patient care.




Clinical, Academic & Research Skills • Give an example of a time when you personally made a significant difference to clinical outcome for a

patient by applying your judgement. What did you do and how do you think the outcome was improved by your judgement? (max. 150 words).

I was called to see an elderly surgical patient who had become anuric. Her urine output had been poor for three days and had been reviewed by several doctors, including senior members of the team. Each time she had been given a fluid challenge, but was never reassessed following this. On reviewing her charts she now had a severely positive fluid balance. Clinically assessing the patient she was fluid overloaded with pitting oedema to the level of her scapula, with a markedly raised jugular venous pressure. On this assessment she did not need further fluid challenges, but input from the renal team. I discussed the case with the renal registrar who suggested transfer to the high dependency unit and haemodialysis. Her renal failure could have been missed and further inappropriate fluid challenges given if I had not assessed the patient carefully and trusted my clinical judgement.

• What do you think is the relevance of medical research to a doctor? If you have been involved in research, use examples from your own research to illustrate this. (max. 150 words).

Without medical research, medicine would never progress. If no one is asking “How can we do this better?” and thinking of ways to test their hypotheses, the quality of medical care would never improve (and would probably stagnate). Evidence for therapies and medicines needs trials to prove or disprove the usefulness of an intervention. Prior to training in medicine I was involved in pre-clinical research, including researching new analgesics, then during my medicine course on clinical research on pain models in healthy volunteers. This showed me the scope there is for doctors to be involved in both pre-clinical and clinical research, and my research has given me an enquiring mind, so that I am not content to assume that we already have the best way of doing things. It has also shown me the advantages in doing research in a team, and how productivity improves when a team works together.




• Please provide evidence of your undergraduate and/or postgraduate academic and research achievements under the following headings (if appropriate).

a) Additional qualifications, e.g. degree(s)/diploma(s) – state class of degree awarded

Awarding Body Date of Qualification

BmedSci 2:1 University of Nottingham December 2003

BMBS University of Nottingham July 2005

b) Prizes and other academic distinctions

Awarding Body Date of Award

None to date

c) Presentations/posters at conferences & relevant publications (shortlisted candidates will be asked to bring copies of all abstracts and publications to the interview)

None to date

d) Describe your experience of clinical audit. Indicate clearly your level of involvement and what you learned about the process.

I instigated, developed and performed an audit on the effects of advice given when prescribing emergency contraception on future effective contraception use. I established current guidelines, and set standards with reference to these. Through retrospective review of notes I compared recent practice to these standards. This enabled me to develop advice on future practice. This was then implemented. I did not have the opportunity to complete the audit cycle through re-audit after the advice was implemented. This was useful experience in the audit cycle, and on extracting data from patient notes. I am currently auditing complications for potentially curative resections for bowel malignancy. I am collating the data retrospectively from patient notes. This data will compare results in our region with national standards.

e) What experience of delivering teaching do you have?

I had a final year medical student shadow me for two weeks at the end of my first foundation year, allowing him practical experience before starting as a house officer himself. He gave me good feedback on the two weeks, allowing him plenty of practical experience while assisting him when needed. I currently hold regular teaching sessions with the fifth year medical student attached to our firm, combining sessions on history taking and examination skills, with background theory on subjects of her choice. I note interesting cases I have encountered and share them with the medical student and house officers. I am supportive to the house officers, giving them relevant teaching when they encounter new clinical problems. I have given presentations at first year foundation teaching sessions, at educational radiology meetings, and at surgical training meetings.




Personal Skills: Specialty-Specific Questions A range of personal skills have been identified as important for this specialty. For each of the skills indicated below, please give an example (preferably recent) from your own experience to illustrate how you dealt with particular situations. You may draw your experiences from work or other activities

• [Vigilance & Situational Awareness] Describe a time when you identified a clinical situation that was changing rapidly. How did you identify this and what did you do? (max. 150 words)

A patient had been referred to the surgical unit by the general practitioner with possible biliary colic. On arrival he appeared unwell, on assessment he was clinically shocked and I began initial resuscitation. He improved markedly becoming more orientated, with improving observations. During further assessment however he deteriorated further. I found he had unequal blood pressures in both arms, and a palpable expansile abdominal mass. I phoned for assistance from my senior, arranged an urgent CT angio and arranged for an anaesthetist to accompany me to the scanner. A thoracic aortic dissection with leaking abdominal aortic aneurysm was seen on the scan and arrangements were made for the patient to go straight to theatre for an open repair.

• [Coping with Pressure] Describe a time when you had a particularly difficult day at work due to workload pressures. What strategies did you use to cope with this situation both during and after this time? What other strategies do you regularly use to cope with stress? (max. 150 words)

I was the surgical senior house officer on call, on a day when my registrar was away leaving me with main responsibility for the ward patients. Due to a rota error there was no house officer, and I was asked to cover their responsibilities as well. I also had a presentation to give. I conducted a brief ward round, ensuring my patients were stable and essential house officer tasks were finished. I arranged with a fellow senior house officer to hold my bleep during the presentation. When I collected this several referrals had arrived. I prioritised these by clinical urgency and began clerking. During a gap in referrals I reviewed my ward patients and checked all jobs had been cleared. By prioritising, regularly reviewing the situation, and utilizing help from colleagues I managed to juggle several different roles effectively.

• [Managing Others & Team Involvement] Describe a time when you had to negotiate something with a colleague(s) or multiprofessional team to achieve something important. What approach did you take? What was the outcome? (max. 150 words)

A patient was investigated under vascular for a possible psuedoaneurysm, which was excluded. His admission chest X-ray showed a large, suspicious lung lesion. He had previously had resection of a bladder tumour, and was under follow-up by urology. The vascular team were unwilling to investigate this further. On discussion with the respiratory team they suggested urology review. The urology team refused and said it was now a respiratory problem. After these difficulties with referral I organised an urgent CT chest and abdomen myself, and discussed the problem with the lung cancer nurse. She made him an urgent appointment in lung cancer clinic and listed him for the multi-disciplinary team meeting. I was then happy for him to be discharged, knowing that he would get the appropriate follow-up and investigations he needed.

ILLUSTRATION

Reference Framework for Target Selection Criteria

CLINICAL, ACADEMIC & RESEARCH SKILLS CLINICAL KNOWLEDGE & EXPERTISE: Capacity to apply sound clinical knowledge and judgement. Positive Indicators • identifies key issues involved, gets to the root cause • elicits necessary detail from patient/colleague • shows sound/systematic judgement in making decisions • is aware of appropriate options • is able to anticipate possible issues

Negative Indicators • overlooks important issues • fails to explore important information / signals • is too quick/unsystematic in making decisions • suggests too narrow a range of options • needs the “full picture” before understanding problem

RESEARCH SKILLS: Understanding of the importance and basic principles of research. Positive Indicators • provides clear and reasoned explanation / argument for

importance of research • demonstrates accurate understanding of purpose and impact of

research • shows awareness of basic principles of research methodology • provides evidence of personal experience of research • provides evidence of personal experience to support argument

Negative Indicators • explanation / argument is unclear or lacks relevance • shows little understanding of purpose and impact of research • demonstrates little awareness of research methodology • provides no evidence of personal experience of research • unable to link own experience to his/her argument

PERSONAL SKILLS EMPATHY & SENSITIVITY: Capacity and motivation to take in others’ perspectives and treat others with understanding. Positive Indicators • responds to needs/concerns with interest /understanding • acts in open, non-judgemental manner • makes effort to understand others’ perspectives • reassures others with appropriate words/actions • attempts to generate safe/trusting atmosphere

Negative Indicators • shows little apparent interest in others’ situations • is quick to judge, makes assumptions • little evidence of understanding others’ needs/concerns • unable to reassure others effectively • fails to generate safe/trusting atmosphere

VIGILANCE & SITUATIONAL AWARENESS: Capacity to monitor developing situations and anticipate dangers/problems. Positive Indicators • is alert to symptoms and signs suggesting conditions which

might progress or de-stabilise rapidly • picks up subtle changes in clinical condition • is able to anticipate possible dangers/problems • demonstrates awareness of performance of team members and

social dynamics within clinical situations

Negative Indicators • pays little attention to changes in clinical picture • fails to pick up on signs that indicate that original plans need to

be revised • is unable to anticipate dangers / problems • lacks awareness of level of competence of team members

and/or social dynamics within the clinical team COPING with PRESSURE: Capacity to operate under pressure. Demonstrates initiative and resilience to cope with changing circumstances. Positive Indicators • remains calm and in control when under pressure • rarely looses sight of wider needs of situation • recognises own limitations and is able to compromise • is able to seek help when necessary • uses strategies to deal with pressure/stress

Negative Indicators • becomes tense, agitated or over-sensitive under pressure • shifts focus to immediate worries/needs, looses perspective • becomes defensive or uncompromising under pressure • tries unsuccessfully to deal with situation alone • uses inappropriate strategies to deal with pressure/stress

ILLUSTRATION MANAGING OTHERS & TEAM INVOLVEMENT: Capacity to work effectively in partnership with others and demonstrate leadership when appropriate. Positive Indicators • participative, non-confrontational approach • negotiates with others, is willing to compromise • respects others’ views, is able to build on others’ ideas • able to delegate and demonstrate leadership when appropriate • gives constructive feedback & support where appropriate

Negative Indicators • confrontational style, is critical of others and their views • sticks rigidly to own agenda and does not negotiate /

compromise • treats some colleagues more favourably than others • fails to delegate or demonstrate leadership when appropriate • provides little support for others (verbal and/or enacted)

COMMUNICATION SKILLS: Capacity for clarity in written/spoken communication and to adapt language as appropriate to the situation. Positive Indicators • adjusts style of questioning / response as appropriate • is able to express ideas clearly to others (written/spoken) • uses inventive language (e.g. humour, analogy) to explain • uses active listening and open, patient-centred questions • makes effective use of non-verbal behaviour (voice, posture etc)

Negative Indicators • is unable to adapt language / behaviour as needed • is patronising / domineering in communicating with others • use of language is too functional / narrow / technical • limited evidence of active listening, overuse of closed questions • fails to engage others at non-verbal level

ORGANISATION & PLANNING: Capacity to organise time/information in a structured/planned manner. Able to think ahead and prioritise conflicting demands. Positive Indicators • thinks ahead, prepares and plans effectively • builds contingencies to deal with shifting demands • prioritises conflicting demands, delivers on time • understands limitations and works within them

Negative Indicators • misses deadline based on reasonable expectations • fails to think ahead, pre-plan, & build effective contingencies • disorganised, fails to manage time effectively • unable to juggle competing demands

PROBLEM SOLVING & DECISION MAKING: Capacity to think beyond the obvious, analytical but flexible mind. Capacity to apply knowledge to new situations and generate functional solutions. Positive Indicators • attempts to think “around” issue, open to new ideas • generates functional solutions • prioritises information/time effectively, identifies key points • able to sift peripheral information to detect root cause

Negative Indicators • makes immediate assumptions about problems • unable to suggest “workable” outcome • disorganised/unsystematic approach • focuses on peripheral issues, unable to identify root cause

PROBITY PROFESSIONAL INTEGRITY: Capacity to take responsibility for own actions. Demonstrates respect for all. Positive Indicators • demonstrates clear respect for patient(s)/colleague(s) • takes full responsibility for own actions, able to admit mistakes • backs own judgement, expresses suggestions / decisions with

appropriate confidence • puts patient(s)/others’ needs before their own when appropriate • consistently acts within professional boundaries

Negative Indicators • lacks sufficient respect for others • avoids taking responsibility for poor decisions/ideas • is tentative when explaining decisions/actions • fails to prioritise patient(s)/others’ needs when appropriate • lacks a sense of operating within clear professional boundaries

ILLUSTRATION

COMMITMENT TO THE SPECIALTY Reasons for choice of specialty and relevant information. Positive Indicators • provides clear, comprehensive rationale for choice of specialty • reasoning demonstrates understanding of the specialty and of

own abilities / needs • demonstrates insight into own career motivations /

goals/development needs • additional information provided is relevant and supports

rationale / choice of specialty

Negative Indicators • unfocused, unclear reasons for choice of specialty • reasoning demonstrates a limited / inaccurate understanding of

the specialty and/or own abilities • shows little insight into own career motivations /

goals/development needs • additional information provided is irrelevant or unclear

LEARNING & PERSONAL DEVELOPMENT: Demonstrates realistic insight into and understanding of the specialty. Commitment to personal and professional development. Positive Indicators • realistic insight into positive & negative aspects of the specialty • provides evidence of efforts made to increase knowledge • is able to reflect and learn from experience/opportunities • recognises & makes use of available learning opportunities • identifies gaps in knowledge & organises appropriate

development; takes an active, structured approach

Negative Indicators • superficial and/or overly positive understanding of the specialty • provides little evidence of actions taken to learn about specialty • shows little evidence of learning from experience • demonstrates limited awareness of potential learning resources • takes a passive approach towards learning opportunities,

demonstrated little commitment to own development RELEVANT EXTRACURRICULAR ACTIVITIES / ACHIEVEMENTS

Positive Indicators • provides a range of examples or 1-2 significant examples • evidence is verifiable with dates and places indicated • provides clear explanation of how activities are relevant to the

specialty, e.g. in content or in relation to transferable skills • examples given demonstrates active commitment to own

development (professional and/or personal)

Negative Indicators • fails to provide examples of activities/achievements • gives little/no indication of dates / places, evidence is

unverifiable • examples given are unclear and their relevance to the specialty

is not explained • range of examples provided is very narrow or superficial • demonstrates little active commitment to own development

of 2:

Scoring Indicators for Shortlisting ENTRY AT ST1

Statements are provided below to give examples of behaviours that would indicate different levels of performance for each shortlisting question, although this is not intended to be an exhaustive list.

COMMITMENT TO THE SPECIALTY

Personal Statement: Reasons for choice of specialty and relevant information.

NE 1 2 3 4

No evidence

Unclear reasons for choice of specialty, lacked understanding of

specialty/own motivation, no or irrelevant additional

information

Rationale for choice of specialty somewhat

unclear/unfocused, limited insight into specialty

and/or own motivation

Clear reasons for choice of specialty, evidence of

some insight into specialty and/or own motivations

Comprehensive rationale for choice of specialty, clear understanding of

specialty/own motivations & relevant additional

information

Learning & Personal Development: Demonstrates realistic insight into and understanding of the specialty. Commitment to personal and professional development.

NE 1 2 3 4

No evidence

Inaccurate understanding of specialty, little evidence

of actions taken to increase knowledge or of

commitment to own development

Superficial understanding of specialty, steps taken to

increase knowledge are limited and/or passive in

approach

Realistic understanding of some aspects of specialty, evidence of active efforts

made to increase knowledge

Realistic insight into (positive & negative) aspects of specialty, evidence of active,

structured & reflective approach to learning &

development

Relevant Extracurricular Activities/Achievements

NE 1 2 3 4

No evidence Unclear & unverifiable

examples, no attempt to explain relevance to

specialty

Example(s) limited in scope and/or somewhat

vague/unverifiable, superficial attempt to link

to specialty

Verifiable example(s), some evidence of

relevance to specialty and/or personal

development

A range or 1-2 significant & verifiable examples,

relevance to specialty & own development clearly explained (e.g. content or

transferable skills)

CLINICAL, ACADEMIC & RESEARCH SKILLS

Clinical Knowledge & Expertise: Capacity to apply sound clinical knowledge and judgement.

NE 1 2 3 4

No evidence

Unsystematic approach to decision-making, failed to

explore or anticipate important information /

issues

Narrow or rigid approach, needed the full picture before understanding

problem, little awareness of own impact

Identified key issues involved, elicited

necessary detail from others and was aware of

appropriate options

Evidence of sound, systematic judgement,

thorough investigation of key issues / options,

understood impact of own actions

Research Skills: Understanding of the importance and basic principles of research.

NE 1 2 3 4

No evidence

Little understanding of purposes or impact of research, explanation /

argument was unclear or lacked relevance

Limited or superficial understanding of the

importance of research, little awareness of research methods

Understanding of the importance of research,

awareness of basic principles of research

methodology

Clear & accurate understanding of the

importance & purposes of research, used personal

experience to support argument where

appropriate

of 2:

Scoring Indicators for Shortlisting ENTRY AT ST1

Relevant Academic & Research Achievements: Including involvement in audit and experience of delivering teaching.

0 points 1 point 2 points

Additional qualifications No evidence Any additional degree(s) or relevant diploma(s)

PhD

Membership exam (OMFS only) No evidence MFDS

Courses (some specialties only) No evidence ALS, ALERTS or equivalent Paediatric life support course (paediatrics)

Prizes / distinctions No evidence Any undergraduate or postgraduate prizes / distinctions

Presentations / posters / publications

Presentation / poster within hospital / workplace

Regional / national presentation / poster Non-peer-reviewed publication

International presentation Peer-reviewed publication

Experience of audit No direct experience Experience of audit e.g. collected data, completed forms, attended meetings

Initiated or managed audit project, evidence of learning from experience

Experience of delivering teaching Informal teaching Formal teaching

PERSONAL SKILLS

Vigilance & Situational Awareness: Capacity to monitor developing situations and anticipate dangers/problems.

NE 1 2 3 4

No evidence Paid little attention to

changes in clinical picture, was unable to anticipate

dangers / problems

Picked up only on clear / obvious signs of rapidly

changing situations, struggled to maintain

vigilance for long periods

Able to identify symptoms and signs suggesting conditions that might

progress or destabilise rapidly, maintains vigilance

despite distractions

Alert to subtle signs and minimal cues suggesting

conditions that might destabilise rapidly, able to anticipate problems, aware

of team members’ performance

Coping with Pressure: Capacity to operate under pressure. Demonstrates initiative and resilience to cope with changing circumstances.

NE 1 2 3 4

No evidence

Became tense or defensive under pressure,

tried unsuccessfully to deal with situation alone,

inappropriate strategies to deal with stress

Found it difficult to remain calm under pressure or

seek help, limited strategies to deal with

pressure & stress

Recognised own limitations and was able to

compromise, identified strategies to deal with

stress

Remained calm under pressure, retained

perspective and sought help when necessary,

appropriate strategies to deal with stress

Managing Others & Team Involvement: Capacity to work effectively in partnership with others and demonstrate leadership when appropriate.

NE 1 2 3 4

No evidence

Confrontational approach, stuck rigidly to own

agenda & was unable/unwilling to

compromise

Little active involvement with others in achieving

goals, treated some colleagues more

favourably than others

Worked in partnership with others and respected their views, provided support

for others

Participative & non-confrontational approach,

able to compromise & build on others’ ideas, delegated effectively

PROBITY

Professional Integrity: Capacity to take responsibility for own actions. Demonstrates respect for all.

NE 1 2 3 4

No evidence

Avoided taking responsibility for poor

decisions/ideas, lacked respect for others, little

understanding of professional boundaries

Was tentative when explaining

decisions/actions, tried to find excuses for errors or

misunderstandings

Recognised responsibilities & backed

own judgement with appropriate confidence,

demonstrated respect for others

Took full responsibility for own actions, demonstrated

clear respect for all, consistently acted within professional boundaries

Exa

mp

leIn

terv

iew

Example Interview Report Form

Blank Interview Proforma

STRUCTURED INTERVIEW EVALUATION FORM

SPECIALTY ________________ LOCATION __________________ Candidate Name: Date: / / (MM DD YY) Selector Name(s): (Provide all selector names) INSTRUCTIONS: The interview contains ________ question areas. The interview should last no more than _____ minutes. We recommend that one interviewer records the candidate responses. The question areas to be addressed are directly linked to the Person Specification. Each question area should be covered in turn. Interviewer 1 asks the questions while the candidate responses are recorded by interviewer 2. Candidate responses should be written into the boxes provided. Ensure that this report form has the candidate’s name recorded and the date and time of the interview. The indicators on this Interview Scoring Form should be used in rating the candidate responses on a 4 point scale (see below). Procedures and definitions for scoring the interview appear in the evaluation manual. Introduce the interview by saying: “Thank you for coming, please take a seat. I am going to ask you to reflect on some specific experiences at work. You will have some questions regarding these events, so don’t be surprised if I stop you at various points. My colleague will be writing throughout” The following scale should be used to determine a score for each question area. 0 No evidence

• No evidence reported

1 Poor • Little evidence of specified positive behavioural indicators • Mostly negative indicators displayed, many of which decisively

2 Areas for Concern • Limited number of specified positive behavioural indicators displayed • Many negative indicators displayed, one or more decisively

3 Satisfactory • Satisfactory display of specified positive behavioural indicators • Some negative indicators displayed, but not clearly and decisively

4 Good to Excellent • Strong display of specified positive behavioural indicators (and possibly others) • Few negative indicators displayed, and these considered minor in status

NOTES (TO BE COMPLETED AFTER THE INTERVIEW)

Blank Interview Proforma

STRUCTURED INTERVIEW EVALUATION FORM 1 - POOR 3 - SATSIFACTORY 2 - AREA FOR CONCERN 4 - GOOD TO EXCELLENT

QUESTION SET 1 (COMMITMENT TO THE SPECIALTY)

Q1 Probes:

POSITIVE INDICATORS • • • • •

NEGATIVE INDICATORS • • • • •

Notes to Justify Rating RATING: 1 2 3 4

QUESTION SET 2 (JUDGEMENT UNDER PRESSURE)

Q2 Probes:




QUESTION SET 3 (SITUATION AWARENESS)

Probes:




OVERALL EVALUATION RATING: 1 2 3 4

Example Interview questions and recording form

STRUCTURED INTERVIEW EVALUATION FORM

SPECIALTY ________________ LOCATION __________________ Candidate Name: Date: / / (MM DD YY) Selector Name(s): (Provide all selector names) INSTRUCTIONS: The interview contains ________ question areas. The interview should last no more than _____ minutes. We recommend that one interviewer records the candidate responses. The question areas to be addressed are directly linked to the Person Specification. Each question area should be covered in turn. Interviewer 1 asks the questions while the candidate responses are recorded by interviewer 2. Candidate responses should be written into the boxes provided. Ensure that this report form has the candidate’s name recorded and the date and time of the interview. The indicators on this Interview Scoring Form should be used in rating the candidate responses on a 4 point scale (see below). Procedures and definitions for scoring the interview appear in the evaluation manual). Introduce the interview by saying: “Thank you for coming, please take a seat. I am going to ask you to reflect on some specific experiences at work. You will have some questions regarding these events, so don’t be surprised if I stop you at various points. My colleague will be writing throughout” The following scale should be used to determine a score for each question area. 0 No evidence

• No evidence reported

1 Poor • Little evidence of specified positive behavioural indicators • Mostly negative indicators displayed, many of which decisively

2 Areas for Concern • Limited number of specified positive behavioural indicators displayed • Many negative indicators displayed, one or more decisively

3 Satisfactory • Satisfactory display of specified positive behavioural indicators • Some negative indicators displayed, but not clearly and decisively

4 Good to Excellent • Strong display of specified positive behavioural indicators (and possibly others) • Few negative indicators displayed, and these considered minor in status

NOTES (TO BE COMPLETED AFTER THE INTERVIEW)


STRUCTURED INTERVIEW EVALUATION FORM 1 - POOR 3 - SATSIFACTORY 2 - AREA FOR CONCERN 4 - GOOD TO EXCELLENT

QUESTION SET 1 (EMPATHY & SENSITIVITY)

Give an example of how you won the trust of a worried and sceptical patient. Probes: Describe what you did and why? What effect did it have on both you and the patient?

POSITIVE INDICATORS • responded to needs/concerns with

interest/understanding • acted in open, non-judgmental manner • attempted to generate safe/trusting

atmosphere • made efforts to understand patient

concerns • reassured patient with appropriate

words/actions

NEGATIVE INDICATORS • showed little apparent interest in patient

situation • was quick to judge, made assumptions • little evidence of understanding patient

needs/concerns • unable to reassure patient effectively • appeared over-sensitive as if personally

involved • failed to generate safe/trusting atmosphere

Observation Notes to Justify Rating RATING: 1 2 3 4

QUESTION SET 2 (COMMUNICATION SKILLS)

Please describe a time when you have needed to explain a complicated procedure or term to a patient. Probes: How did you do this? What was the outcome?

POSITIVE INDICATORS • adjusted style of questioning/response

as appropriate • able to express ideas clearly to others • clarity in both verbal and written

communication. • flexible style to suit recipient(s) • tailored communication, established

relationship of respect with others

NEGATIVE INDICATORS • unable to adapt language/behaviour as

needed • imprecise response, rambled in

communicating, lacked focus • limited evidence of active listening • confused/ambiguous verbal communication • communication style was not tailored to suit

recipient(s), is functional, narrow


QUESTION SET 3 (PROBLEM-SOLVING)

Describe an example of when you have used creative thinking to solve a problem at work. (related to a patient or to the organisation) Probes: What did you do? What was the result?

POSITIVE INDICATORS • attempted to think “around” issue • open to new ideas/possibilities • generated functional, workable

solution • prioritised information/time effectively • identified key points • sifted peripheral information to detect

root cause

NEGATIVE INDICATORS • made immediate assumptions about

problem • dealt with issue narrowly or dogmatically • was unable to suggest “workable” outcome • was disorganised/unsystematic • focused on non-important/peripheral issues • unable to identify key issues/root cause



QUESTION SET 4 (ORGANISATION & PLANNING)

People organise their time in different ways. What approaches & strategies do you use to plan/protect your time for training? Probes: How do you prioritise conflicting demands? How do you justify your actions? What is most important?

POSITIVE INDICATORS • thinks ahead, prepares, plans

effectively • co-ordinates activities appropriately • builds contingencies to deal with

shifting demands • understands limitations, constraints &

works within them • prioritises conflicting demands • juggles competing demands,

delivered on time

NEGATIVE INDICATORS • little evidence of thinking ahead, pre-

planning, considering knock-on effects • unable to cope effectively with unexpected

occurrences • failed to re-prioritise & build effective

contingencies • fails to juggle competing demands • disorganised, fails to manage time

effectively


QUESTION SET 5 (PROFESSIONAL INTEGRITY)

Describe a time when you had to tell a patient that a mistake had been made, either by you or someone else. Probes: What did you do? How did you justify your actions? How did it affect your work afterwards?

POSITIVE INDICATORS • demonstrates respect for

patient(s)/colleague(s) • positive when dealing with problems • able to admit mistake/learn from it • committed to equality of care for all • put patient(s) needs before own when

appropriate • backs own judgment appropriately

NEGATIVE INDICATORS • lacked sufficient respect for others • treated issues as problems rather than

challenges • avoided taking responsibility for poor

decisions/ideas • failed to prioritise patient(s) needs when

appropriate • tentative in justifying decisions/actions


QUESTION SET 6 (COPING WITH PRESSURE)

Please describe a time when pressure at work has caused you to feel upset or angry. Probes: What was the cause of your feeling pressured? What did you do? What was the outcome?

POSITIVE INDICATORS • remained calm/under control • aware of wider needs of situation • recognises own limitations & able to

compromise • knows when to seek help • uses strategies to deal with

pressure/stress • responds quickly & decisively to

unexpected circumstances

NEGATIVE INDICATORS • shifted focus largely to immediate

worries/needs • defensive or uncompromising • tried unsuccessfully to deal with situation

alone • used inappropriate strategies to deal with

the issue(s) • hesitant & unsure when faced with

unforeseen circumstances



QUESTION SET 7 (COMMITMENT TO THE SPECIALTY)

What has influenced your decision to follow a career in this specialty? Probes: Why do you think you are particularly suited to this specialty? What other medical careers have you considered and why?

POSITIVE INDICATORS •

NEGATIVE INDICATORS •


OVERALL EVALUATION RATING: 1 2 3 4

Up

skill

ing

in

Sel

ecti

on

Wo

rksh

op

Upskilling in Selection Workshop:Selector Skills, Trainer Skills,

Trainer Slides, FAQs

Selector Skills – Best Practice

1. Introduction to Observing, Recording, Classifying and Evaluating (ORCE)

2. Shortlisting skills

3. Advanced Skills for Interviewers

1. Observing, Recording, Classifying & Evaluating (ORCE) Selectors must observe, record, classify and evaluate responses during the various elements of the selection process in order to implement the selection methods accurately and fairly. Research clearly shows that selectors require thorough and rigorous training for the following reasons:

• To ensure that selectors can make decisions with confidence. The only way that this can be enhanced is if each selector interprets the rating/scoring schedule in the same way and therefore uses it in the same manner as other selectors. This is a pre-requisite to achieve reliability of assessment.

• To ensure that selectors understand the various selection criteria on which the

selection methods (and ultimately the selection decisions) are based. This entails becoming familiar with the scoring indicators that accompany each competency so that components may be recognised during shortlisting/interview. A reference framework for target selection criteria is provided to help guide selectors. These are shown as positive and negative indicators against each selection criteria (see over the page).

• To understand the aims of shortlisting/interview thoroughly, familiarise selectors with

all the necessary materials, and develop an understanding of what candidates are required to do in each element of the selection process.

• To upskill selectors in the assessment procedure and in particular -

During the interview:

Observing the candidates’ responses.

During the interview:

Recording responses.

After shortlisting/interview:

Classifying recorded information. Evaluating evidence.

2. Shortlisting Skills In the case of shortlisting, selectors should:

1. Familiarise yourself with the targeted question area prior to scoring. The scoring indicators will help selectors in doing this. Scoring indicators are anchored to each point on a four point scale (1-4).

2. Read through the written response fully to gain an understanding of the response.

3. Indicate on the 1-4 descriptors your score for the response.

4. Note, there is a box for no evidence (NE)

General Rating Guidelines You may find the following guidelines useful in ensuring that ratings are consistent and fair:

1. A scale of 1-4 is given to assess candidates’ performance for most parts of the selection process. There is a tendency among raters to prefer the middle of the scale (2-3). This is ‘Central Tendency’. You should not be afraid of using the extremes of the scale as this will help to differentiate between candidates.

2. Beware of consistently over-rating or under-rating a candidate’s performance. This

means that you have tended to score at the top or lower end of the scale. This is important to ensure fairness.

3. Advanced Skills for Interviewers In this section we explore advanced skills for interviewers including levels of questioning, using probe questions and issues regarding note taking. We are assuming that everyone who comes forward for training has had experience in interviewing in the past and has been trained in Equal Opportunities. 3.1 Levels of Questioning in Interviews Questioning in a structured interview can be a like “peeling an onion”, but without the tears! This section outlines the concepts of questioning at different levels as a strategy for collecting useful and valid information during interviews. It is a proven strategy when you need to get information beyond the ‘facts’ and surface level (or even coached…) responses.

Facts/ Data

Attraction Meaning/

Values/ Attitudes

2

3

1

Level 1. Factual Information/Data Questions relevant at this level are the least threatening to the candidate and they are necessary to piece together the nature of their background, such as their employment history, qualifications and other relevant information. In this way, they provide the interviewer with a more complete and ‘whole person’ understanding of the candidate.

When you want to understand and judge a candidate’s values and attitudes, this involves 3 key levels of questioning (as highlighted in the diagram). That’s why it’s referred to as ‘peeling the layers of an onionBy probing further in an interview, the assessment allows you to elicit useful and valid information on which to make your judgement. Here we give examples of questions which may be asked at each leve

An example Level 1 questions are as follows;

Where did you complete you Foundation training?

What attachments have you completed?

How often did you meet with your educational supervisor?

Who helped you complete you work base assessments?

Level 2. Meaning, Attraction, Interest A third level of question relates to the meaning and importance something holds for the candidate. Questions at this level usually begin with:

What do you feel about……….?

What does ………… mean to you?

What do you get out of ……….?

At this level of questioning we are moving more towards the candidates ‘feeling’ about the specialty of choice (for example) and what meaning it has to them. This is important as an interviewer could make assumptions which could lead to invalid decisions being made. The intention is to peel off the layers to start uncovering meaning, attraction and interest. Examples: Here are some examples of the style of questions that might be asked at level 2.

What do you feel about working in a specialty that requires you to do ………..?

What does it mean to you to pursue a career in ………….?

What does it mean to you when ……………………?

When a patient does ……. how do you feel?

As an interviewer you may already have a certain amount of information on the candidate from their application form. However, you may need to check out some facts and you may need to establish facts about equivalence of certain qualifications, such as F2 competencies. The interviewer will need to ask several factual questions with follow-up probes to ascertain the facts.

Level 3. Values/Attitudes The next level of question is aimed at probing those items the candidate sees as important. The answers provided by the candidate will tell the interviewer what he or she places value on and why. They also provide information on the candidate’s attitudes. Therefore the answers from these questions are important in coming to a more complete understanding of the individual. This is particularly important because the interviewer will find this information very useful in reaching a judgement about a candidate’s attitudes and values. Questions here are usually of the type:

Why do you feel that ……………..?

Why is ……………… important to you? For example, if the question area relates coping with pressure and resilience, answers to this style of questioning might reveal (for example) that the candidate clearly places more value on patient safety than on workload. It might also reveal that whilst the candidate is willing to work long hours under pressure, they also recognise the need for a positive work-life balance. In this way, the interviewer could probe further to gain evidence and find out what the candidate actually does to maintain a positive work-life balance. However, it may be at this stage that you still need further evidence in order to make your judgement. It’s then necessary to probe further. Continuing to probe answers leads to more information about the candidate’s attitudes and values.

3.2 Using Probe Questions

Many people think that structured interviews are about delivering exactly the same question to all candidates and not following up on the candidate response. It is obviously important to deliver the same opening question to all candidates applying for the same post in any given area. Best practice structured interviews do have previously agreed question areas that directly link to the relevant Person Specification. However, the main responsibility (and skill) of the interviewer is to collect appropriate evidence relating to the question domain. Once the evidence is collected, an appropriate professional judgement can be made. The open question in an area should be consistently delivered between candidates. However, since interviews are an encounter between at least two human beings they are, by their very nature, idiosyncratic (no matter how much structure we impose). Best practice clearly shows that what makes the difference in quality of interviews is the extent and quality of interviewer probing. Probe questions are the follow-up to the opening questions. They help uncover the evidence needed to make a fair and valid assessment. A skilled interviewer, through the use of effective (not aggressive) probe questions a potentially ‘coached’ response can be exposed. On the positive side, a skilled interviewer will provide an opportunity for the candidate to fully describe their capabilities.

The following are a series of sample probing questions that will assist in exploring the level of depth and understanding about the applicant’s experience and the related competencies; and the actions taken specifically by the applicant, what lead to the actions, the results of the actions taken and what was learned from the experience. These questions elicit specific details about the who, when, where, what, why and how of the applicant's experience. Who:

• Who else was involved? • Who else helped you with this situation?

When:

• How long ago did this happen? • When did all this take place?

Where:

• Where did this take place?

What:

• What was the [situation/ issue / problem]? • What were the [results / outcomes]? • What actions did you decide to take, and when? • What was your role? • What part did you play in making the decision to _______? • What did you say, or do that [did / didn’t] work? • What did they do? • What was the presentation about? • What was the message you were trying to get across? • What were you trying to convince or persuade them of? • What options were available? • What steps did you take to ensure you met deadlines? • What techniques did you use to keep yourself and your work organized? • What did you do when your work started to get behind? • What steps did you take to get them all done? • What was the deadline? • What were the consequences of missing the deadline? • What actions did you take to ensure you met the deadline? • What needed organizing? • What improvements did you hope to see? • What did you learn from this experience? • What would you do differently if [you were asked to do it/ happened] again?

Why:

• Why was this situation a particular challenge? • Why was it important to convince or persuade them? • Why was this person difficult? • Why did you take that approach? • Why was it important to meet the deadline?

How:

• How did you develop this idea? • How did you convince your supervisor to adopt it?

• How did it help the organization? • How did you prepare for it? • How did you handle [it / that person]? • How did you size up the situation? • How was the challenge met or the problem finally solved? • How did you determine priorities in scheduling your time? • How well did this meet the needs of the business?

Other:

• In the past [6 months / 1 year], how often have you been called upon to do this? • In the past year, how often have experienced this type of difficulty? • Did you receive any feedback on your actions? If yes, from whom? What was said?

3.3 Observation & Recording in Interviews Extensive research shows over many years in the selection literature shows that there are many sources of error and bias that can act to reduce the reliability, validity and fairness of interviews as follows. Many of these sources of error are with regard to observation biases, as described below.

3.3.1 Observation: Potential Sources of Error The primary function of a selector during the interview is to observe and record what candidates say and do in response to the question they have been asked. This is not an easy task and certain pitfalls may be encountered which could influence the objectivity of the selector. These pitfalls have been identified by rigorous research on selection and assessment over several years and are outlined below.

• Discrimination: Most of us hold some form of stereotype about a person or group of people. These can be powerful barriers to accurate observation. Recognising your personal stereotypes can be useful in helping you overcome the prejudice they might normally create.

• ‘Haloes and Horns’: First impressions we make of people influence how we see

them thereafter. Research consistently demonstrates that the Halo Effect occurs when a person makes a positive impression and we then allow it to colour or influence everything they say or do. The opposite can also apply, so that a negative impression created in the first place could result in a poor outcome for the person. It is therefore extremely important that when we are observing behaviour we do so with a clear mind and we are aware of these potential sources of bias. We must aim to be fair and unbiased when observing and recording behaviour.

• Prejudice and ‘Red Rags’: We all have our ‘petty’ dislikes and points of view and

usually we don’t appreciate having these views questioned or ridiculed. It is therefore imperative when observing that these prejudices do not influence the way we observe and record the candidate’s responses. To ensure this objectivity, make a point of recording the behaviour as it actually happened and not how you interpret it. Similarly, the candidate may use a word, or may have a habit, which to you is like a ‘red rag to a bull’. The temptation is to become irritated and to stop listening. This must be avoided.

• Attribution Effects: Sometimes when we observe ‘inappropriate’ behaviour in a

person we attribute it to an inherent part of the person’s personality (whilst we see the same behaviour in ourselves as being due to circumstances or the situation). Be aware of this and remember that a person’s behaviour is a product of personality and the environment and during the selection process the environment can be stretching and pressured.

• Primacy and Recency Effects: Another human tendency in social encounters is to remember the behaviour observed or the comment made either at the beginning or the end of the encounter. The bit in the middle is often neglected. Ensure you have based your assessment on the whole of the interview by recording your observations as you go along.

• Concentration: It is extremely difficult to concentrate on something for more than 10

minutes without concentration starting to wane. Be aware of this issue as you might unintentionally miss important information. The more you can concentrate the more accurate your observations will be.

3.3.2 Recording Each selector must make written notes of what the candidate’s response was during the selection process. For panel interviews, we recommend at one interviewer delivers the question area, whilst another records the information. It is difficult to do both reliably. An observation/recording sheet should be available for panel members this purpose. You should make factual recordings and should not include any inferences about what the response or behaviour means. For example, you might write: “asked for question to be repeated twice”, or “sighed when asked for further details of research project” (rather than “did not understand question”, or “quickly became exasperated”). Factual information is transparent, easier to classify, helps scoring and is also more useful when discussing the candidate’s response and justifying the score. Avoiding inference at this stage has also been shown to improve the quality of the assessment. When recording behaviour you should try and follow the following guidelines:

• Accurate Recording: An accurate record should reflect all the relevant information for the candidates you are assessing. Do not leave the writing up until after the interview as information will become distorted. Always record during the interview itself.

• Complete Recording: Your records should be as complete an account as is

possible in the time allowed. This will provide an overall picture of what the candidate said and when and allow you to judge the comments of candidates in context. This will not be possible if you have failed to make thorough notes or missed chunks of dialogue. If you develop shorthand, make sure that it is easily understood and remembered and that the same shorthand applies to all candidates.

• Language and Movement: Record actions as well as words where appropriate, but

be descriptive rather than judgemental e.g. “maintained eye contact with all panel members”, rather than “good eye contact”

• Non-judgemental Records: All the records that you take of an interview should be

non-biased accounts of what actually happened. They should not reflect what personal feelings you had of what was said by the candidate, e.g. if the candidate points out that a decision would not make clinical sense, record “Candidate put forward point that decision would not make clinical sense”. Do not make judgements on the statement, e.g. “Candidate made a poor decision”. Again, your judgement on whether this was poor or excellent would come later in the process, after the interview has finished and you have gathered your evidence.

• Specific Records: Recorded statements should be specific and describe what took

place or was said, e.g. “Candidate stated that her particular strengths were honesty, an empathic approach and logical thinking skills” is more specific and accurate than “Candidate listed her key strengths”.

Selector Exercise: Examples of Interviewer Notes The following are examples of observation notes recorded by selectors as evidence of a candidate response in a structured interview. Some are effective and record what the candidate actually said. Other notes are ineffective examples as they are subjective judgements or are vague, general and do not describe the actual response. Place a tick in the appropriate column indicating if the statement is an effective or ineffective example.

Interviewer Notes Effective Example

Ineffective Example

1. Was very creative in his solutions to the problem posed

_______ _______

2. Contacted a senior colleague (the consultant) in the first instance to gain his views on the matter

_______ _______

3. Tested the understanding of the relatives before decision was made _______ _______

4. Was very insensitive to the needs of her colleagues _______ _______

5. Mature, considered approach to questions _______ _______

6. Suggested to colleagues that he make a note of the situation regarding the difficult patient

_______ _______

7. Presented a superb rationale for his decision _______ _______

8. Inappropriately dressed, not professional _______ _______

9. Couldn’t understand a word he was saying _______ _______

10. Referred him to a cardiologist for another opinion _______ _______

11. A wide ranging discussion of the issues _______ _______

12. Demonstrated a good knowledge of treatment options _______ _______

13. Recorded the conversation with the patient noting his refusal to take tablets to discuss with colleagues

_______ _______

14. Her standard of clinical competence leaves much to be desired _______ _______

15. Gave the patient an information leaflet, made a follow up appointment _______ _______

16. Spent 20 minutes talking with the patient, listened to his concerns _______ _______

17.

Lack of insight to the issues presented

_______ _______

18. Unable to cope with the pressures and demands of the situation _______ _______

19. Described the benefits and consequences of not taking the treatment to the patient

_______ _______

20. Disrespectful towards her seniors _______ _______

Trai

ner N

otes

Ex

erci

se 1

Pu

rpos

e

Lear

ning

Poi

nts

Pr

oces

s/ F

orm

at

H

ints

and

Tip

s

Ti

me

Allo

wed

(1

5 m

inut

es)

base

d on

25

dele

gate

s in

a

wor

ksho

p 5

min

utes

for

parti

cipa

nts

to

com

plet

e th

e ex

erci

se.

10 m

inut

es to

ru

n th

roug

h an

swer

s in

pl

enar

y

Dem

onst

rate

s ho

w c

andi

date

re

spon

ses

shou

ld b

e re

cord

ed.

O

bser

vatio

n no

tes

reco

rded

in

inte

rvie

ws

shou

ld b

e de

scrip

tive,

fa

ctua

l and

indi

cate

the

cand

idat

e’s

resp

onse

.

N

otes

sho

uld

not b

e ju

dgem

enta

l, ev

alua

tive

and

cont

ain

info

rmat

ion

that

isn’

t rel

evan

t and

inap

prop

riate

.

Rec

ordi

ng in

form

atio

n in

appr

opria

tely

redu

ces

valid

ity a

nd

in s

ome

circ

umst

ance

s co

uld

have

le

gal i

mpl

icat

ions

.

S

tage

1. P

artic

ipan

ts w

ork

on th

eir o

wn

initi

ally

to c

ompl

ete

the

exer

cise

.

Sta

ge 2

. Fa

cilit

ator

s re

ad th

e fir

st it

em

alou

d (“

Was

ver

y cr

eativ

e in

his

sol

utio

ns

to th

e pr

oble

m”).

Ask

del

egat

es to

rais

e th

eir r

ight

arm

as

to w

heth

er th

ey b

elie

ve it

is

an

effe

ctiv

e or

inef

fect

ive

exam

ple

of

obse

rvat

ion

note

s in

an

inte

rvie

w.

Th

en a

sk th

ose

who

giv

e th

e in

corr

ect

answ

er to

just

ify th

eir r

espo

nse.

Lis

ten

to

the

resp

onse

and

do

not g

ive

the

corr

ect

answ

er. T

hen,

invi

te th

e pa

rtici

pant

s w

ho

gave

the

corr

ect a

nsw

er to

exp

lain

why

. A

fter t

his

disc

ussi

on, g

ive

the

who

le g

roup

th

e co

rrec

t ans

wer

.

Allo

w 5

-10

min

utes

for

dele

gate

s to

com

plet

e on

thei

r ow

n.

Cor

rect

ans

wer

s de

liver

ed a

nd

disc

usse

d in

10-

15

min

utes

.

1

Best Practice Selection Best Practice Selection Methods:Methods:

ShortlistingShortlisting & Advanced & Advanced Interviewing trainingInterviewing training

National Selection into Specialty TrainingNational Selection into Specialty Training

ObjectivesObjectives

Provide Provide backgroundbackground & & rationalerationale to to national specialty selectionnational specialty selectionIntroduce Introduce key conceptskey concepts to best practice to best practice selectionselectionUpskillUpskill selectors in short listing & selectors in short listing & interview methodsinterview methodsProvide opportunities to Provide opportunities to practicepracticeselector skillsselector skills

TimetableTimetable

1.001.00 Context, key concepts & evidence Context, key concepts & evidence 1.301.30 Selection in practice I: Short listingSelection in practice I: Short listing

−− Selector skills Exercise 1Selector skills Exercise 1-- Evaluating CB questionsEvaluating CB questions−− Practical considerationsPractical considerations

2.302.30 Selection in practice II: InterviewsSelection in practice II: Interviews−− Introduction to core skillsIntroduction to core skills−− Selector skills Exercise 2 Selector skills Exercise 2

3.003.00 Tea Tea 3.153.15 Selection in practice II: InterviewsSelection in practice II: Interviews cont’dcont’d

−− Selector skills Exercise 3 Selector skills Exercise 3 −− Practical considerationsPractical considerations

3.453.45 Timetabling, logistics & evaluationTimetabling, logistics & evaluation4.304.30 Review & CloseReview & Close

Context for National Selection

ContextContextIn In ““Proposals for Reform of the SHO gradeProposals for Reform of the SHO grade””Sir Liam DonaldsonSir Liam Donaldson argues that: argues that:

““Reform must take account ofReform must take account of……weak selection weak selection && appointment procedures: appointment procedures: these are not standardised these are not standardised & & are frequently are frequently not informed by core competenciesnot informed by core competencies’’. .

UK Strategy Group agreement to national selectionPMETB Panel review 25th August 2006

Framework for National SelectionFramework for National SelectionCOPMeD Steering Group for Selection & Recruitment into Specialty Training

Key PointsKey PointsUK Strategy Group agreed framework for UK Strategy Group agreed framework for national selection July 2006national selection July 2006MTAS will be used by all 4 UK countries MTAS will be used by all 4 UK countries Nationally agreed person specs for each Nationally agreed person specs for each specialty & for each entry level (ST1, ST2, ST3) specialty & for each entry level (ST1, ST2, ST3) Nationally agreed application form, reference Nationally agreed application form, reference form, minimum standards for interviewform, minimum standards for interviewSelector training for short-listing & interview

essential

2

Why Change?Why Change?

Historically we have done an effective job.Historically we have done an effective job.Little formal evaluationLittle formal evaluationOur aim is to Our aim is to improveimprove efficiency & effectiveness efficiency & effectiveness for for allall stakeholdersstakeholders (trainees, admin, interviewers, HR)(trainees, admin, interviewers, HR)−− cost, fairness, reliability & validity (standards)cost, fairness, reliability & validity (standards)

ST1 is new: We ST1 is new: We mustmust deliver a process that is ‘fit deliver a process that is ‘fit for purpose’for purpose’

WhatWhat’’s new?s new?

This is the This is the firstfirst step. For 2007, there will be step. For 2007, there will be uniform targeting of agreed selection uniform targeting of agreed selection criteria (via the criteria (via the Person SpecificationsPerson Specifications). ). Minimum requirement of a Minimum requirement of a 30 minute 30 minute structured interviewstructured interviewIdentifying Identifying best practicebest practice (no re(no re--invention)invention)Resource pack Resource pack & opportunities to further & opportunities to further upskillupskill selectorsselectors

What levels will be selected into Specialist Training in 2007?

Specialty-specific − Levels recruited to depend on comparison

between competencies detailed in the new & old curricula

Recruitment will occur at ST1- ST4 in some Specialties

− Distinct application forms− Different recruitment rounds

How do trainees apply for How do trainees apply for Specialist Training?Specialist Training?

Advance information in Medical PressAdvance information in Medical PressSingle alert in Medical Press & electronic Single alert in Medical Press & electronic informationinformationSingle electronic application per recruitment Single electronic application per recruitment roundroundLongLong--listed nationally against person listed nationally against person specification specification Apply for up to 2 preferred Specialty groups in 2 Apply for up to 2 preferred Specialty groups in 2 preferred Units of Application* preferred Units of Application*

−− GP 1 GP 1 UoAUoA−− May no longer express ‘no geographical preference’May no longer express ‘no geographical preference’

*Unit of application- a Deanery or cluster of Deaneries working together

Recruitment & SelectionNational Advert

Central Co-ordination of Offers of Posts:0 4321

Successful

Round 1

Run-Through Grade Posts:2 Specialties,2 UOA &FTSTAs

E- Applications co-ordinated & sent to chosen UOAs*

Applications co-ordinated &sent to chosen UOAs*

Assessment RankingSelection

Shortlisting

Central Co-ordination of Offer of Post

Unsuccessful

Vacant Run-Through Grade Posts& FTSTAs

Round 2

UnsuccessfulRe-enter

Shortlisting Shortlisting ShortlistingShortlisting





Appointed toTraining

programme

Offers to Applicants

Training Programme

10 VacanciesAppointable

TraineesRanked 1- 45

1

45

Applicants will receive a single e-letter listing their offers and will have a limited period in which to reply

o Acceptance is a binding commitment to the UOA and the employer

o Applicants breaking the agreement are disqualified from any further participation in the recruitment and selection process except in very exceptional and unforeseen circumstances

3

Ranking of Applicants

Training Programme


Ranked 1- 45

1

45

There will be a continuous process of issuing more offers to appointable candidates in Round 1 of Selection

Allocation of Posts

Round A of Offers

Training Programme


Ranked 1- 45

Electronic generation of offers to trainees ranked 1-10

1

45

Allocation of Posts

Round A of Offers

Training Programme


Ranked 1- 45


1

45

5 Posts accepted by the trainees → 5 Vacancies filled

Allocation of Posts

Round B of Offers

Training Programme


Ranked 1- 45


1

45

Allocation of Posts

Round B of Offers

Training Programme


Ranked 1- 45


1

45

3 Posts accepted by the trainees → 8 Vacancies filled

Allocation of Posts

Round C of Offers

Training Programme


Ranked 1- 45


1

45

4

Allocation of Posts

Round C of Offers

Training Programme


Ranked 1- 45


1

45

1 Post accepted by the trainees → 9 Vacancies filled

Allocation of Posts

Round C of Offers

Training Programme


Ranked 1- 45


1

45

1 Post accepted by the trainees → 9 Vacancies filled

1 Post entered into Round 2

Selection Process Overview-The Timeframe

TimeframeAugust 2006-7

Aug Identify No of Programmes/ Posts Sept Agree person specification for ST1, 2 & 3

Jan Finalise complete selection/ interview schedule 6 Jan Alert in Medical Press- Application start

22 Jan- 2 Feb Submit applications2 Feb Application form to selection panels

By 23 Feb Shortlist applicants24 Feb One email confirming interviews or not

27 Feb Trainees confirm attendance or not28 Feb- 13 Apr Complete selection activities:Identify appointable applicants

2007

06

19-26 Apr Round 1 Offers made & accepted27 Apr Trainees with no offers asked to reapply

28 Apr Advertise unfilled postsBy 27 May Shortlist applicants

23 June One email of offers/ rejections4- 22 June Complete Selection

26 June Acceptance/rejection of offer

Round 1

Round 2Indicative

Implications of TimeframeSelectors will need to have the following availability:

Provisional DatesShortlisting

− 2- 23 Feb− 14- 27 May

Selection− 28 Feb- 13 April− 4- 22 June

Selection Panel should be available to complete short-listing within the timeframe as well as to attend the selection process for candidates

Key Concepts & Evidence: Best Practice Selection

5

Key conceptsKey conceptsSelection tools must be Selection tools must be standardised, fair, standardised, fair, defensible, reliable, valid, costdefensible, reliable, valid, cost--effective & effective & feasible. feasible. To achieve this, the skills & attributes tested at To achieve this, the skills & attributes tested at selection must be based on the appropriate selection must be based on the appropriate Person SpecificationPerson SpecificationThe Person Spec details the selection The Person Spec details the selection criteriacriteria & & methodsmethods..CandidateCandidate reactions are increasingly importantreactions are increasingly importantSelector skillsSelector skills central to reliability & validitycentral to reliability & validity

Research on Selection Systems

Research & implementation projects to Research & implementation projects to develop selection tools that are develop selection tools that are standardisedstandardised, , fair, defensible, reliable, valid, costfair, defensible, reliable, valid, cost--effective & effective & feasible. feasible. 1. Develop & validate selection criteria for various

specialties2. Design & validate selection tools that assess core

criteria

Growing literature in medicineGrowing literature in medicinePatterson et al. (Patterson et al. (BMJBMJ, 2005; , 2005; BMJBMJ 2001; 2001; BJGPBJGP, 2000; , 2000; Arch. Dis. Child. Arch. Dis. Child. 2006; TOG in press, 2006, RCS)2006; TOG in press, 2006, RCS)

Research on selection methodsResearch on selection methodsStructured interviews can be reliable & valid selection tools. Structured interviews can be reliable & valid selection tools. Reliability of Reliability of assessmentassessment & of & of assessorsassessors using using standardisedstandardised rating scales & documentation. Rigorous rating scales & documentation. Rigorous trainingtraining is critical. is critical.

Research shows Selection Centres (designed Research shows Selection Centres (designed appropriately) are best predictor of future job performanceappropriately) are best predictor of future job performance

RReliabilityeliability && validity validity gains gains -- combining different selection combining different selection tools, tools, standardised scoring systemsstandardised scoring systems to measure to measure key key competenciescompetencies egeg. 3 x 10 minute stations (= 30 minutes, with 2 . 3 x 10 minute stations (= 30 minutes, with 2 assessors per panel)assessors per panel)

Research Research consistently demonstrateconsistently demonstratess that application form that application form datadata & references& references hahaveve limited validitylimited validityValidityValidity can be can be significantly significantly improvedimproved by providing by providing standardised standardised competencycompetency--based based rating scales rating scales

Principles of Reliability & Validityin Selection

Defining reliability & validity

Reliability: “The instrument measures consistently under varying conditions”

Validity: “The instrument measures what it claims to measure”

Evaluating validity in selection1.1. Faith validityFaith validity

““The person who The person who sold me the sold me the selection tool/ test selection tool/ test was very plausible”was very plausible”

2.2. Face validityFace validity““The test looks The test looks plausible”plausible”

3.3. Content validityContent validity“The test looks “The test looks plausible to experts”plausible to experts”

4.4. Criterion validityCriterion validityPredictive validityPredictive validity

“The tool/test predicts who “The tool/test predicts who will be competent doctors”will be competent doctors”

5.5. Incremental validityIncremental validity“How much additional value “How much additional value does using another test/tool does using another test/tool provide?”provide?”

6.6. Construct validityConstruct validity““The tool measures The tool measures something meaningfulsomething meaningful””

6

Research on selection in medicineResearch on selection in medicinePractically, for some specialties we can’t interview all applicants.We need to shortlist. Options?– Psychometric testing, MCQs?

Academic success predicts very little of the variance in Academic success predicts very little of the variance in subsequent clinical performance subsequent clinical performance (e.g. Ferguson et al, BMJ; 2002)(e.g. Ferguson et al, BMJ; 2002)

For ST2, ST3 exams/research more relevant to indicate For ST2, ST3 exams/research more relevant to indicate competence. The is not possible for differentiating at ST1competence. The is not possible for differentiating at ST1

We recommend a combination of questions to add We recommend a combination of questions to add predictive power predictive power –– this has been pilotedthis has been pilotedMTAS can search for plagiarismMTAS can search for plagiarismFor 2007, this methodology is fit for purposeFor 2007, this methodology is fit for purpose..For the future, evaluation is necessary, on a national basis. For the future, evaluation is necessary, on a national basis. Future innovations could be a Machine marked test for Future innovations could be a Machine marked test for shortlistingshortlisting

Research on selection in medicineResearch on selection in medicine

Relevant research references for those interested:Relevant research references for those interested:Patterson, F., Ferguson, E., Norfolk, T., Lane, P. (2005). Patterson, F., Ferguson, E., Norfolk, T., Lane, P. (2005). A new selection system to recruit general practice A new selection system to recruit general practice registrars: preliminary findings from a validation study. registrars: preliminary findings from a validation study. BMJ, 330, 711BMJ, 330, 711--714714Patterson, F., Lane, P., Ferguson, E., Norfolk, T. (2001). Patterson, F., Lane, P., Ferguson, E., Norfolk, T. (2001). A competency based selection system for GP trainees. A competency based selection system for GP trainees. BMJ, Career Focus, 323:2BMJ, Career Focus, 323:2Patterson, F., Ferguson, E., Lane, P.W., Farrell, K., Patterson, F., Ferguson, E., Lane, P.W., Farrell, K., Martlew, J., Wells, A.A. (2000). A Competency model Martlew, J., Wells, A.A. (2000). A Competency model for general practice: implications for selection, training for general practice: implications for selection, training and development. and development. Br J Gen Pract, 50, 188Br J Gen Pract, 50, 188--193193Randall, R., Davies, H., Patterson, F. & Farrell, K. (in Randall, R., Davies, H., Patterson, F. & Farrell, K. (in press) Selecting doctors for postgraduate training in press) Selecting doctors for postgraduate training in paediatrics using a competency based assessment paediatrics using a competency based assessment centre Arch. Dis. Child. 91: 444 centre Arch. Dis. Child. 91: 444 -- 448.448.

Selection in Practice 1:Shortlisting

Shortlisting

• Documentation & scoring mechanisms for shortlisting will be on a national basis

• Nationally agreed person specifications, application forms & scoring frames created for each specialty at ST1, 2 and 3.

• This section is important for selectors to understand the paperwork & their role in the short listing process

National Documentation-The Person Specification

Person SpecificationsPurpose

− Usability (especially candidate perspective)− Standardisation within specialties− Operationalisation of assessment (e.g. AF,

interview questions, scoring framework)

Two sections:−Eligibility Criteria:

Minimum entry requirements, all essential−Selection Criteria:

Used to rank candidates Essential/ desirable criteriaSpecialty-specific criteria (some common domains)Commitment to the specialty

7

Person Specification eg ST1 Anaesthesia

Essential Eligibility:• GMC Registration• FP Competencies

Career Progression:• No unexplained career gaps• Less than 12 months experience

(at SHO level) in this Specialty (not including FP modules) by August 2007

Personal Skills:Vigilance & Situational Awareness,Coping with Pressure, Managing others & team involvement,Problem-solving & decision making, Empathy & Sensitivity, Communication Skills, Organisation & Planning

Person Specification ST2 and ST3 Anaesthesia

Essential Eligibility: Career Progression:

Differ from ST1 in the entry criteria

ST1 • GMC Registration• FP Competencies

• No unexplained career gaps• Less than 12 months experience

(at SHO level) in this Specialty (not including FP modules) by August 2007

• GMC Registration• FP & ST1 Competencies

• No unexplained career gaps• At least 12 months and less

than 3 years experience (at SHO level) in this Specialty (not incl. FP modules) by Aug 2007

ST2

• GMC Registration• FP & ST1&2 Competencies• Primary FRCA examination

passed

• No unexplained career gaps• At least 2 years experience (at

SHO level) in this Specialty (not including FP modules) by August 2007

ST3

National Documentation-The Application Form

Application FormsApplication Forms

EligibiltyEligibilty−− Personal details, GMC Registration, Professional Personal details, GMC Registration, Professional

Qualifications, Eligibility for entry into Specialist Qualifications, Eligibility for entry into Specialist training, Previous posts, Refereestraining, Previous posts, Referees

Selection CriteriaSelection Criteria−− Clinical, Academic & Research skills, Probity, Clinical, Academic & Research skills, Probity,

Evidence of commitment to the Specialty, Personal Evidence of commitment to the Specialty, Personal skills relevant to the Specialtyskills relevant to the Specialty

Questions directly linked to person specificationQuestions directly linked to person specification

Application ProcessApplication Process

Specialty specific formsSpecialty specific formsLong listing via MTAS (electronic portal)Long listing via MTAS (electronic portal)ShortlistingShortlisting in in UoAUoA by trained by trained shortlistersshortlisters(minimum of 2)(minimum of 2)Lay involvement, determined by the Lay involvement, determined by the UoAUoA(as per Orange Guide)(as per Orange Guide)Scoring frames providedScoring frames provided

Shortlisting Process

8

Short-listing Process Panel members carry out shortlisting (as per Orange Guide)

− Panel members do not have to mark ALL of a single application form.

− Deaneries may choose to split questions between panel members.

Benefits in terms of speed & calibrationDifferent methods/options:

− All in one room – marking/ decisions− Sent out to short-listers & returned− electronic

GMC reporting for falsification on Application Form

Shortlisting Scoring & Indicators

What paperwork do I need for shortlisting?Materials1. Framework reference for targeted criteria

This helps to ensure that all selectors have a shared understanding of the criteria being assessed

2. Shortlisting scoring indicatorsThis is where scores are recorded for each question against specific descriptors/ ‘word pictures’

3. Candidates responseThis may be paper based or on-line, depending on your UoA

Example Questions: Application Form

Specialty-SpecificClinical, academic & research skills

[Vigilance & Situational Awareness] Describe a time when you identified a clinical situation that was changing rapidly. How did you identify this and what did you do? (max. 150 words)

GenericClinical, academic & research skills

What do you think is the relevance of medical research to a doctor? If you have been involved in research, use examples from your own research to illustrate this. (max. 150 words).Describe your experience of clinical audit. Indicate clearly your level of involvement and what you learned about the process.

Example Questions: Application Form

What steps have you taken to develop your understanding of this specialty? (max. 150 words)

Please provide a brief statement describing the reasons why you have chosen to train for a career in this specialty. Please include any information about yourself that you feel is important or relevant in this regard. (max. 75 words)

Framework for Target Selection CriteriaClinical, Academic & Research Skills

9

Indicators for Target Selection Criteria at Shortlisting-Commitment to the Specialty

Framework for Target Selection CriteriaPersonal Skills

Framework for Target Selection CriteriaProbity

Shortlisting Scoring Indicators

Comprehensive rationale for choice of specialty, clear understanding of

specialty/own motivations &

relevant additional information

Clear reasons for choice of specialty,

evidence of some insight into specialty and/or own motivations

Rationale for choice of specialty

somewhat unclear/unfocused, limited insight into specialty and/or own motivation

Unclear reasons for choice of

specialty, lacked understanding of

specialty/own motivation, no or

irrelevant additional

information

No evidence

4321NE

Personal Statement: Reasons for choice of specialty and relevant information.

COMMITMENT TO THE SPECIALTY

Candidate Responses I’m shortlisting, what do I do?Read the question you are scoringFamiliarise yourself with the framework indicators for the targeted criteria to ensure understandingRead the candidates responseUsing the scoring indicators, circle one score based on the evidence presented

Training essential to ensure understanding of criteria & callibration of assessors

10

Evaluation & rating process

Standardised system Standardised system of documentation & of documentation & scoring framework scoring framework allows for more allows for more validvalid & & fair assessmentsfair assessmentsReduces potential for error & biasReduces potential for error & biasIncreases transparencyIncreases transparency

An example……..


Evaluating & scoring the application form

Practical considerations in Shortlisting

Work needed on calibration & reliability Work needed on calibration & reliability Each question marked by 2 different short Each question marked by 2 different short listerslistersScores aggregated to get total & rangeScores aggregated to get total & rangeScores returned to MTAS including Scores returned to MTAS including UoAUoA cut offcut offSystem allocates interview according to System allocates interview according to UoAUoAinterview capacity & candidate scores interview capacity & candidate scores

−− egeg if 100 interview slots, the top 100 candidates on if 100 interview slots, the top 100 candidates on shortlistingshortlisting offered interview. As these slots are offered interview. As these slots are accepted or declined, interview is offered to next on accepted or declined, interview is offered to next on list until you reach cut off, or none left.list until you reach cut off, or none left.

Selection in Practice 2:Interviews

Selector Skills - The Interview

Interviews in SelectionInterviews in Selection

2 main types of interview in selection− Behavioural (past behaviour = best predictor of future behaviour)

− Situational (scenarios)

− Both useful, dependent on interviewer skills

Skills: Peeling, probing, familiarisation, observerecord, classify, evaluate

11

Example Situational Interview Question Negative Indicators

• insist on operating• threaten a critical incident• consults senior anaesthetist

without discussing with SpR• tell the patient anaesthetist

“refusing” to anaesthetise• just accepts it

Positive Indicators• consider the reasons (eg

overnight, co-morbidity)• review the decision to operate

and assess priority• discuss it with the anaesthetist• involve the surgical senior cover• deal with the patient

appropriately

EVALUATION 1 2 3 4

Management Scenario:Anaesthetic SpR refuses to anaesthetise a patient for appendicectomy

Negative Indicators• Failing to take a history of the nature

of the accident• Failure to take adequate history

including possibility of asthma• Failure to examine patient including

trachea • Sending patient for CXR without

excluding tension pneumothorax• Failing to assess circulation in arm

Positive Indicators• Structured approach to the question• Awareness of possible vascular

injury to arm • Awareness of possible

pneumothorax• Of risk of associate intra-abdominal

injury – ruptured spleen • Awareness of risk of head injury

EVALUATION 1 2 3 4

Clinical Scenario: 12 yr old boy – who has come off his BMX –he has a broken arm (supracondylar fracture) and is complaining of breathlessness.

Fairness in Interviews

Forbidden Questions Forbidden Questions Sex DiscriminationSex Discrimination

Do you intend to have a family in the near Do you intend to have a family in the near future?future?I see you are engaged to be married. Does I see you are engaged to be married. Does that mean you will be moving away from that mean you will be moving away from the area?the area?Do you think you would be able to gain the Do you think you would be able to gain the respect of males who report to you?respect of males who report to you?

Forbidden QuestionsForbidden QuestionsRacial DiscriminationRacial DiscriminationWe like our employees to look smart. Do you We like our employees to look smart. Do you have to wear a turban and a beard?have to wear a turban and a beard?Why do you call yourself British with an Why do you call yourself British with an accent like that? accent like that? Would you, like so many other Indians I have Would you, like so many other Indians I have employed in the past, keep wanting long employed in the past, keep wanting long holidays to go back to India?holidays to go back to India?Being Irish, are you going to lose your Being Irish, are you going to lose your temper if annoyed with fellow workers?temper if annoyed with fellow workers?

12

Familiarise

Interviewers need to know: Interviewers need to know: −− the questions they are askingthe questions they are asking−− how they relate to the person specification how they relate to the person specification −− what probes are appropriatewhat probes are appropriate

Be familiar with the relevant indicators for Be familiar with the relevant indicators for the questionthe questionBe familiar with the rating scale to be usedBe familiar with the rating scale to be used

What’s the difference between a skilled interviewer & novice?Questioning styles…..Questioning styles…..open questions, one at a timeopen questions, one at a timeManaging silences….Managing silences….Making the question feel authentic! FollowMaking the question feel authentic! Follow--up up essential to uncover what “you” (the candidate) essential to uncover what “you” (the candidate) actually didactually did

PeelPeel the layers the layers 1.facts 2. meaning 3. attitude1.facts 2. meaning 3. attitude

ProbeProbe for evidence for evidence who, what, why, when…etcwho, what, why, when…etc

Verify what the trainee actually did, or would do Verify what the trainee actually did, or would do in scenariosin scenarios

321

Example Interview Report Form

Exemplar Interview Report Form


Questioning StylesQuestioning Styles

Observation: Sources of error & bias

Haloes & Horns effectsHaloes & Horns effectsPrejudice (race, gender, etc.)Prejudice (race, gender, etc.)Stereotyping & ‘Red Rags’Stereotyping & ‘Red Rags’ConcentrationConcentration

−− ‘Primacy & recency’ effectseffects

Observation biasObservation bias−− ‘Similar-to-me effect’/ Personal Liking

Temporal extensionTemporal extension− Decision made within 3 seconds

First impressions last First impressions last − “Can I suspend judgement?”

13

‘Real World’ Selector Stereotypes

white stilettoswhite stilettoslimp handshakeslimp handshakeswhite sockswhite socksmale earringsmale earringstattoostattoosMillwallMillwall fansfansmale long hairmale long hairsweaty palmssweaty palms

designer stubbledesigner stubblecloseclose--set eyesset eyesdoubledouble--barrel barrel surnamessurnamesVolvo driversVolvo driversperoxide hairperoxide hairheavy makeheavy make--upupbelly showing!belly showing!

A TRAINING POST

RecordingLegible!AccurateComplete

− If in doubt, include itNon-judgmental

− Record appropriately, do not judge or evaluate while recording

Specific – need to be able to provide explanation for judgment later on

Selector skills Exercise 3

Effective recording & evaluatingEffective recording & evaluating

Evaluation & Rating Process

4 = good to excellent3 = satisfactory 2 = areas of concern 1 = poor [0] = No evidence

Standardised system of markers allows formore valid & fair assessments & increases transparency

Interviews/Selection Centres

MethodsMinimum of 30 minute structured interviewQuestions/exercises must be;− Based on the relevant Person Spec. − Consistent across interviewers/ interviewees− Previously agreed scoring framework that links to the

Person SpecExample questions & calibration exercises provided

14

Interviews/Selection CentresLogistics

Chairperson/ facilitator appointedVenue, admin support, materials planned in advance

ContentMinimum of 30 minutes contact− Panel interview; Split-panel (two 15 minutes); Three

10 minute exercisesMultiple selection tools encouragedSelection centres (focus also on demonstrating, not reporting)

SelectorsAll trained, lay involvement essential

Candidate reactionsFactors affecting reactions use organisational justice theories:

− Distributive justice: Perceived fairness regarding equity (outcome consistent with expectations) & equality (opportunities).

− Procedural justice: Information, feedback, job-relatedness, recruiter effectiveness.

Applicants prefer multiple opportunities to demonstrate skills

National Documentation-The Structured ReferenceTab 5

Reference Reports

Evidence suggests that choice of reference provider & structure of the form have the most impact on the quality of information providedDesigned appropriately & in relation to the person specification, they can be improvedA standardised, structured report form has been designed and nationally agreedMinimum of 2 referees, covering the last two years of employment. Candidate guidance on selecting these.Guidelines on how & where the reference report is used in the selection process is provided (at interview stage only and via the Chair of the panel)

Structured Reference (Draft)

National Documentation-Candidate evaluationTab 10

15

FeedbackFeedbackSelection decision uploaded to MTAS; offers Selection decision uploaded to MTAS; offers ONLYONLYby MTAS at the end of the selection process, or the by MTAS at the end of the selection process, or the national system will not worknational system will not workFeedback offered Feedback offered only afteronly after the whole of the the whole of the selection process. selection process. This is This is essentialessential so there is fairness to all so there is fairness to all candidatescandidatesWe must ensure fairness to all candidates We must ensure fairness to all candidates UoAUoA s will be asked to s will be asked to auditaudit to ensure national to ensure national adherence adherence


Practical considerations

TimetableLogisticsCandidate experience

ReviewNational specialty selection in 2007 includes national National specialty selection in 2007 includes national person specifications, application forms & referencesperson specifications, application forms & referencesKey conceptsKey concepts in best practice selection inform new in best practice selection inform new practicepracticeShortShort--listing & interview questions listing & interview questions MUSTMUST be directly be directly linked to person specslinked to person specsSelector skills training vital to contribute to Selector skills training vital to contribute to standardised, fair, defensible, reliable, valid, coststandardised, fair, defensible, reliable, valid, cost--effective & feasibleeffective & feasible process.process.Selection system not static Selection system not static –– continuous continuous improvementimprovement

Final questions?

Thank you……

Eva

luat

ion

&A

ud

itD

ocu

men

tatio

n

Evaluation & Audit Documentation

Best Practice Selection

1

Evaluation

1. Principles of best practice evaluation in selection 2. Auditing process


2

1 Principles of best practice evaluation in selection Evaluation is critical in the development of the future selection process to identify areas for improvement and ensure adherence to PMETB principles and standards for entry to Specialist Training. Figure 1 provides an outline of the main elements involved in designing and implementing a personnel selection procedure and how evaluation forms an integral part of this.

Fig. 1 The personnel selection, design and validation process

Before considering how well selection methods work it is necessary to be clear about what it means for a selection method to work. A list of various features of personnel selection methods that are important is presented in Table 1. A comprehensive evaluation of any selection method would require a thorough examination of the method in relation to each of the features given below. Table 1 Major evaluative standards for personnel selection procedures

1. Discrimination

The measurement procedures involved should provide for clear discrimination between candidates.

2. Validity and Reliability

The technical qualities of the measurement procedures must be adequate. Several different (e.g. content, predictive or concurrent) validation processes are available for assessing validity.

Job Analysis To derive a;

• Person specification • Job description • Competency model

Identify Selection Criteria

Attract Candidates

Self-selection to the job role

Choose Selection Methods Judgements based on reliability, validity, legality, fairness, cost and candidate reactions

Implement Selection Methods and Make Selection Decisions

Evaluation • Evaluate candidate reactions • Empirical validation studies • Utility assessment


3

For example, if you demonstrate that your assessment tool (for example, a structured interview) is measuring competencies that are critical to successfully performing a given job, you have content validity evidence. If you demonstrate that your assessment tool (for example, a written test) predicts actual performance on the job, you have evidence of the predictive validity of the assessment tool.

3. Legality & Fairness

The measures must not discriminate unfairly against members of any specific subgroup of the population (e.g. ethnic minorities).

4. Administrative Convenience/Practicality

The procedures should be acceptable within the organisation and capable of being implemented effectively within the organisation’s administrative structure. Those administering the procedures may need appropriate training.

5. Cost and Development Time

Given the selection decisions (e.g. number of jobs, number of candidates, type of jobs) involved, the costs involved and the time taken to develop adequate procedures need to be balanced with the potential benefits. This is essentially a question of utility.

6. Candidate Reactions

Candidate reactions have important consequences for an organisation. In an extreme case, an applicant who has a negative reaction to a selection procedure could make a legal challenge on the basis of unfair discrimination. Research has tended to explain the different factors that affect applicant reactions using theories of organisational justice:

• Distributive justice focuses on perceived fairness regarding equity (where the selection outcome is consistent with the applicant’s expectation) and equality (the extent to which applicants have the same opportunities in the selection process).

• Procedural justice refers to the formal characteristics of the selection process such as information and feedback offered, job-relatedness of the procedures and methods, and recruiter effectiveness.

Four main factors seem to account for positive or negative applicants reactions where selection methods are;

(1) based on a thorough job analysis and appear more job relevant

(2) less personally intrusive,

(3) do not contravene procedural of distributive justice expectations, and

(4) allow applicants to meet in person with the recruiters.

Candidates also seem to prefer multiple opportunities to demonstrate their skills and that the selection system is administered consistently for all candidates.


4

10.2. Auditing process The information collected at selection is invaluable at helping design training and developments plans and activities. In accordance with best practice guidelines in terms of evaluation of the selection process, the following has been agreed with the PMETB (August 2006): .

(1) Selectors, applicants and administrators will be asked to complete feedback proforma and evaluation forms regarding the selection process (including candidate reactions regarding perceptions of fairness and procedural justice). Results will be reported to relevant stakeholder groups for modification of the existing process if appropriate

(2) Deaneries/UoAs will be asked to complete a self-assessment of selection methods as part of

the auditing of this process, which could inform the statutory quality assurance process Examples of feedback proformas for this purpose are presented in this section.

Comments on application form questions

This form asks about your experience of going through the specialty training selection process. The answers

you give will not form part of your selection, they are only to help us improve the selection system in the

future. Please answer each question by ticking the appropriate box according to the scale provided, wewould also be grateful to receive your comments in the spaces provided.

Age: _____ Gender: Male Female

Ethnic origin: Area of Qualification:

White Mixed UK Africa

Asian Chinese Europe Other

Black Other Asia

1= strongly disagree, 2= disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree

Application Form Questions 1 2 3 4 5

1. The content of the application form questions was clearly related to the job.......

2. A person who scored well on the application form questions will be a good doctorin this specialty

3. The content of the application form questions did not appear to be unfair...........

4. The content of the application form questions seemed appropriate….................

5. How long did it take you to complete the application form questions?

Hours: ___________ Minutes: ______________

1= strongly disagree, 2= disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree

Candidate’s Evaluation of the Selection Process

The Interview 1 2 3 4 5

1. The content of the interview was clearly related to the job………………

2. A person who scored well on the interview would be a good doctor in this

specialty……….

3. The interview gives applicants the opportunity to show what they can really

do………………………………………………………………………………

The whole Selection Process

1. I was treated politely during the selection process………………………………

2. The staff involved in the interviews treated candidates with respect duringthe interview process…………………………………………….

3. I was satisfied with my treatment at the interview…………………….

4. The content of the interview did not appear to be unfair………………..

5. The content of the interview seemed appropriate……………………….

6. Overall I was given a good opportunity to show my skills and abilities today...

7. The interview was conducted with a high degree of professionalism….

8. The interview centre helped me to learn something about myself……………..

We are interested in your experience today in comparison to other selection procedures you have been

through during your medical career. Please circle the appropriate answer for each question.

1. In comparison to other selection processes I have been through, the level of professionalism shown at

interview was…

A lot less Less Slightly less The same Slightly more More A lot more

2. In comparison to other selection processes I have been through, the relevance of the content of the

interview questions to work as a doctor was…


3. In comparison to other selection processes I have been through, the level of opportunity I had to show my

skills and abilities today was…


Please comment on your experience of going through the selection process. We would be particularly

interested in how your experience today compares to other selection procedures you have been

through, and whether it has helped you to learn anything about your own skills and developmentneeds.

Thank you for completing this evaluation form

This selection round sees a large change to the selection process for specialty training. We are very keen to receive feedback from you about how this has worked. This page asks for your comment on the short listing and interviewing process. Interviews: we would like to know your views about: a) The content of the interview questions – did they give candidates a proper chance to demonstrate competencies? Were there any issues around fairness? b) Was there anything about the questions that made them easy or difficult for you to assess? c) Was there anything that made the practical set up and administration of the interviews particularly easy or difficult?

Selectors Evaluation of the Selection Process

Application form questions – Short Listing: Interviews: General Comments on the selection process:

Thank you for completing this evaluation form

Deaneries/UoAs will be asked to complete a self-assessment of selection methods as part of the auditing of the selection process. This will inform the statutory quality assurance process and will be monitored by PMETB as part of the associated statutory quality assurance process.

Unit of Application Appointment Level Specialty Name of Chair

Selection Process: Please indicate with a TICK either Yes or No against the statements

YES NO 1. Membership of the interview panels/ selectors at selection centres complies

with guidelines set out in the Orange Guide to appointments to specialty training (or future equivalent documentation)

2. A chair has been appointed locally for the selection process, whose responsibility it is to quality assure delivery of selection methods to ensure compliance with PMETB standards

3. All persons involved in interview panels/ selection centres have been trained in the principles of best practice selection.

4. Selection panels include a lay person to participate in the selection process. The lay person/people involved must also be trained in best practice selection and assessment

5. Interview questions/selection centre exercises are directly and clearly related to criteria contained in the person specification

6. Interview questions/selection centre exercises were scored according to the national standardised scoring framework

7. Relevant feedback will be provided to applicants if requested

8. Selectors and candidates have completed evaluation forms regarding the selection process

Signature of Chair Signature of HR Date

Audit of Interview/ Selection Centre Process: Chair to complete

Comments on Selection Process

Recruitment to Specialist Training

Documents

Transcript of Recruitment to Specialist Training