Clinical Skills Verification Rater TrainingMODULE 3

Setting the Clinical Skills Verification Examination Climate and Giving Feedback

Karen Broquet , M.D.

Eugene Beresin, M.D.

Nyapati Rau , M.D.

Module 3Pre-test

1. Which of the following is assessed in the Clinical Skills Verification Evaluation?

a. Patient Doctor Relationship

b. Clinical Formulation

c. Differential Diagnosis

d. Treatment Planning

Module 3Pre-test

2. In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid?

a. Arrange Mock Oral Board-style evaluations

b. Use Standardized Patients

c. Perform the exercise during routine work

d. Make use of the Hawthorne Effect

e. Add a multiple choice test following the interview

Module 3Pre-test

3. Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation?

a. Any rating form can be used

b. Process notes provided by the resident

c. Faculty should use a rating form he/she most prefers

d. Faculty should all use the same rating form

Module 3Pre-test

4. Which of the following is considered formative assessment?

a. Year-end reviews of performance

b. Pass/fail examinations

c. Specific feedback of behavior in real time

d. Anonymous 360 degree evaluations sent to the resident

e. Periodic summative evaluations

CSV Exam MUST Assess…

• Ability to establish doctor-patient relationship

• Conduct a clinical interview

• Oral presentation of patient history and clinical findings

But can (and should) ALSO be….

• Valuable educational experience

• Opportunity for• Direct observation of clinical

skills by faculty• Formal evaluation of clinical

skills• Focused feedback

Feedback is Formative

Process by which the teacher provides the learner with information about his/her performance for the purpose of improving performance (Ende 1983)

• Without specific and focused feedback, learners:• Rely on self-assessment (unreliable!)• May establish undesired behaviors and/or

extinguish desired behaviors

Evaluation is Summative

“Did I Pass the exam!?!”

Feedback = Coach

• Timely, immediate, ongoing

• Direct

• Verbal

• Face to face

• Guiding future performance

Evaluation = Judge

• Summative

• Not ongoing, occurs at the end of time together

• Assesses whether or not learner met performance standards

Quality of both feedback and evaluation depends on the observer’s detection, perception and recall of the resident’s

performance

How to maximize this:• Incorporate direct observation into clinical

teaching routine• Brief (daily rounds) and not so brief (CSV exam)• Normalizes observation/assessment experience• Hawthorne effect: the more we know we are

assessed, the better we perform• Write it down!

• Specific, accurate feedback is better feedback• Also helps with accurate evaluation

• Prepare for the encounter

Setting Expectations

• How long will the encounter be?• Who will be observing?• What are the goals of the encounter?

• i.e., is this a formal CSV evaluation?• Assessment parameters?• What rating instrument/form will be used?• When will “results” be given?• What are the consequences of “passing” or

“failing” a CSV exam?

What Rating Form/Instrument to Use?

• Multiple forms accepted by the ABPN

• Forms vary by level of detail

• All faculty/CSV raters in a program should be trained on and use the same form

How Long Should the CSV Exam Be?

• ABPN: “about” 30 minutes for interview and 15 minutes for presentation

• AADPRT CS Assessment Task Force: 30-45 minutes for interview, 60-75 minutes total

• Programs should choose one format and use consistently

Guidelines for giving effective feedback – whatever the setting/encounter

Effective/Meaningful Feedback is…

• approached with teacher and learner working as allies with common goals

• given immediately in a private environment• well timed and expected• based on first-hand (observed) data• focused on specific behaviors, not

generalizations• limited to behaviors that are remediable

Effective/Meaningful Feedback is…

• regulated in quantity (no more than 2-3 items for improvement)

• limited to actions and decisions, not assumed intentions or interpretations

• phrased in descriptive, nonjudgmental language

• closed with suggestions for improvement and a mutually agreed upon action plan

Barriers to Effective Feedback

Faculty• Time

• Lack of feedback skills

• Feels like being critical

• Want learners to like us

• Gestalt vs specific observations

• Impact of CSV ‘failure”

Resident• Time

• Lack of feedback experience

• Feels like being criticized

• Don’t want to expose weaknesses

Residents CRAVE feedback

To minimize barriers, faculty can…

• Understand the profound and beneficial educational value of feedback

• Provide the proper climate and preparation for the experience

• Elicit reciprocal feedback from trainees• Gain the skills for giving both constructive

and reinforcing feedback• Use a consistent, common rating instrument

Cultural Phenomena – International Medical Grads

• Lack of feedback Experience• Diverse cultural backgrounds• Power dynamics between teacher and learner• Accepted communication patterns• Attitudes towards critique• Vulnerability in evaluation • Stress and difficult social situations • Experience with discrimination

Important for all learners – especially IMGs

• Learner centered approach

• Explicitly prepare the resident for the expectations of the exam and the provision of feedback

• The importance of the teacher and the IMG working together cannot be over emphasized.

• Create a safe learning environment and clear expectations and educational objectives

Important for all learners – especially IMGs

• Invite learner to indicate his/her learning needs and provide a self-assessment.

• Normalize feedback as regular educational activity that includes areas to be reinforced and areas requiring improvement.

• Be clear when feedback is formative or summative (i.e evaluation)

Module 3Post-test

1. Which of the following is assessed in the Clinical Skills Verification Evaluation?

a. Patient Doctor Relationship

b. Clinical Formulation

c. Differential Diagnosis

d. Treatment Planning

Module 5 Post-test

1. Which of the following is assessed in the Clinical Skills Verification Evaluation?

a. Patient Doctor Relationship

Diagnosis, biopsychosocial formulation and treatment planning are not assessed

Module 5Post-test

2. In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid?

a. Arrange Mock Oral Board-style evaluations

b. Use Standardized Patients

c. Perform the exercise during routine work

d. Make use of the Hawthorne Effect

e. Add a multiple choice test following the interview

Module 5Post-test

2. In assessment of clinical skills evaluations, which of the following methods is most likely to demonstrate what a resident actually “does” according to Miller’s pyramid?

c. Perform the exercise during routine work

Module 5Post-test

3. Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation?

a. Any rating form can be used

b. Process notes provided by the resident

c. Faculty should use a rating form he/she most prefers

d. Faculty should all use the same rating form

Module 5Post-test

3. Which of the following rating methods is recommended for a documented Clinical Skills Verification Evaluation?

d. Faculty should all use the same rating form

Although the ABPN has approved a variety of different rating forms for the CSV exam, consistent use of and feedback regarding the same form by all raters in a program is associated with improved inter-rater reliability

Module 5Post-test

4. Which of the following is considered formative assessment?

a. Year-end reviews of performance

b. Pass/fail examinations

c. Specific feedback of behavior in real time

d. Anonymous 360 degree evaluations sent to the resident

e. Periodic summative evaluations

Module 5Post-test

4. Which of the following is considered formative assessment?

c. Specific feedback of behavior in real time

Annual performance reviews, 360 degree evaluations and pass-fail examinations are all types of summative assessment.