Using Script Concordance for Clinical Reasoning

Assessment Rebecca Maldonado, MS, PA-C

Associate Professor, University of Colorado Physician Assistant

Program

Kristine Himmerick, MS, PA-C

Assistant Professor, Northern Arizona University Physician

Assistant Program

Session Objectives 1. Describe the theory of script concordance testing

(SCT) as a measure of clinical reasoning.

2. Understand the reliability of script concordance

testing.

3. Recognize the parameters for constructing an

expert clinician panel.

4. Comprehend the development techniques for a

script concordance test item.

5. Understand the use of the SCT calculator for

performing item analyses for both the expert panel

responses and learner responses to SCT items.

What is Clinical Reasoning?

• “the thinking and decision-making processes associated

with professional practice”

o-Higgs

Hierarchy of Clinical Reasoning

• Knowledge o Didactic knowledge

o Clinical skills

• Cognition o Application of knowledge to practice

• Metacognition o Thinking about the decision making process

o Thinking about thinking (Flavell, J. H. 1979)

Awareness of the learning process

Recognition of the limitations

of memory

Appreciation of perspective

Capacity for self-critique

Ability to select appropriate strategies for different

situations

(Croskerry, 2003)

Metacognition

• Foundational knowledge and experience are

required to progress o Novice - Advanced beginner – Competence – Proficiency – Expert

• Beginners are taught rules with little context.

• As the learner gains experience in applying the

knowledge to real world situations, he/she

advances toward expertise. (Benner, 1982)

Novice to Expert Continuum

Novice to Expert Thinking

Novice Monitoring, Reflection

Developing

Experienced

More

Less

Low High

Reliability, Accuracy

Cost, Time,

Effort

(Graber 2009)

Expert

How can medical educators evaluate clinical reasoning?

Can the novice to expert continuum be objectively

measured?

Scripts are… • Medical knowledge structures which link

relevant clinical and pathophysiologic information

• Scripts frame an individual’s expectations (hypotheses) in clinical scenarios

• Initiated during early medical training AND developed through clinical exposure and practice.

Scripts Concordance Testing is… … a written assessment of

clinical reasoning under

conditions of uncertainty

…clinical data interpretation

…quality of acquired scripts

Specific facet of

clinical reasoning

competence:

Hinges on:

A proxy

indicator of: …EXPERTISE

‘Gold Standard’:

Acknowledged experts in field

Script Concordance Test (SCT)

Multiple Choice Questionnaire (MCQ)

Ill-defined problem with context of uncertainty

Well-defined problem with no uncertainty

More than one

acceptable answer

Only one correct answer

Assess Reasoning

Assess Knowledge

Clinical Reasoning: Key Steps

SCT Question Format

Patient history and nonverbal

communication Script

Activation

Case Vignette

Hypothesis Generation

"If you were thinking…"

Data Collection Data Interpretation

"And then you find…"

Hypothesis Evaluation

"Your hypothesis becomes more or less

likely?" (-2, -1, 0, +1, +2)

SCT Principles 1. Examinees are faced with challenging,

realistic clinical situations and must choose between several appropriate response options

2. Item format reflects the way information is processed in clinical problem-solving situations

3. Scoring takes into account the variability of responses of experts to clinical situations

SCT Reliability • Alpha-coefficients of 0.70-.090

across multiple disciplines

(Lubarsky, 2011)

Creating SCT Items

• Create a small case vignette

• Must contain ambiguity or be missing information

• Appropriate for examinee’s training

Example:

Matt is a 26-year old male who presents to the ED

with a history of left-sided chest pain and shortness of

breath.

Creating SCT Items

• Select hypotheses for testing • Must be plausible

• Represent common or life-threatening possibilities

Examples:

• Pneumonia

• Pulmonary embolism

• Acute coronary syndrome

• Spontaneous pneumothorax

• Rib fracture/chest trauma

• Anxiety/panic attack

Creating SCT Items • Write the SCT item

• Test the link of new information into the context of the clinical

vignette

• Introduce key features which may have a broad range of

possible responses among clinicians

• Write items which assess reasoning within the case vignette

across the spectrum of the response scale

Example:

If you were thinking: And then you find: This diagnosis becomes:

Pneumonia The patient is afebrile -2 -1 0 1 2

Spontaneous Pneumothorax

The onset of chest pain/SOB was while running

-2 -1 0 1 2

Acute coronary syndrome

The patient uses cocaine -2 -1 0 1 2

-2: Very unlikely , -1: Somewhat unlikely, 0: Neither more or less likely, 1: Somewhat likely, 2: Very likely

Creating Investigative/Therapeutic SCT Items

• Write the SCT item

• Test the link of new information into the context of the clinical

vignette

• Introduce key features which may have a broad range of

possible responses among clinicians

• Write items which assess reasoning within the case vignette

across the spectrum of the response scale

Example:

If you were thinking of: And then you find: This treatment becomes:

Ordering a chest x-ray The patient is not tachypneic -2 -1 0 1 2

Ordering troponin levels

The patient’s EKG is normal -2 -1 0 1 2

Ordering a D -dimer The patient’s Well’s criteria score was <2

-2 -1 0 1 2

-2: Contraindicated , -1: Less indicated 0: Neither more or less indicated, 1: Somewhat indicated, 2: Very indicated

Creating SCT Items: More Cases or More Questions?

Gagnon et al., 2008

Vetting the SCT items: 1. Construct an Expert Panel

A. Utilize faculty or community providers with

experience in item content areas(Petrucci, 2013)

B. Expert panel size 10-15 members (Gagnon, 2005)

0,40

0,45

0,50

0,55

0,60

0,65

0,70

0,75

0,80

n=5 n=10 n=15 n=20 n=25 n=30 n=38

mean (sd) min value max value

Vetting the SCT items: 2. Scoring Expert Panel Responses

A. Discard or re-write items identified by panel as confusing,

erroneous, etc.

B. Use aggregate responses, don’t look for consensus (Norman,1985)

C. Score the responses using modal distribution (Charlin, 2010)

D. In panels <15, remove “outlier” responses (Gagnon, 2011)

Scoring Key for 1 SCT item (panel size= 10)

Answers -2 -1 0 1 2 (1) (2) (3) (4) (5)

Number of panel members Who chose this answer 0 0 2 5 3

Number of panel members Who chose this answer divided By the modal answer 0/5 0/5 2/5 5/5 3/5

Score for each response On this item 0 0 .4 1 .6

Vetting the SCT items: Choosing Moderate Variability Items

Use SCT calculator to determine variability of panel

responses for each item.

Staff-students

0

0,5

1

1,5

Low variability Moderate variability High variability

Charlin, 2006

SCT Calculator: Expert Panel Response Entry

1. Must be entered using 5-point Likert numbering 2. Once expert panel responses are entered, click to the 3rd worksheet tab to see item statistics

SCT Calculator: Expert Panel Item Stats

http://www.cpass.umontreal.ca/sct.html Rebecca.maldonado@ucdenver.edu for Excel file

A.Very likely

B. Somewhat more

likely

C.Neither more or

less likely

D.Somewhat

unlikely

E. Very unlikely

SCT Item #1 - Audience Response:

If you were thinking: And then you find:

PleuritisChest pain began

with Exercise

A)

Very

likely

B)

Somewhat

more

likely

C)

Neither

more or

less likely

D)

Somewhat

unlikely

E)

Very

unlikely

This diagnosis becomes:

A. B. C. D. E.

20% 20% 20%20%20%

Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath.

A.Very likely

B. Somewhat more

likely

C.Neither more or

less likely

D.Somewhat

unlikely

E. Very unlikely

SCT Item #2 - Audience Response:

A. B. C. D. E.

20% 20% 20%20%20%

Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath.

If you were thinking: And then you find:

Chest trauma / pain /

strain

Chest pain was

improved by lying

down

A)

Very

likely

B)

Somewhat

more

likely

C)

Neither

more or

less likely

D)

Somewhat

unlikely

E)

Very

unlikely

This diagnosis becomes:

A.Very likely

B. Somewhat more

likely

C.Neither more or

less likely

D.Somewhat

unlikely

E. Very unlikely

SCT Item #3 - Audience Response:

A. B. C. D. E.

20% 20% 20%20%20%

Matt is a 26-year old male who presents to the ED with a history of left-sided chest pain and shortness of breath.

If you were thinking: And then you find:

Pneumothorax

Patient exam

revealed a mid-line

trachea

A)

Very

likely

B)

Somewhat

more

likely

C)

Neither

more or

less likely

D)

Somewhat

unlikely

E)

Very

unlikely

This diagnosis becomes:

Expert Panel Responses: SCT Items 1-3

Administering the SCT Items

1. Options for student exams

2. SCT calculator for student scoring & item

analysis

SCT Calculator- Scores &Item

Analysis of Learner Responses

Avoiding Pitfalls: Use this “Do” list

1. Write short clinical vignettes

2. Write questions for which the answer is not clearly

defined in books or clinical guidelines.

3. Keep the number of questions per case to < 5.

4. Solicit more expert panel members than you need by

at least 30%.

Questions

Many thanks to Kevin Bogart, MS, PA-C for his contributions to this presentation.

References Benner, P. (1982). “From novice to expert”. American Journal of Nursing, 82(3), 402-407.

Charlin, B. (2010). “Assessment in the context of uncertainty using the Script Concordance test: more meaning for scores”. Teaching and Learning in Medicine, 22(30), 180-186.

Charlin B, Boshuizen H, Custer E, Feltovich P. (2007 )”Scripts and clinical reasoning”. Medical Education. 41:1178-84.

Charlin B, Gagnon R, Pelletier J, et al. (2006 )”Assessment in the context of uncertainty: the effect of variability within the panel of reference”. Medical Education.18: 22-7.

Croskerry, P. (2003). "Cognitive forcing strategies in clinical decisionmaking." Ann Emerg Med 41(1): 110-120.

Flavell, J.H. (1979). “Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry”. American Psychologist, 34(10), 906-911.

Fournier J, Demeester A, Charlin B. (2008 )”Script Concordance Tests: Guidelines for construction”. BioMedCentral, Medical Informatics and Decision Making. 8: 18.

Gagnon, R, et al. (2008) “Script Concordance testing: more cases or more questions”. Adv. Health Sci Educ. 14: 367-375.

Gagnon R, Charlin B, Coletti M, Sauve E, van der Vleuten C. (2005) “Assessment in the context of uncertainty: How many members are needed on the panel of reference of a script concordance test”. Medical Education. 39: 284-91.

Graber, M. L. (2009). "Educational strategies to reduce diagnostic error: can you teach this stuff?" Adv Health Sci Educ Theory Pract 14 Suppl 1: 63-69.

Higgs, J. and M. A. Jones (2000). Clinical reasoning in the health professions. Oxford ; Boston, Butterworth-Heinemann.

Lubarsky S, Charlin B, Cook D, Chalk C, van der Vleuten C. (2011 ) “Script concordance testing: a review of published validity evidence”. Medical Education. 45:329-338.

Norman, G. (2005). Research in clinical reasoning: past history and current trends. Med Educ, 39(4), 418-427.

Petrucci, AM, et al. (2013) “Assessing clinical judgment using the Script Concordance test: the importance of using specialty-specific experts to develop the scoring key”. Am J Surg. 205(2): 137-140.