The Jimmy A Young Memorial Lecture
description
Transcript of The Jimmy A Young Memorial Lecture
![Page 1: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/1.jpg)
The Jimmy A Young Memorial Lecture
July 17, 20147:00 to 8:30 AMMarco Island, FL
1
![Page 2: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/2.jpg)
The NBRC has honored Jimmy’s memory and the contributions he made to respiratory care through this program since 1978 .
Jimmy Albert Young, MS, RRT1935 –1975
2
![Page 3: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/3.jpg)
Jimmy Albert Young, MS, RRT was one of the profession’s most outstanding and dedicated leaders
– 1935 – born in South Carolina– 1960 – 1966 – served as Chief Inhalation
Therapist at the Peter Bent Brigham Hospital in Boston
– 1965 – earned the RRT credential, Registry #263
– 1966 – 1970 – served in several roles including director of the education program at Northeastern University in Boston
– 1970 – became director of the Respiratory Therapy Department at Massachusetts General Hospital
– 1973 – became the 22nd President of the American Association of Respiratory Care
– 1975 – was serving as an NBRC Trustee and member of the Executive Committee when he passed away unexpectedly
In a 15-year career,-achieved the RRT-directed an education program-directed a hospital department-served as AARC President-served as an NBRC trustee
3
![Page 4: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/4.jpg)
The Clinical Simulation Examination
Then and Now
4
![Page 5: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/5.jpg)
Presenter• Robert C Shaw Jr PhD RRT FAARC
– NBRC Assistant Executive Director and Psychometrician
5
![Page 6: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/6.jpg)
Conflict of InterestI have no real or perceived conflicts of interest that relate to this presentation. Any use of brand names is not meant to endorse a specific product, but to merely illustrate a point of emphasis.
![Page 7: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/7.jpg)
Learning Objectives
• Compare elements of the current RRT credentialing system to elements of the system that is planned for January 2015
• Compare the value of information that has been provided by results from the Clinical Simulation Examination to other elements of the RRT credentialing system
• Describe features of the 20-problem Clinical Simulation Examination for which candidates should be prepared by January 2015
![Page 8: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/8.jpg)
OBJECTIVE 1
Compare elements of the current RRT credentialing system to elements of the system that is planned for January 2015
![Page 9: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/9.jpg)
Compare Current to Future
RRT Program Elements
Number
CurrentJanuary
2015Hours of testing time 9 7Examinations 3 2Testing sessions for a candidate who passes on first attempts 3 2Sets of test scores 4 2Passing points 4 3Multiple-choice items to assess competencies broadly 240 140Patient management problems to assess competencies deeply 10 20Typical number of sections in a patient management problem 10 (8-12) 5 (4-6)
![Page 10: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/10.jpg)
OBJECTIVE 2
Compare the value of information that has been provided by results from the Clinical Simulation Examination to other elements of the RRT credentialing system
![Page 11: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/11.jpg)
QUESTION
Is there a measurement reason for the Clinical Simulation Examination to exist?
![Page 12: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/12.jpg)
RESEARCH HYPOTHESIS 1
Scores from the Clinical Simulation Examination added information beyond the information from multiple-choice examination scores when predicting membership in three groups for candidates who sought the RRT credential.
![Page 13: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/13.jpg)
What defined the three groups of candidates?
Credential Status Label for group
Examination Outcome
CRT Written Registry Clinical Simulation
CRTcertification
pass
fail fail
certification +1 pass fail
fail pass
RRT registration pass pass
![Page 14: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/14.jpg)
Defining the Population
• Date range for examination attempts– October 22, 2009 through February 27, 2012
• A subset of 9,081 candidates had achieved CRT and made a first attempt at the remaining examinations for RRT (and were not outlying cases)– Written Registry– Clinical Simulation
• Information gathering (IG)• Decision making (DM)
![Page 15: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/15.jpg)
Statistical Model and Method
• Step-wise discriminant analysis with automatic variable selection– Predict group membership from multiple variables,
each of which is continuously distributed– Dependent variable
• certification, certification+1, and registration groups– Independent variables
• First run included four sets of scores– CRT, Written Registry, Clin Sim IG, and Clin Sim DM
• Second run included two scores – CRT and Written Registry
![Page 16: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/16.jpg)
Standardizing Examination Scores
• Raw score ranges– CRT = 0 to 140– Written Registry = 0 to 100– Clinical Simulation, varied by test form
• IG = a variable-min to a max in the range of 200-300• DM = a variable-min to a max in the range of 140-170
• Each raw score was converted to a z-score where z = (x – mean) / S
![Page 17: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/17.jpg)
Results from Run 1
Step Test ScoresWilk’s Lambda -
Proportion of unexplained variance
F Test for Entry
Value p
1 Clinical Simulation DM .421 6251.4 <.0001
2 Written Registry .296 3803.5 <.0001
3 Clinical Simulation IG .295 2540.7 <.0001
4 CRT .295 1907.8 <.0001
Predictions about memberships in the registration group were accurate for 92.4% of the cases
![Page 18: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/18.jpg)
Discriminant Score Equation
• Discriminant score =1.026 (Clin Sim DM z-score)+ 0.975 (Written Registry z-score)+ 0.091 (CRT z-score)- 0.010 (Clin Sim IG z-score)- 0.689
• Clin Sim DM and Written Registry scores were nearly equal and the dominant contributors to predictions about group memberships
![Page 19: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/19.jpg)
Results from Run 2
Predictions about memberships in the registration group were accurate for 85.4% of the cases
Step
Test Scores
Wilk’s Lambda -Proportion of unexplained
variance
F Test for Entry
Value p
1 Written Registry .447 5607.3 <.0001
2 CRT .433 2361.6 <.0001
.42
.30
92.4%
![Page 20: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/20.jpg)
Conclusions• The research hypothesis was accepted
– Scores from the Clinical Simulation Examination add information about RRT achievement beyond what is available from multiple-choice examination scores
• If the Clinical Simulation Examination was removed from the system, there would be a 7% loss of accurate RRT classifications– Incompetent candidates would become RRT– Competent candidates would be denied RRT
![Page 21: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/21.jpg)
RESEARCH HYPOTHESIS 2
Although there were four sets of test scores, three tests, and two types of tests, RRT competencies were based on only one cognitive construct.
![Page 22: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/22.jpg)
Examination Type CharacteristicsCharacteristic Multiple-Choice Clinical Simulation
Option-response scoring dichotomous (0 or 1) polytomous (-3 to 3)
Linkages between stimulus-response elements
independent items independent problems,dependent sections
Potential for branching units to which a subset of candidates are directed
no yes
Cut point determination method
external to test development
Integrated with test development
Cost to produce $ $$
![Page 23: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/23.jpg)
Risks from Using Multiple Examinations with Different Characteristics
IGDM
CRTWR
Simulation Multiple-Choice
Type of Examination
CRTIG
DM
WR
Advanced Entry
Level of Examination
![Page 24: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/24.jpg)
Statistical Model and Method
• Principal components analysis with cross-validation– Explore the underlying variance structure within four
sets of test scores• CRT• Written Registry• Clinical Simulation
– IG– DM
– Is useful for confirming a hypothesis, in this case the assertion that there is a common characteristic expressed by the four test scores
![Page 25: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/25.jpg)
Preliminary Result 1As an indicator of sampling adequacy
-KMO should be at least .50
-Sig value should indicate statistical significance
As indicators of positive cross-validation
-KMO values should be about the same
Sampleswhole random
split 1random
split 2sample size 9,081 4,557 4,224
Kaiser-Meyer-Olkin Measure of
Sampling Adequacy.775 .777 .772
Chi-Square 18259.56 9358.25 8913.45
Bartlett's Test of
Sphericity
df 6 6 6
Sig. .000 .000 .000
![Page 26: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/26.jpg)
Preliminary Result 2As indicators of making a sufficient contribution to the principal component solution
-Communality values should be at least .50, otherwise a variable should be removed
As indicators of positive cross-validation
-Values across each row should be similar
Communality Values Extraction Samples
Scores whole random split 1 random split 2 CRT .768 .765 .773
Written Registry .765 .771 .759
Clin Sim IG .589 .595 .583
Clin Sim DM .701 .713 .690
![Page 27: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/27.jpg)
Primary ResultThe threshold for a consequential eigenvalue is 1.0
or
Components at the inflection point and beyond lack consequence
![Page 28: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/28.jpg)
Conclusions-The research hypothesis was accepted
There was only one principal component to which all four sets of test scores were linked
-Potential risks associated with using a multiple-examination system were avoided
![Page 29: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/29.jpg)
Summary from Both Studies
• Within the population of new RRTs each year, accurate classifications occur more often because there are multiple examinations
• Risks associated with a credentialing system based on multiple examinations were avoided
![Page 30: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/30.jpg)
Study Limitations
• These were population studies involving a recent period of more than 2 years
• Unless characteristics of candidates or examinations change, I expect these results will generalize into the future– Candidates: program admission criteria, program
duration, program intensity – Examinations: number of instruments, types of
measurements
![Page 31: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/31.jpg)
OBJECTIVE 3
Describe features of the 20-problem Clinical Simulation Examination for which candidates should be prepared by January 2015
![Page 32: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/32.jpg)
Rationale for Changing the Simulation Examination
• Instant scoring demands selection of problems for each new test form that have not changed– After a decade, keeping examination content
current became an increasing challenge
32
![Page 33: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/33.jpg)
Solution• Give the examination committee smaller
content elements from which test forms are assembled– Halve the number of sections in problems– Double the number of problems
• Hold testing time the same at 4 hours
33
![Page 34: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/34.jpg)
ENHANCE PSYCHOMETRIC PROPERTIES
As long as other changes will be made . . .
34
![Page 35: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/35.jpg)
Standardize Test Forms More Thoroughly
35
Type of ProblemSpecifications
Current 10-Problem Future 20-ProblemA1. COPD Conservative Care 1 or 2 2A2. COPD Critical Care 1 or 2 2B. Trauma 1 or 2 3C. Cardiovascular 1 or 2 3D. Neurological / Neurosurgical 1 or 2 2E. Pediatric 1 2F. Neonatal 1 2G. General Medical / Surgical optional 4
![Page 36: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/36.jpg)
Problems Each Candidate Will See• 4 about COPD• 4 about children• 4 about general medical / surgical• 3 about trauma• 3 about cardiovascular• 2 about neuro
– Likely one neuromuscular and one neurologic
![Page 37: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/37.jpg)
Simulation Examination Scores
![Page 38: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/38.jpg)
Advantages of a one score and one cut system
• A test with more items and more points than its predecessors will yield more accurate scores as indicators of candidates’ abilities– Pass and fail decisions become more accurate
• Accuracy is gained without an increase in test administration time– Fee for the Clinical Simulation Examination stays
the same
![Page 39: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/39.jpg)
A Potential Disadvantage of a Combined Score
• Compensation can occur unless the cut score policy is changed– Someone within a few points of passing based on
decision making performance could pass by acquiring a higher percentage of available information gathering points
![Page 40: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/40.jpg)
New Cut Score PolicySection Type
Cut Score Range
Current New
DM 60% to 70% 60% to 70%
IG 60% to 70% 77% to 81%
The cut score for a test form must be the sum of MPLs from the two types of sections such that those section MPLs fall within the two ranges shown in the table
Implementation has mandated addition of options labeled as required among positively-scored options in IG sections
![Page 41: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/41.jpg)
Conforming to the PolicyOne IG Section
Option Current January 2015
1 -2 -2
2 1 1R
3 2R 2R
4 -1 -1
5 2R 2R
6 -1 -1
7 -2 -2
8 1 1
9 2R 2R
10 -2 -2
11 1 1
12 -1 -1
MPL, Max, % 6, 9, 67% 7, 9, 78%
![Page 42: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/42.jpg)
WHY THE CUT SCORE POLICY CHANGE MATTERS
Illustrations that follow came from one test form
![Page 43: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/43.jpg)
DM % Score
9080706050403020100
DM Score DistributionFr
eque
ncy
200
150
100
50
0
Std. Dev = 11.42 Mean = 60N = 2331.00
MPL range remains 60%-70%
![Page 44: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/44.jpg)
IG % Score
9080706050403020100
IG Score DistributionFr
eque
ncy
400
350
300
250
200
150
100
50
0
Std. Dev = 5.76 Mean = 80N = 2331.00
MPL range has been 60%-70%
![Page 45: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/45.jpg)
SCATTERPLOT OF IG & DM SCORES
Ref Lines @ IG & DM MPLs
IG SCORES
240220200180160140120100806040200
DM
SC
OR
ES
130120
110
100
90
80
70
60
50
40
30
20
100
PassNo case in this quadrant
![Page 46: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/46.jpg)
SCATTERPLOT OF IG & DM SCORES
Ref Lines @ -0.05 Z MPLs
IG SCORES
240220200180160140120100806040200
DM
SC
OR
ES
130120
110
100
90
80
70
60
50
40
30
20
100
People in this quadrant would pass under the current system
Pass
![Page 47: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/47.jpg)
Highlights for Students• The numbers of problems by patient type will be
constant for each candidate• Testing time remains 4 hours
– 22 problems will be presented– Results will be based on responses to 20 problems
• As a result of a problem-splitting procedure– Some problems will not offer IG sections– Candidates will see the same number of IG sections
across the whole examination as they currently see
![Page 48: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/48.jpg)
Highlights for Students (cont.)• Responses will be summed across IG and DM
sections that a candidate enters to produce one score to which a cut score will be compared– The cut will equal the sum of MPL values across
sections along the critical path• Compared to the current examination,
responses in IG sections will be consequential– Reduced tolerance for errors
![Page 49: The Jimmy A Young Memorial Lecture](https://reader036.fdocuments.net/reader036/viewer/2022062501/56815f2d550346895dcdf8d8/html5/thumbnails/49.jpg)
QUESTIONS