Stability of Job Analysis Findings and Test Plans over
Time
Calvin C. Hoffman, PhD
Los Angeles County Sheriff’s Department
Presented to PTCSC on April 14th, 2010
Coauthors: Carlos Valle, Gabrielle Orozco-Atienza, & Chy Tashima.
INTRODUCTION
• Job analysis (JA) provides foundation for many human resources activities [recruitment, placement, training, compensation, classification, and selection (Gatewood & Feild, 2001)].
• In content validation research, JA is used to minimize the “inferential leaps” in selecting or developing selection instruments.
INTRODUCTION
• There is little guidance on how often to re-validate or revisit the validity of selection systems (Bobko, Roth, & Buster, 2005).
– Uniform Guidelines (1978) - “There are no absolutes in the area of determining the currency of a validity study.”
– SIOP Principles – “…organizations should develop policies requiring periodic review of validity of selection materials and methods.”
INTRODUCTION
• Our position is that if “revalidation” is needed, researchers must pay attention to job analysis.
• For example:– Changes in duties?– Changes in technology?– HR systems changes?– Changes in required KSAs?
SETTING
• Extensive litigation regarding sergeant promotional process.
• Consent decree governed all aspects of JA, selection system design, selection system operation, and actual promotions for over 25 years.
• Organizational policy requires updating job analysis every five years.
– Policy does not consider important factors such as costs and legal context.
SETTING• 2006 Sergeant Exam - Conducted extensive multi-
method JA.
• 2009 Sergeant Exam – Unsure about need for additional JA given recency of JA data.
• SIOP Principles - “The level of detail required of an analysis of work is directly related to its intended use and the availability of information about the work. A less detailed analysis may be sufficient when there is already information descriptive of the work” (p.11).
CURRENT STUDY
• Few changes in sergeant job were expected during the three-year span.– Could conclude that new JA is not needed,
and reuse the existing 2006 test plan. – Given history of litigation surrounding this
exam, the Principles would support conducting an additional JA.
• Choosing to err on the side of caution, we performed a slightly abbreviated JA to support 2009 exam.
CURRENT STUDY
• Study examined the stability of the JA data over a two-year span. Focuses on the similarity of:– task and KSA ratings by two independent
groups of incumbents– the test plans for the written job knowledge
test.
METHOD - 2006 JA
• Structured JA interviews were conducted on-site with incumbents, along with job observation, facility tours, and “desk observation”.
• From these data sources, a work-oriented job analysis questionnaire (JAQ) was drafted consisting of major tasks and KSAs.
• JAQ survey (incumbents), SME linkage ratings.
METHOD - 2009 JA
• Did not conduct additional JA interviews.
• Relied on the existing 2006 JAQ as a starting point for the 2009 JA effort.
• Otherwise, followed same process.
METHOD
Invited
ParticipatedResponse
RateSampling Method
2006 Exam
91 69* 76%Incumbents (Sergeants) chosen by Personnel Captain.
2009 Exam
65 49* 75%
Incumbents
(Stratified random sample) chosen by researchers.
*Both JAQs were administered online via a web survey
Participants
METHOD
• SMEs (2006 N = 13, 2009 N = 10) in both studies performed linkage ratings to establish relationship between task domains and KSA domains using a 4-point relevance scale.
– JAQ x linkage ratings data were further reviewed and fine-tuned by SMEs.
• JAQ data helped determine relative weight and content of test plans (written test, appraisal of promotability, and structured interview).
RESULTS - TASKS
Correlation
Mean Task Importance
Rating t-test
2006 Test
r =.83
3.7Dependant t (28) = 7.65; p < .001 two-tailed; d = 1.74
2009 Test
3.4Independentt (56) = 3.23; p < .01 two-tailed; d = .85
RESULTS - KSAs
Correlation
Mean KSA Importance
Rating t-test
2006 Exam
r =.96
3.8
Dependant
t (29) = 3.94; p < .001 two-tailed;
d = 1.54
2009 Exam
3.7Independentt (56) = 0.90; p > .05 two-tailed; d = .23
RESULTS – TEST PLAN
Total Items
Knowledge Domains
From JA
Domains Omitted
from Test Plan
Recall v. Reference
CorrelationM KSA
Importance Rating
2006 Test
102 31 6
84%Agreement
r =.85
3.8
2009 Test
104 30 3* 3.7
*Of six knowledge domains omitted in 2006, three knowledge domains were retained in 2009, for a total of 13 items. All were included as Reference items in 2009.
DISCUSSION• JA data were highly stable over time, despite
significant M differences observed.
• Mean task importance ratings (r = .83)
– About 1.0 standard deviation larger than meta-analytic findings of intrarater reliability (rate-rerate) of JA ratings data reported by Dierdorff and Wilson (2003), r = .68 (n = 7,392; k = 49) over an average of 6 months.
• Mean KSA importance ratings (r = .96)
– No comparison could be made because NO estimate of KSA stability over time could be located in literature.
• Test plans (r =.85)
– Number of items allocated to specific knowledge domains was highly similar.
DISCUSSION
• Differences between mean task ratings might be attributable to:
– differences in the selection of JAQ respondents
– decreased sensitivity in organization regarding sergeant promotional exam (i.e., no new lawsuits!).
DISCUSSION
• Findings did not translate into major differences in the test plans resulting from the JA efforts even with:
– different SMEs,
– different survey respondent selection methods,
– significant differences in mean task and KSA ratings.
DISCUSSION
• Although five new domains were included in the 2009 test, they were incorporated as Reference questions wherein candidates are provided resource material to answer questions.
CONCLUSION
• We considered costs and risks in determining whether to revalidate our selection system.
– The greater the number of intervening years between validation studies, the higher the risk the organization assumes.
– The shorter the intervals of time between revalidation efforts, the costlier and more burdensome they are for the organization.
• We were conservative due to the legal context. Might have followed a different path if guidelines regarding revalidation efforts were clearer.
CONCLUSION
• We encourage researchers and practitioners to conduct and share any research findings that might help in creating detailed practice guidelines on revalidation efforts.
• More information is needed to close the disconnect between the requirement to maintain currency of validity information and the lack of clear guidance regarding how often is “often enough.”
Questions?
• Copies of slides and the conference paper are available:– Email request to: [email protected]
Top Related