Wade Vagias - Identifying and Testing Predictors of Compliance with Recommended LNT Practices
Identifying and Developing Predictors of Job...
Transcript of Identifying and Developing Predictors of Job...
![Page 1: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/1.jpg)
1
Identifying and Developing Predictors of Job Performance
Patrick J. CurtinDeborah L. WhetzelKenneth E. GrahamCaliber Associates
Pre-Conference Workshop, IPMAACJune 22, 2003
![Page 2: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/2.jpg)
2
Agenda
• Introductions• Test Plan• Cognitive Ability• Training and Experience• Biodata• Interviews
![Page 3: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/3.jpg)
3
Introduction
• Who are we?• Who are participants?
![Page 4: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/4.jpg)
4
Test Plan – Overview
• Summarizes information from job analysis and literature review of assessment methods
• Provides a rationale for the use of particular test and assessment methods
• Provides a blueprint for the development and validation of tests and assessments
![Page 5: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/5.jpg)
5
Test Plan - Process• Steps
• Conduct job analysis to identify KSAs• Identify potential measurement
methods• Assess selection methods against
psychometric and operational criteria• Create a KSA by method matrix
• Determine how best to assess relevant KSAs
![Page 6: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/6.jpg)
6
Test Plan –Centerpiece is the Matrix
• The matrix is used to link KSAs to measurement methods• KSAs are rows• Measurement methods are columns• Rater judgments are in the cells
![Page 7: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/7.jpg)
7
Test Plan – Matrix Example Selection Method KSA
Cognitive Ability
Training and Experience Biodata Interviews
Knowledge of business management (such as quality control, human resource management)
Knowledge of math including addition, subtraction, multiplication, and division
Skill in dealing with all types of people
Skill in managing own time and coordinating with others
Ability to have an impact on others in the organization and display energy and leadership
Ability to communicate information and ideas orally so others will understand
![Page 8: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/8.jpg)
8
Test Plan – Identifying KSAs• Knowledge - A body of information
applied directly to the performance of a function.
• Skills - A present, observable competence to perform a learned psychomotor skill.
• Abilities - A present competence to perform an observable behavior or a behavior which results in an observable product.
![Page 9: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/9.jpg)
9
Example KSAs• Example Managerial Knowledges
• Knowledge of business management (such as quality control, human resource management)
• Knowledge of math including addition, subtraction, multiplication, and division
• Example Managerial Skills• Skill in dealing with all types of people• Skill in managing own time and coordinating with
others• Example Managerial Abilities
• Ability to have an impact on others in the organization and display energy and leadership
• Ability to communicate information and ideas in speaking so others will understand
![Page 10: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/10.jpg)
10
Test Plan – Identifying measurement methods
• Example measurement methods:
•Cognitive ability•Training and experience•Biodata•Interviews
![Page 11: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/11.jpg)
11
Test Plan
• Methods need to be assessed against Psychometric & Operational criteria
• Psychometric criteria• Reliability evidence• Validity evidence• Content validation support• Subgroup differences
![Page 12: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/12.jpg)
12
Test Plan
• Operational criteria• Applicant acceptance• Resistance to compromise• Consistency of administration and scoring• Development cost• Development time• Operational cost
![Page 13: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/13.jpg)
13
Test Plan – Selection Tools by Criteria Example
E VALUATIO N OF SELECTIO N T O O LS A GAINST C RITERIA Selection Tools
M easures of Basic
AttributesM easures of Background, Training, and Experience
M easures of Declarative K now ledge and Procedural K now ledge and Skill
Criteria Cog
nitiv
e A
bilit
y
Pers
onal
ity
Acc
ompl
ishm
ent
Rec
ord/
Po
rtfo
lio
Inte
rvie
w
Bio
data
GPA
Tas
k-ba
sed
Que
stio
nnai
re
Job
Kno
wle
dge
SJT
Wor
k Sa
mpl
e/
Job
Try
out
Rol
e Pl
ay
Lea
derl
ess
Gro
up
In-b
aske
t
Ess
ay
Validation Evidence H M M M M L M H H H L L M L
Content Validation Support M M H H M L H H H H H H H M
Reliability H H M M M H H H M H M M M M
Subgroup differences H L L L L H M M M L L L M H
Applicant acceptance M L M H L M M H H H H M M M
Resistance to compromise H L H M M H M H M H L M M M
Consistency of administration and scoring
H H M M H H H H H M L L M M
Development cost L L M M H L L M H H H M H M
Development tim e L L M M H L L M H H H M H M
O perational cost L L M H L L L L L H H H H M
![Page 14: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/14.jpg)
14
Test Plan Exercise: Completing the Matrix
Selection MethodKSA
Cognitive Ability
Training and Experience Biodata Interviews
Knowledge of business management (such as quality control, human resource management) Knowledge of math including addition, subtraction, multiplication, and division Skill in dealing with all types of people
Skill in managing own time and coordinating with others
Ability to have an impact on others in the organization and display energy and leadershipAbility to communicate information and ideas orally so others will understand
![Page 15: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/15.jpg)
15
Test Plan – Final Review
• Do the selection methods provide “coverage” of the KSAs?
• Are the Psychometric and Operational aspects of the chosen methods appropriate given the restrictions of a particular testing scenario?
• In what order should the methods be used?
![Page 16: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/16.jpg)
16
Test Plan - Compromises
• To balance among competing concerns such as adverse impact and operational costs, the following should be considered:• Multiple hurdle approach• Whole job measurement approach
![Page 17: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/17.jpg)
17
Test Plan – In Summary
• The final test plan• identifies suitable measurement methods
given a host of concerns and limitations• demonstrates the expected relationships
between KSAs and measurement methods.• provides a blueprint for the development
and validation of personnel selection procedures.
![Page 18: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/18.jpg)
18
Cognitive Ability
• Definition • Cognitive ability has been
conceptualized in more than one way. The two main conceptualizations are based on cognitive ability being thought of as a unified whole “g” or thought of as a collection of intelligence facets.
![Page 19: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/19.jpg)
19
Cognitive Ability• Examples of Off-the-shelf Cognitive
Ability Tests• The Wonderlic Personnel Test • Employee Aptitude Survey – a basic skills
battery• Watson-Glaser Critical Thinking Appraisal• Wechsler Adult Intelligence Scale III • Raven’s Progressive Matrices• Stanford-Binet Intelligence Scale• Woodcock Johnson III Tests
![Page 20: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/20.jpg)
20
Cognitive Ability
• Psychometric Characteristics• Reliabilities can range from the mid
.80s to the low .90s (Hunter & Hunter, 1984; Schmidt & Hunter, 1998).
![Page 21: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/21.jpg)
21
Cognitive Ability
• Psychometric Characteristics• Validity of cognitive ability tests to
predict performance is approximately.51 (Hunter & Hunter, 1984; Schmidt & Hunter, 1998).
![Page 22: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/22.jpg)
22
Cognitive Ability
• Psychometric Characteristics• Subgroup differences -- A difference
of approximately one standard deviation has been repeatedly found between African Americans and White subgroups on cognitive ability testing (Hunter and Hunter, 1984; Jensen, 1980; Schmitt, Clause, &Pulakos, 1996).
![Page 23: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/23.jpg)
23
Cognitive Ability• How to Select a Cognitive Ability Test
• Mental Measurements Yearbooks (University of Nebraska Press)
• Tests in Print (University of Nebraska Press)
• Test Critiques (Pro-Ed – Austin, TX) • Tests (Pro-Ed – Austin, TX) • A.P.A. (www.apa.org) • Test Publishers
![Page 24: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/24.jpg)
24
Cognitive Ability
• Issues to Consider• Administration time• Special accommodations• Retesting policy
![Page 25: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/25.jpg)
25
Cognitive Ability
• Reasons for developing a new cognitive ability test• Alternate forms are required• Tests become outdated• Client wants a proprietary test• Test for newly hypothesized or
specialized ability
![Page 26: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/26.jpg)
26
Cognitive Ability• Steps for developing a new cognitive
ability test• Develop test specifications• Create items• Conduct technical reviews• Conduct sensitivity reviews• Create test administration instructions• Conduct pilot test of items• Analyze item data• Select items and assemble test• Write documentation
![Page 27: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/27.jpg)
27
Training and Experience•Definition
•Training and experience (T&E) selection instruments are based on the notion that past behavior and experience are powerful predictors of future behavior (Ash, Johnson, Levine, & McDaniel; 1989).
![Page 28: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/28.jpg)
28
Training and Experience• Past behavior and experience
captured by T&E instruments include job-relevant tasks, job performance, training, and overall knowledge, skills, and abilities (KSAs) that an individual may have gained from previous jobs.
![Page 29: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/29.jpg)
29
Training and Experience• Holistic Judgment
• Not a formally scored T&E evaluation method but rather a more general overview of an individuals resume.
• Lacks both structure and objectivity.
![Page 30: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/30.jpg)
30
Training and Experience• Traditional Point Method
• Consists of a mechanical formula set out in a formal schedule.
• The number of months or years of different types of relevant training, education, and experience are allotted a certain number of points.
![Page 31: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/31.jpg)
31
Training and Experience• Improved Point Method
• Examines an applicants’ specific job behaviors/duties as indicators of job-related KSAOs.
• Less arbitrary and a more adequate approach to assigning fairly specific point scores to applicants based on their individual backgrounds.
![Page 32: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/32.jpg)
32
Training and Experience• Grouping Method
• Applicants are divided into a small number of groups on the basis of simultaneous consideration of training and experience.
• Rather than being ranked over a continuous range of scores, applicants assigned to each group are assigned to the same score.
![Page 33: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/33.jpg)
33
Training and Experience• Behavioral Consistency Method
• Has been referred to as the accomplishment record method (Hough, 1984 cited in Ash et al., 1989).
• Rank order applicants on the basis of the type of achievement behaviors that are necessary for optimum performance in the target job.
![Page 34: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/34.jpg)
34
Training and Experience• Self-rating T&E Rating Method
• Composed of three different methods: Task-based, KSAO-based, and The Illinois Job Element Method.
• Require applicants to evaluate and judge their own past tasks that they have performed or KSAOs that they have acquired.
![Page 35: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/35.jpg)
35
Training and Experience•Psychometric Characteristics
•Past research has found interraterreliability estimates of T&E ratings were found to be around .80, which is quite desirable (Gatewood and Field, 1988).
![Page 36: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/36.jpg)
36
Training and Experience•Psychometric Characteristics
•Meta-analyses yielded the following average estimated validities: behavioral consistency (.45), self-rating KSAO (.20), self-rating task (.15), and point method (.11).
•These results, with the exception of the point method, are based on few studies (McDaniel, Schmidt, and Hunter, 1988).
![Page 37: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/37.jpg)
37
Training and Experience•Psychometric Characteristics
•Because the method yields higher scores to those applicants with the largest amount of work experience and education; females, minorities, and young applicants may not have had the time or opportunity to accumulate as many experience and training points as white, male and older applicants.
![Page 38: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/38.jpg)
38
What is an Experience Questionnaire?
• Task-based questionnaire (TBQ) • Measures experience with tasks from job analysis• Assumes experience performing job tasks• Not usually appropriate for entry-level jobs• Minimal written communications skills required• Applicants indicate experience performing job tasks
![Page 39: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/39.jpg)
39
What is an Experience Questionnaire?
• KSA based questionnaire (KSABQ) • Measures experience as indicator of KSAs.• No need for experience with specific tasks• Can be used for entry-level jobs• Minimal written communications skills required• Applicants indicate experience performing activities
related to the job.
![Page 40: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/40.jpg)
40
Steps to Develop TBQ
• Identify tasks•Create counterfeit items (faking)
• Develop scoring system• Pilot test form
![Page 41: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/41.jpg)
41
Identify Tasks
• Include tasks that are:• Critical• Needed-at-entry
• Limit TBQ to tasks that qualified applicants could have experience performing.
![Page 42: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/42.jpg)
42
Develop Scoring System• Award points for increasing amounts of
experience performing tasks.• Number of times.
• Reflect the asymptotic relationship between experience and performance.• Estimate asymptotes based on job analysis
and MQ data.• SMEs review and adjust asymptotes.• Assign points linearly below asymptote.
![Page 43: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/43.jpg)
43
Example Scoring System
100% of performance asymptote or more580% of performance asymptote460% of performance asymptote340% of performance asymptote220% of performance asymptote10% of performance asymptote0
Level of ExperiencePoints
Points Associated with Levels of Experience
![Page 44: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/44.jpg)
44
Pilot Test TBQ
• Examine readability:• Flesch-Kincaid• Linguist
• SMEs participate in “think-aloud”workshop.
• SMEs try to identify counterfeit tasks.
![Page 45: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/45.jpg)
45
Challenges• Faking• Collecting accurate/reliable
information• Setting performance asymptote• Measuring education/training-based
experience
![Page 46: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/46.jpg)
46
Faking• Three levels of protection:
• Certification of information accuracy• Statement certifying veracity of responses.• TBQ lists penalties (e.g., dismissal) for falsification.
• Verifiers• References that can support accuracy of information.
• Counterfeit items• Nonsensical tasks that applicant could not perform.• Include multiple opportunities to “correct” responses.
![Page 47: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/47.jpg)
47
Collecting Accurate/Reliable Information
• Number of times is best predictor, but hard to estimate.
• Calculate from frequency and duration.• More verifiable.• Difficult when tasks/activities vary in frequency.
![Page 48: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/48.jpg)
48
Collecting Accurate/Reliable Information
12345
67895
3. Replacing the gears in a clock.
12345
67895
2. Replacing the hands on a clock.
12345
678951. Replacing the batteries in clocks.
MonthsYearsFrequency
Duration
Who can verify your work
experience with this activity?
How much work experience do you have performing this activity?Please rate your experience
with each of the activities below:
Skill in repairing clocks
![Page 49: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/49.jpg)
49
Collecting Accurate/Reliable Information
• Number of times is best predictor, but hard to estimate.
• Provide anchoring examples.• More flexible.• Less verifiable.• Applicants may overlook them.
![Page 50: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/50.jpg)
50
Collecting Accurate/Reliable Information
8 years or more
22 months or more
5 months or more
96 times or more
6 years18 months4 months77 times
5 years13 months3 months58 times
3 years9 months2 months38 times
2 years4 months1 month19 times 1 6
2 7
3 8
4 9
5
------0 times
1. Replacing the batteries in a clock.
or every month for about…
or every week for about…
every day for about…
Who can verify your work
experience with this activity?
(Mark all corresponding
verifier numbers that apply)
To perform this activity the number of times listed in the option to the left, you’d have to perform the task…
How many times have you performed this activity? (Mark the
option closest to your level of experience).
Verifier(s)ExamplesFrequency
Activity
Skill in Repairing Clocks
![Page 51: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/51.jpg)
51
Setting Performance Asymptote• Difficult to estimate asymptote:
•Some tasks may not have one.•Interpersonal KSAs.•Complex or rapidly changing jobs.
•Many different routes.•Experience.•Education/training.•Hybrid (e.g. apprenticeships).
•Does not account for individual differences.•Intelligence.•Openness to experience.
![Page 52: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/52.jpg)
52
Measuring Education/Training-Based Experience
• Difficult to equate education/training:• Different schools.• Different instructors.• Different course materials.• Person X situation interaction.
• Collect information on activities performed as part of education or training.
![Page 53: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/53.jpg)
53
Training and Experience•Strengths and Weaknesses
• The holistic judgment and traditional point methods, which are the two most used methods tend to result in decisions that lack validity.
• Behavioral consistency, improved point method, and grouping methods will be effective once organizations decide to use these in place of the holistic judgment and traditional point methods. Cost more to develop and implement than the other procedures due to increased job analysis data requirements.
![Page 54: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/54.jpg)
54
Biodata•Definition
• Biodata consists of background information from various life history domains, including education, work experiences and accomplishments, developmental experiences, hobbies, and social interactions.
• Biodata is typically elicited through the use of questionnaire items, which ask people to recall and report typical behaviors and experiences that have occurred in the past.
• The focus of biodata is to assess a pattern of historical experiences that are manifestations of predictor constructs (e.g., self confidence).
![Page 55: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/55.jpg)
55
Biodata• Example item for quality and
quantity of workAbout how many hours in a day do you
spend in constructive work?a. Less than 8b. 8-10c. 11-13d. 14-16e. More than 16
![Page 56: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/56.jpg)
56
Biodata• Example item for quality and
quantity of workHow often do you do a task over and over
again until it is perfect?a. alwaysb. oftenc. sometimesd. almost nevere. never
![Page 57: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/57.jpg)
57
Biodata• Example item for problem
solvingAbout how many non-fiction books have
you read in the past year?a. noneb. onec. twod. threee. four or more
![Page 58: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/58.jpg)
58
Biodata• Example item for problem
solvingHow often have you invented something to
serve a needed purpose?a. neverb. one or two timesc. three to five timesd. six to ten timese. more than ten times
![Page 59: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/59.jpg)
59
Biodata• Scaling methods
• Empirical scaling procedures select and weight items based on their ability to differentiate membership in higher and lower-performing criterion subgroups (Hogan, 1994)
![Page 60: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/60.jpg)
60
Biodata• Scaling methods
• Rational scaling procedures involves developing items based on an individual difference variable (based on job analysis). Item selection is based on judgments of relevance of item to construct (Hough & Paullin, 1994).
![Page 61: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/61.jpg)
61
Biodata• Psychometric Characteristics
• Empirical scales—test-retest reliability .60-.96
• Rational scales—internal consistency low .70s; coefficient alpha .66-.78
![Page 62: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/62.jpg)
62
Biodata• Psychometric Characteristics
• Empirical scales—Meta-analysis .36 with Ability to perform; .34 with Performance of duties
• Rational scales—Initial validities in low .40s; shrinkage roughly .05-.10.
![Page 63: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/63.jpg)
63
Biodata• Psychometric Characteristics
• Low adverse impact• But may be negatively biased
due to range restriction (pre-selection)
• Correlation with cognitive ability .51
![Page 64: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/64.jpg)
64
Biodata• How to generate items
1. Identify the purpose of the instrument. 2. Precisely define the criteria of interest
(e.g., performance ratings, absenteeism, turnover).
3. Specify the variables to be measured by identifying job tasks/behaviors and knowledge, skills, abilities, motivation, and other characteristics necessary to effectively perform job behaviors.
![Page 65: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/65.jpg)
65
Biodata• How to generate biodata
items4. Develop items that reflect hypotheses
about the influence of prior experience, behaviors, and/or situations on the knowledge, skills, abilities and other important characteristics needed for successful job performance.
![Page 66: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/66.jpg)
66
Biodata• How to generate biodata
items5. A second panel of SME’s should screen
the items and eliminate those that are not objective, not relevant, or prone to socially desirable responding. Experienced item developers who are not familiar with the items should review them at this stage.
![Page 67: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/67.jpg)
67
Biodata• How to generate biodata
items6. The final instrument will require 8–15
items per construct, there should be a sufficient number of items at the pretest stage to evaluate.
7. Administer screened items to a try-out sample of 100 – 200.
![Page 68: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/68.jpg)
68
Biodata• How to generate biodata items
8. Conduct statistical screening of items using the following criteria 1. Items must show sufficient variability (standard
deviation of at least .80) 2. Item distributions must be consistent with the
hypotheses for which they were generated.3. A coefficient alpha of .50 or above is suitable for
a rationally constructed biodata scale. Support for the reliability of the scales may be demonstrated by lack of shrinkage across samples.
![Page 69: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/69.jpg)
69
Biodata• Faking controversy
• Correcting for social desirability does not improve criterion-related validity of personality scales (Ones, Viswesvaran & Reiss, 1996).
• Criterion-related validity decreases when people are told to fake (Graham, McDaniel, Douglas & Snell, 2002).
![Page 70: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/70.jpg)
70
Employment Interview• Definition
• Interviews involve the exchange of information between an interviewer and an interviewee, either face to face, by telephone, or in writing.
![Page 71: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/71.jpg)
71
Employment Interview• Types of Interviews
• Structured• Unstructured
![Page 72: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/72.jpg)
72
Employment Interview• Types of Questions
• Situational• Past behavior
![Page 73: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/73.jpg)
73
Employment Interview• To improve quality of evaluation
• Base questions on a job analysis• Ask exact same questions• Do not allow questions from candidate
till end of interview• Rate each answer or use multiple scales• Take notes• Use multiple and same trained
interviewers across candidates
![Page 74: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/74.jpg)
74
Employment Interview• Administration modes
• Telephone interviews• Validity of a telephone-
administered interview is comparable to face-to-face interviews.
• Require no travel for the interviewer or interviewee and can result in significant cost savings for geographically dispersed employers.
![Page 75: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/75.jpg)
75
Employment Interview• Administration modes
• Written structured interviews• Candidates provide written
responses to questions; administered in a group setting
• Require no travel for the interviewer or interviewee and can result in significant cost savings for geographically dispersed employers.
![Page 76: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/76.jpg)
76
Employment Interview• Psychometric Characteristics
• Reliability .67 structured interviews
• Reliability .84 job-related structured interviews
• Reliability .68 job-related unstructured interviews
![Page 77: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/77.jpg)
77
Employment Interview• Psychometric Characteristics
• Validity• Structured .44• Unstructured .33• Situational .50• Job-related .39• Psychological .29
![Page 78: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/78.jpg)
78
Employment Interview• Psychometric Characteristics• Correlations with other constructs
• High structure interviews focused more on applied mental skills, direct job knowledge, applied social skills and organizational fit.
• Low structure interviews focused more on general mental ability, background credentials, some aspects of personality, and physical attributes.
![Page 79: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/79.jpg)
79
Employment Interview• Psychometric Characteristics
• Research on subgroup differences has yielded mixed results
![Page 80: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/80.jpg)
80
Employment Interview• Psychometric Characteristics• Incremental validity beyond GMA
• Higher levels of structure result in greater incremental validity of GMA than lower levels of structure
• Correlation of .40 between interviews and GMA shows that 16% of variance in interview represents cognitive ability
![Page 81: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/81.jpg)
81
Employment Interview• How to develop structured
interviews1. SMEs review KSAs and describe
incidents of effective and ineffective performance in writing.
2. SMEs sort incidents into KSAs and rate the behaviors described in incidents for effectiveness (retranslation).
3. Select incidents allocated to KSAs by 75% of SMEs and that have low SD on effectiveness (< 1.0).
![Page 82: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/82.jpg)
82
Employment Interview• How to develop structured
interviews4. Write questions from the incidents
retained from Step 3. 5. SMEs think of people whose
performance on the job they would rate as outstanding, average, and poor, and provide written narratives describing how those people would respond to each question.
![Page 83: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/83.jpg)
83
Employment Interview• How to develop structured
interviews6. SMEs then rate the simulated responses
and those responses on which there is a high degree of rater agreement are retained as anchors for a rating scale.
![Page 84: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/84.jpg)
84
Summary - what we covered
• Questions• Test Plan• Cognitive Ability• Training and Experience• Biodata• Interviews
![Page 85: Identifying and Developing Predictors of Job Performanceannex.ipacweb.org/library/conf/03/curtin.pdf · Identifying and Developing Predictors of Job Performance ... had the time or](https://reader034.fdocuments.net/reader034/viewer/2022051405/5a7a43ae7f8b9a97398c695a/html5/thumbnails/85.jpg)
85
Identifying and Developing Predictors of Job Performance
Patrick J. CurtinDeborah L. WhetzelKenneth E. GrahamCaliber Associates
Pre-Conference Workshop, IPMAACJune 22, 2003