Validation, Meta-analyses, and the Scientific Status of Selection
description
Transcript of Validation, Meta-analyses, and the Scientific Status of Selection
![Page 1: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/1.jpg)
Validation, Meta-analyses, and the Scientific Status of
Selection
Neal Schmitt
Invited Address at the
IOOB Student Conference
Alliant University, San Diego, CA
![Page 2: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/2.jpg)
Outline
• Discuss the nature of validity evidence
• Review the bases for validity claims in employee selection research
• Discuss limitations of data base on which these claims are made
• Propose a large scale multi-national collection and sharing of data on individual difference-performance relationships
![Page 3: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/3.jpg)
Validity and Validation
• Validity is the degree to which inferences we draw from test data about future job performance are accurate.
• Standards and Principles both emphasize construct validity: Different forms of evidence– Content– Criterion-related – Construct– Consequential
![Page 4: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/4.jpg)
Construct Validity involves several inferences (Binning & Barrett,
1989)
![Page 5: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/5.jpg)
Construct Validity involves several inferences (Binning & Barrett,
1989)
Predictor measure is an accurate depiction of the intended predictor construct
1. # of Tattoos2. GRE Test Score3. MMPI Scores
1. ID w. Blue Collar Social Class
2. GMA3. Mental
Dysfunction
![Page 6: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/6.jpg)
Construct Validity involves several inferences (Binning & Barrett,
1989)
Criterion measure is an accurate depiction of the intended criterion construct (i.e., Performance Domain)
1. Truck driving ability2. Grad school
performance3. Prison guard
performance
1. # Truck driving accidents2. Grades in Grad school3. Prison guard
performance ratings
1. # of Tattoos2. GRE Test Score3. MMPI Scores
1. ID w. Blue Collar Social Class
2. GMA3. Mental
Dysfunction
![Page 7: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/7.jpg)
Construct Validity involves several inferences (Binning & Barrett,
1989)
Predictor measure is related to the criterion measure—uncorrected validity coefficient in the usual criterion-related study
1. Truck driving ability2. Grad school
performance3. Prison guard
performance
1. # Truck driving accidents2. Grades in Grad school3. Prison guard
performance ratings
1. # of Tattoos2. GRE Test Score3. MMPI Scores
1. ID w. Blue Collar Social Class
2. GMA3. Mental
Dysfunction
![Page 8: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/8.jpg)
Construct Validity involves several inferences (Binning & Barrett,
1989)
Predictor measure is related to the criterion construct – Typically a validity coefficient corrected for unreliability in the performance measure . This is of primary interest in selection context.
1. Truck driving ability2. Grad school
performance3. Prison guard
performance
1. # Truck driving accidents2. Grades in Grad school3. Prison guard
performance ratings
1. # of Tattoos2. GRE Test Score3. MMPI Scores
1. ID w. Blue Collar Social Class
2. GMA3. Mental
Dysfunction
![Page 9: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/9.jpg)
Construct Validity involves several inferences (Binning & Barrett,
1989)
1. Truck driving ability2. Grad school
performance3. Prison guard
performance
1. # Truck driving accidents2. Grades in Grad school3. Prison guard
performance ratings
1. # of Tattoos2. GRE Test Score3. MMPI Scores
1. ID w. Blue Collar Social Class
2. GMA3. Mental
Dysfunction
![Page 10: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/10.jpg)
Construct Validity involves several inferences (Binning & Barrett,
1989)
Predictor construct is related to the criterion construct; the primary theoretical/scientific question
1. Truck driving ability2. Grad school
performance3. Prison guard
performance
1. # Truck driving accidents2. Grades in Grad school3. Prison guard
performance ratings
1. # of Tattoos2. GRE Test Score3. MMPI Scores
1. ID w. Blue Collar Social Class
2. GMA3. Mental
Dysfunction
![Page 11: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/11.jpg)
Construct Validity involves several inferences (Binning & Barrett,
1989)
What about the predictor construct domain and the criterion domain?
1. Truck driving ability2. Grad school
performance3. Prison guard
performance
1. # Truck driving accidents2. Grades in Grad school3. Prison guard
performance ratings
1. # of Tattoos2. GRE Test Score3. MMPI Scores
1. ID w. Blue Collar Social Class
2. GMA3. Mental
Dysfunction
![Page 12: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/12.jpg)
Three ways to link observed predictor score to criterion construct
• Content validity: A sampling strategy in which the predictor measure is a direct sample of the criterion (example: a clerical applicant is asked to perform a word processing task). There must be psychological fidelity in representing critical aspects of the criterion construct domain
![Page 13: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/13.jpg)
Second way to support the inference that the predictor score relates to
criterion construct. Criterion-related validity: Must provide evidence for two links (5 and 8 in the figure).
![Page 14: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/14.jpg)
Construct Validity involves several inferences (Binning & Barrett,
1989)
1. The observed predictor score is correlated with the observed criterion 2. The criterion measure is an adequate representation of the criterion construct (usually a
judgment that criterion deficiency or contamination are not a problem). Evidence from job analysis, CFA analyses, etc.
![Page 15: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/15.jpg)
Third way to provide evidence that the observed predictor is related to the criterion construct:
construct validation.
This is construct validity and requires support for two links.
1. Observed predictor scores are representative of the predictor construct (test items map onto the specification of the hypothesized construct) .
2. The predictor construct domain is linked to the criterion construct domain (logical and judgmental supported by empirical research)
![Page 16: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/16.jpg)
Construct Validity Includes
• Concerns about content
• Concerns about construct – Relationships with measures of theoretically
similar and dissimilar constructs– Freedom from the usual “biasing” factors:
e.g., social desirability, faking– Process variables
![Page 17: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/17.jpg)
Examples of construct validation
• Mumford et al (PPsych, 1996): Biodata– Job-oriented variables (biodata items related to
actual job tasks)– Worker-oriented variables (biodata items
related to constructs thought to underlie job performance: Linkage 6 in the figure)
– Biodata scales were developed to predict performance in laboratory tasks (Linkage 6)
– Biodata scales were developed to predict performance of foreign service officers (Linkage 7)
![Page 18: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/18.jpg)
Physical Ability
• Arvey et al. (JAP, 1992) and Arnold et al. (JAP, 1980)– Job analyses (use of force reports, SME descriptions,
survey)– Literature review led to hypothesized importance of
endurance and strength– Collection of data on physical tasks (some face-valid
and some not—handgrip and situps—Linkage 6)– Collection of supervisory ratings on many aspects of
physical performance– Tested and confirmed a model that hypothesized the
same set of endurance and strength factors plus a rating factor – good fitting model (Linkage 7)
![Page 19: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/19.jpg)
Criterion-Related Validity ExampleProject A (more later)
1.Multiple sources of evidence that observed predictor scores were related to observed criterion.
2.Great deal of effort devoted to collection of performance data from multiple informants and in job samples. Estimates of possible contaminants and reliability.
![Page 20: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/20.jpg)
Data base on validity of existing measures is based primarily on
criterion-related validity • Job analysis• Specification of the performance domain• Hypotheses about the relevant predictor
constructs (KASOs)• Selection or construction of predictor variables• Collection of data on predictor and criterion
(concurrent and predictive validity)• Analysis of relationships between predictor and
criterion measures• Conclusions and implementation
![Page 21: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/21.jpg)
Meta-analysis: Summary of Criterion-related research and basis for Validity Generalization Claims
• Prior to the use of VG work, there was a general belief that validity of tests was unique in each situation in which a test was used: Situational Specificity
• Schmidt & Hunter (1977) showed that much of the variability in observed validity coefficients could be explained by artifacts of the study in which validity was estimated
• S & H computed the sample-size weighted averages of existing validity estimates for cognitive ability-performance relationship
• Bare-bones analysis corrected for differences in sample size only and found much of the variability in observed validity coefficients was explained
![Page 22: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/22.jpg)
Validity Generalization
• In addition, computations of credibility intervals around the averages of validity coefficients revealed that for some KASO-performance relationships, one could expect to find non-zero relationships most of the time. That is, if we are examining a situation in which the performance construct is similar to that in the validity data base, we should be able to use the predictor (or a measure of the same construct) with confidence that the validity will generalize.
![Page 23: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/23.jpg)
Results of Meta-analyses: Cognitive Ability
• Hunter and Schmidt efforts were largely restricted to reanalysis of GATB data collected by the US Department of Labor
• Hunter (1983)
• Hunter & Schmidt (1977) used data from Ghiselli (1966)
• Hunter & Hunter (1984)
• Schmidt & Hunter (1998)
![Page 24: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/24.jpg)
Other Sources of Information about Cognitive Ability
• Schmitt et al. (1984) used all published validity studies between 1964 and 1982
• Salgado and colleagues (Salgado et al., 2003; Bertua, Anderson, & Salgado, 2005) provided European data
![Page 25: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/25.jpg)
Results
• Average observed validity coefficients are almost all in the .20s. Most estimates of ρ are in the .40s and .50s.
• The lower bound of the 90% credibility interval is always substantial, from .10 to the .50s meaning it is unlikely that one would find a nonsignificant validity coefficient when using cognitive ability tests to select employees.
• Very similar results are reported for specific cognitive abilities such as memory, numerical ability, perceptual ability, psychomotor ability, spatial ability
![Page 26: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/26.jpg)
Personality Sources
Performance Outcomes
• Barrick and Mount (1991): Review of published and unpublished sources from the 1950s through the 1980s
• Tett, Jackson, & Rothstein (1991): Reanalysis of existing data exploring the role of hypothesis testing and directionality as moderators of validity
• Hurtz and Donovan (2000): Published and unpublished research on Big Five only
• Barrick, Mount, & Judge (2001) summarized 15 prior meta-analytic studies
• Salgado (1997) European community
![Page 27: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/27.jpg)
Results: Performance
• Observed validity: .04 to .22 across the Big Five
• Corrected validity: .04 to .33 across the Big Five (highest validity in Tett et al(1991))
• 90% credibility interval did not include zero for Conscientiousness and occasionally Emotional Stability
![Page 28: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/28.jpg)
Results: Other Criteria
• Most frequent alternate criterion is OCB• Much smaller sample sizes (usually less
than 2000)• Observed validity ranges from .04 to .24• Corrected validity ranges from .04 to .27• Other personality traits including customer
service orientation, core self-esteem, self efficacy, integrity tests: validities ranged from .14 to .50.
![Page 29: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/29.jpg)
Summary
• We have an extensive data base that supports the use of tests of major constructs, certainly cognitive ability and conscientiousness across most occupations
• Validities are such that significant utility can result from the use of measures of these constructs
• The validity of these constructs is generalizable
![Page 30: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/30.jpg)
Problems are with the nature of the primary data base: Consider Landy comments (2007)
• Without empirical studies, VG dies of its own weight OR meta-analysis can make a silk purse out of a sow’s ear, but you need the ears to begin with—so it is wrong to abandon criterion-related validity studies
• We are burdened with the empirical studies of the 50’s, 60’s and earlier that viewed the world through g and O (overall performance lenses)
![Page 31: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/31.jpg)
Performance Criteria
• Accidents
• Quitting
• Intentions to quit
• Completion of training
• Letters of commendation
• Performance appraisals
• Reprimands
• Tardiness
• Absences ???
• Quantity: The number of units produced, processed or sold (sales volume)
• Quality: The quality of work performed (error rates, inspection of product)
• Timeliness: How fast work is performed ( units produced per hour)
• Creativity: many white-collar jobs- keep track of creative work examples and attempt to quantify them.
• Adherence to Policy: • OCB – Extra role or contextual behaviors (enhancing the
environment) • Self-Appraisal• Peer Appraisal: • Team Appraisal• Assessment Center
Performance: 1.cognitive, motor, psychomotor, or interpersonal behavior
2.controllable by the individual
3.relevant to organizational goals
4.scalable in terms of proficiency
Campbell, J.P., McCloy, R.A., Oppler, S.H., & Sager, C.E. (1993). A theory of performance. In N. Schmitt & W.C. Borman (Eds.), Personnel selection. Jossey-Bass: San Francisco.
![Page 32: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/32.jpg)
Performance CriteriaResults of a structural equation analysis of survey research involving 20,000 respondents. The construct of Personal Satisfaction Motivation measured by 3 indicators and the constructs of Organizational Performance and Individual Performance measured by 2 indicators each
![Page 33: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/33.jpg)
Performance CriteriaResults of a structural equation analysis of survey research involving 20,000 respondents. The construct of Awards and Bonuses Motivation measured by 3 indicators and the constructs of Organizational Performance and Individual Performance measured by 2 indicators each
![Page 34: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/34.jpg)
Job Performance As A Multidimensional Concept
34
• A Multidimensional Model of Job Performance:– The eight performance dimensions are:
• Job specific task proficiency• Non-job-specific task proficiency• Written and oral communication task proficiency• Demonstrating effort• Maintaining personal discipline• Facilitating peer and team performance• Supervision/leadership • Management/administration
![Page 35: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/35.jpg)
CRITERIA OF GOOD PERFORMANCE
1. Job-specific task proficiency (substantive or technical task)
2. Non-job-specific task proficiency (task done by most everyone on the job)
3. Written and oral communication task proficiency
4. Demonstrating effort (sustained goal directed exertion)
5. Maintaining personal discipline (avoiding negative behavior )
6. Facilitating peer and team performance (supporting and moral boosting)7. Supervision / leadership (OPM: Persons who accomplish work through the direction of other people.
First level supervisors personally direct subordinates without the use of other, subordinate supervisors. A second level supervisor directs work through one layer of subordinate supervisors.)
8. Management / administration (OPM: Persons who direct the work of an organizational unit, are held accountable for the success of specific line or staff functions, monitor and evaluate the progress of the organization toward meeting goals, and make adjustments in objectives, work plans, schedules, and commitment of resources.
35
![Page 36: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/36.jpg)
DETERMINANTS OF WORK PERFORMANCE (from Blumberg & Pringle, 1982)
Capacity
OpportunityWillingness
Performance
![Page 37: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/37.jpg)
First Recorded US Army Efficiency Report
• Lt. Col. Alex Denniston A good natured man• Major Crolines A good man, but no officer• Capts. Martel, Crane, Wood All good officers• Capt. Shotwell A knave despised by all• Capt. Reynolds Imprudent and of most violent
passions• Capt. Porter Stranger, little known • Lt. Kerr Merely good, nothing
promising• Lts. Perrin, Scott, Ryan, Elworth Low vulgar men, Irish and
from the meanest walks of life• Ensign Mehan Very dregs of the earth, unfit
for anything under heaven.
![Page 38: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/38.jpg)
What’s wrong with the empirical studies that underlie our meta-analyses
• Small sample sizes or nonexistent sample sizes (Schmidt & Hunter, 1977)
• Lack of information on the two major artifacts necessitating use of hypothetical distributions: Range restriction and criterion unreliability
• Mostly concurrent criterion-related studies with various design flaws: Sussman & Robertson (JAP, 1989)
![Page 39: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/39.jpg)
Flaws continued
• Lack of sample descriptions limiting the use of the data base in studying the role of demographic characteristics
• Lack of information on organizational characteristics: Concern about organizational characteristics does not imply a return to situational specificity notions
• Conceptualization of performance: Use of single item overall performance index
• Age of the data base: At least in the cognitive area, we have few studies less than 30-40 years old and they range from 30-60 or more years
![Page 40: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/40.jpg)
Are the studies “old”: Cognitive ability example
• <1930 18• 1931-1950 42• 1951-1960 39• 1961-1970 42• 1971-1980 30• 1981-1990 34• >1991 32
• Bertua et al• Salgado et al• Vinchur et al (clerical performance)• Schmitt et al. (1964-1982)• Schmidt & Hunter (1977) < 1966• Hunter (GATB) < 1980 most likely
![Page 41: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/41.jpg)
Why is age a problem?
• Changes in work have implications• Teamwork – we must measure team performance on the
criterion end and attend to interpersonal KSAOs• Virtual work– we must attend to the context in which
work takes place in measuring KASOs• Contingent and temporary work—is there need for a
different system of selection than that used for regular workers
• Cognitive requirements of jobs have likely both increased and decreased
• Technical and nontechnical work• Constantly changing work• Global work
![Page 42: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/42.jpg)
Single most important validation study has been Project A
• Goal was to generate criterion variables, predictor measures, analytic methods, and validation data that would result in an enhanced selection and classification system for the military services in the U.S.
• 276 entry level occupations• In 1990, 800,000 person force• Screened 300,000 to 400,000 annually to select 120,000
to 140,000• ASVAB consisting of 10 subtests and a number of
composites was, and is, the primary selection tool
![Page 43: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/43.jpg)
Performance Domain (performance is behavior, not the result of behavior): Binning & Barrett -
Linkage 8
• Task analyses and critical incident analyses
• Generating the critical performance dimensions
• Constructing measures of each dimension– Rating scales– Job knowledge tests– Hands-on job samples– Archival records
![Page 44: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/44.jpg)
Data Analyses
• Content analyses of performance measures• Principal components analyses • Development of a target model of performance
for all jobs• Confirmatory factor analyses
– Core technical proficiency– General soldiering proficiency– Effort and leadership– Personal discipline– Physical fitness and military bearing
![Page 45: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/45.jpg)
Predictor Domain (Binning & Barrett, Linkage 6)
• Extensive literature review (non-cognitive, cognitive, and psychomotor areas)
• Expert judges in the field were consulted as to predictor-criterion relationships for a variety of criterion content categories
• These judgments were factor analyzed to come up with a preliminary list of predictors thought to be relevant to various performance domains
• Practical time constraints were considered in arriving at a list of paper-and-pencil cognitive ability tests, computer-administered psychomotor tests, and paper-and-pencil noncognitive measures
![Page 46: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/46.jpg)
Validity Estimates
• Both concurrent and predictive samples
• Observed validity (Binning & Barrett, Linkage 5)
• Corrected validity for restriction of range and criterion unreliability (Binning & Barrett, Linkage 9) using actual empirical values
![Page 47: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/47.jpg)
Project A was initiated 25 years ago: What’s new?
• Interest in multi-level issues
• Changes in work have escalated
• Interest in the persons evaluated: Reactions
• Sustainability is an increasingly important issue
• Technology change
![Page 48: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/48.jpg)
Proposal for Major New Exploration of KASO-Performance Relationships
• Large Sample:– Multiple organizations– Multiple jobs– Multiple countries– Collection of demographic data: Gender,
ethnic status, first language, age, disability status, etc.
![Page 49: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/49.jpg)
Careful conceptualization and measurement of Performance Constructs
• Task Performance
• OCBs
• Adaptive performance with focus on willingness and ability to learn
• Job or organizationally specific outcomes (e.g., vigilance or safety)
• Retention
• Counterproductivity
![Page 50: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/50.jpg)
Predictor Measures
• Careful specification of the predictor domain and hypothesized linkages to performance constructs
• Construct studies and related literature surveys of predictor measures
• Assessment of “biasing” factors and mode of stimulus presentation issues
![Page 51: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/51.jpg)
Design
• Predictive
• Data streaming
• Followup of those not selected
• Data should be collected and archived to allow for assessment and correction for artifacts and relationship changes over time
![Page 52: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/52.jpg)
Collection of context or macro level variables
Organizational characteristics: Job, industry type, customer service requirements, size, age, hierarchy, private vs. public, etc.
Type of work schedule and workplace: Virtual work, temporary work, etc.
Economy
Country and culture
![Page 53: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/53.jpg)
Allowance for ancillary studies
• Mode of data collection: Web-based versus computer-supervised versus paper and pencil
• How do we manage faking?
• New predictor or outcome measures
• Time and ability-outcome relationships
![Page 54: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/54.jpg)
Examination of Multi-level issues
• Team performance: Is the whole different than the sum of its parts? How do individual difference – team performance measures relate? How does team composition affect performance? Does culture or political system have any influence on these relationships
• Do relationships hold across individuals, teams, units, organizations?
• Are there cross-level effects?• E.g., what is the effect of culture on selection
practices? If people within a culture are more similar on some trait than those from different cultures, validity findings will not generalize.
![Page 55: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/55.jpg)
Placement and Classification Issues
• We know from large scale military studies and simulations that a classification model can improve the utility of selection procedures rather dramatically even when validity differences in composites are relatively tiny. Key here are the intercorrelations between predictors as well as the validity. Utility increases come from decreases in selection ratios for individual jobs.
• We have few applications in civilian arena
![Page 56: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/56.jpg)
Reactions to selection procedures
• Fairness• Relevance • Feedback• Interpersonal treatment• Consistency• Reconsideration opportunity• We know there are cultural differences in the
use of and reaction to selection procedures, but have no studies of the impact on validity
![Page 57: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/57.jpg)
Sustainability (Kehoe et al., In press)
• Continued benefit must be recognized• Organizational fit (is the selection system
consistent with the organizational culture)• Tradeoffs: Quality versus speed and
administrative simplicity• Cost• Luck associated with who is in charge• Perceived fairness• Perceived defensibility
![Page 58: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/58.jpg)
Provision for data-sharing
• Must address individual and organizational confidentiality issues
• Division of labor and maintenance of data base
• Has been done for decades using survey research: “No name” group
![Page 59: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/59.jpg)
Summary and Conclusions
• We have an extensive data base that supports our inferences about the use of many of our selection procedures
• That data base in “old” and deficient in a variety of ways
• Modern theory and research can improve the data base in important practical and scientific ways
• There is a need for a macro cooperative study that is global, cross-cultural, multi-organizational, and continuous
![Page 60: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/60.jpg)
Thank you very much!
![Page 61: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/61.jpg)
Contamination and deficiency
• Contamination– Pulling in irrelevant
information
• Examples– Asking salespeople
about the area football team
– Subjective judgments about the “right” way to dress
• Deficiency– Missing relevant
information
• Examples– Failing to assess a
security guard’s aggressive disposition
– Failing to measure whether someone who persists at tasks has the ability to “let go” when the task is over
Unit 2, Lecture 3: Introduction to Selection
![Page 62: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/62.jpg)
Content validity: Are you measuring correctly?
Unit 2, Lecture 3: Introduction to Selection
Measurement Construct
Contamination Valid Deficiency
![Page 63: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/63.jpg)
Contamination and deficiency can be traded off with scope
Unit 2, Lecture 3: Introduction to Selection
Measurement Construct
Contamination Valid Deficiency
![Page 64: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/64.jpg)
Content Validity
• Developing measures– Look at the job analysis you’ve already done– Compare your current and proposed methods of
assessment to the KSAO or competency matrix for the job– Try to develop new measures of assessment that are
especially relevant for each of the components of the job– Reject all measures that are not demonstrably related to
documented KSAOs or competencies
Unit 2, Lecture 3: Introduction to Selection
![Page 65: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/65.jpg)
Content Validity
• Example for a dentist, who must know or have skill with:– Techniques needed to diagnose and treat dental injuries and diseases– Chemical composition, structure, and properties of substances and of
the chemical processes and transformations that they undergo. – Principles and processes for providing customer and personal services. – Business and management principles.– Principles and methods for curriculum and training design, teaching and
instruction.– Principles and methods for promoting and selling products or services.
• What methods could you use to assess each of these?
Unit 2, Lecture 3: Introduction to Selection
![Page 66: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/66.jpg)
Face Validity
• Face validity– Do the elements of the selection system look like they’re
getting at something important?– Can also incorporate how applicants react to a measure– This isn’t really a form of recognized validity
• No conventional way to test for face validity• No specific standard for a test’s minimum level of face validity
– Why can face validity steer you wrong?• Facially valid (to some) but useless measures• Facially invalid (to some) but useful measures
Unit 2, Lecture 3: Introduction to Selection
![Page 67: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/67.jpg)
Face validity
• Methods for assessment and improvement– Review items that have been used by other
organizations– Survey current workforce or a general audience
about your proposed method to assess whether the measures look relevant
– Potentially have a legal expert look over items • Remember: lawyers don’t always use statistics to
build their arguments, so face validity could matter in court
Unit 2, Lecture 3: Introduction to Selection
![Page 68: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/68.jpg)
The Logic of Prediction and Criterion Validity
Unit 2, Lecture 3: Introduction to Selection
Predictor Measure X
Predictor Construct
Criterion Measure Y
Criterion Construct
This path, which can be estimated statistically, is criterion-related validity
![Page 69: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/69.jpg)
Two Designs for Criterion Validity
• Predictive validation– Collect predictor
data at hire– Collect criterion data
after a span of time
• Advantages?
• Problems?
• Concurrent validation– Collect predictor
data and criterion data using current employees
• Advantages?
• Problems?
Unit 2, Lecture 3: Introduction to Selection
![Page 70: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/70.jpg)
Returning to our design strategy
• Imagine we have a number of job applicants– We have a measure
of “job readiness”• Examples?
– We also have job performance measures
• Examples?
Job test Performance
Joe S. 25 30
Nate D. 15 20
Fran G. 30 25
Al C. 19 21
Marie S. 12 14
Nancy G. 8 15
Ramon F. 22 33
Lou M. 4 12
Wei S. 29 29
Unit 2, Lecture 3: Introduction to Selection
![Page 71: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/71.jpg)
Criterion-related Validity: A Graphical Illustration
• The line cuts through the middle of the data to create a best guess of the relationship between test scores and performance.
• The expected performance of a person with score a is lower than the performance of a person with score b.
Unit 2, Lecture 3: Introduction to Selection
performance
a b
x
xx
x
x
x
x
x
x
x x
x x
x
x
x
x x
xx
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
xx
x
x
x
test score
![Page 72: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/72.jpg)
A Sample Correlation-3
-2-1
01
23
Job
per
form
ance
-3 -2 -1 0 1 2 3Predictor score
• R=0.50• This is close to the
intelligence-performance relationship
• Also close to the job knowledge-performance relationship
• Also close to the composite correlation for all personality dimensions
Unit 2, Lecture 3: Introduction to Selection
![Page 73: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/73.jpg)
A Sample Correlation-3
-2-1
01
23
Job
per
form
ance
-3 -2 -1 0 1 2 3Predictor score
• R=0.30• This is close to the
personality-performance relationship for the most valid personality traits
Unit 2, Lecture 3: Introduction to Selection
![Page 74: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/74.jpg)
A Sample Correlation-3
-2-1
01
23
Job
per
form
ance
-3 -2 -1 0 1 2 3Predictor score
• R=0.00• This is close to the
relationship between handwriting samples and job performance
Unit 2, Lecture 3: Introduction to Selection
![Page 75: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/75.jpg)
An Alternative Way to Think about This Relationship
Unit 2, Lecture 3: Introduction to Selection
x
xx
x
x
x
x
x
x
x x
x x
x
x
x
x x
xx
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
xx
x
x
x
hirednot hired
unsatisfactoryperformance
satisfactoryperformance
false positive
false negative
true negative
true positive
cut score
test score
job performance
![Page 76: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/76.jpg)
Accuracy of Prediction
Unit 2, Lecture 3: Introduction to Selection
![Page 77: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/77.jpg)
Accuracy of Prediction
Unit 2, Lecture 3: Introduction to Selection
![Page 78: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/78.jpg)
Standards for Selection
• Cut scores– The minimum level of performance that is acceptable for an
applicant to be considered minimally qualified
• Methods of selecting from applicants– Minimum qualification
• Hire sequentially based on the first applicants who score above the cut score for the job
– Top down hiring• Give job offers starting from the most qualified and progressing to the
lowest score (ensuring all scores are above the cut point)
– When is each method used? What are the advantages of each method?
Unit 2, Lecture 3: Introduction to Selection
![Page 79: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/79.jpg)
Baseline Cut Score
Unit 2, Lecture 3: Introduction to Selection
x
xx
x
x
x
x
x
x
x x
x x
x
x
x
x x
xx
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
xx
x
x
x
hirednot hired
unsatisfactoryperformance
satisfactoryperformance
false positive
false negative
true negative
true positive
![Page 80: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/80.jpg)
Increase cut score
Unit 2, Lecture 3: Introduction to Selection
x
xx
x
x
x
x
x
x
x x
x x
x
x
x
x x
xx
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
xx
x
x
x
hirednot hired
unsatisfactoryperformance
satisfactoryperformance
false positive
false negative
true negative
true positive
![Page 81: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/81.jpg)
Reduce Cut Score
Unit 2, Lecture 3: Introduction to Selection
x
xx
x
x
x
x
x
x
x x
x x
x
x
x
x x
xx
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
xx
x
x
x
hirednot hired
unsatisfactoryperformance
satisfactoryperformance
false positive
false negative
true negative
true positive
![Page 82: Validation, Meta-analyses, and the Scientific Status of Selection](https://reader035.fdocuments.net/reader035/viewer/2022070405/56813e3e550346895da821ba/html5/thumbnails/82.jpg)
Of Course, It’s Best to Improve Validity
Unit 2, Lecture 3: Introduction to Selection
x
xx
x
x
x
x
xx
x xx x
x
xx
x x
xx
x
x
x
x
x
xx
xx
x
x
x
x
xx
x
x
x
x
x x
x
x
xx
xx
x
x
x
hirednot hired
unsatisfactoryperformance
satisfactoryperformance
false positive
false negative
true negative
true positive
x
xx
x
x
x
x
xx
x xx
x
x
xx
x x
xx
x
x
x
x
x
xx
xx
x
x
x x
xx
x
x
xx
x
x
x
x
xx
xx
x
xx
hirednot hired
unsatisfactoryperformance
satisfactoryperformance
false positive
false negative
true negative
true positive
Low validity
High validity