PSY 6430 Unit 6

1

PSY 6430 Unit 6PSY 6430 Unit 6

Job AnalysisJob Analysis

Tonight and Monday: LectureMonday, 3/27: Exam

2

Introduction: Job Analysis (NFE) Introduction: Job Analysis (NFE) Job analysis is the bedrock of any selection

procedure and is required from a legal perspective if the selection procedure results in adverse impact and is challenged in court

There are several different types of job analysis procedures

I am going to focus on two Task analysis (and task questionnaire) Position Analysis Questionnaire (PAQ)

Commercially available structured job analysis questionnaire; an off-the-shelf job analysis questionnaire

From the hand-out I gave you from the 6th edition of the text

Job analysis is the bedrock of any selection procedure and is required from a legal perspective if the selection procedure results in adverse impact and is challenged in court

There are several different types of job analysis procedures

I am going to focus on two Task analysis (and task questionnaire) Position Analysis Questionnaire (PAQ)

Commercially available structured job analysis questionnaire; an off-the-shelf job analysis questionnaire

From the hand-out I gave you from the 6th edition of the text

(just need a little intro before I talk about the project)

3

Introduction: Job Analysis (NFE) Introduction: Job Analysis (NFE) The authors first talk about a job analysis interview

But that is rarely used by itself Used by itself only when there are only a few job incumbents in

the position Usually combined with the task analysis and used only as the

first step in the process The task analysis and PAQ are the most popular types

of job analysis The authors also talk about the Critical Incidents

Technique, but I am not going to focus on that Critical incidents technique focuses on excellent and poor ways

of handling important job-related situations/scenarios It’s great to use for developing interview questions and

situational tests but does not result in a comprehensive job analysis

The authors first talk about a job analysis interview But that is rarely used by itself Used by itself only when there are only a few job incumbents in

the position Usually combined with the task analysis and used only as the

first step in the process The task analysis and PAQ are the most popular types

of job analysis The authors also talk about the Critical Incidents

Technique, but I am not going to focus on that Critical incidents technique focuses on excellent and poor ways

of handling important job-related situations/scenarios It’s great to use for developing interview questions and

situational tests but does not result in a comprehensive job analysis

(always started with the exemplary performers; great to use after a comprehensive analysis to develop some of theinstruments/methods for selection; main usefulness after a comprehensive analysis – supplement; face validity)

4

*Learn for the exam! Next slide shows the relationship.

Relationship of job analysis to type of validity procedure*

Relationship of job analysis to type of validity procedure*

As I indicated last week, the type of validity procedure you use is dependent upon several factors Sample size Whether the KSA is observable or not Whether you use an off-the-shelf test or write the test What type of cut-off score procedure you want to use

As I indicated last week, the type of validity procedure you use is dependent upon several factors Sample size Whether the KSA is observable or not Whether you use an off-the-shelf test or write the test What type of cut-off score procedure you want to use

5

Relationship of job analysis to type of validity procedure

Relationship of job analysis to type of validity procedure

Once you decide what validity procedure you want to use, then you must make sure that your job analysis procedure is appropriate for that type of validity procedure

Once you decide what validity procedure you want to use, then you must make sure that your job analysis procedure is appropriate for that type of validity procedure

Validity Procedure Type of Job Analysis

Content validity

Criterion-related validity

Validity generalization

Task analysis only

Task analysis, PAQ, any type

Any type is OK, should be thesame as the type usedby the other company

(required by Uniform Guidelines; you cannot use the PAQ or any other type - I’ll explain why you cannot use the PAQ a bit later; predictive and concurrent; any type because the proof is in the correlation unlike content solely expert judgment; job component - PAQ)

6

Task Analysis ProjectTask Analysis Project

Most companies are now using content validity, so I am focusing on the task analysis

Due Monday, April 22 (final exam week) Worth 70 points (Units 9 and 10) Get started now This is a time consuming project

Most companies are now using content validity, so I am focusing on the task analysis

Due Monday, April 22 (final exam week) Worth 70 points (Units 9 and 10) Get started now This is a time consuming project

(but I obviously couldn’t assign it until we got to this point in the course)

7

Task Analysis ProjectTask Analysis Project Task analysis is the first step in the development

of a task inventory (job analysis questionnaire) I am not going to have you develop the questionnaire

or get ratings for the task statements or KSAs Project consists of just the first step in the

process - the identification of tasks and KSAs linked to each of the tasks

Three models in the course pack Lindsay Street, Thor Flosason, & Julie Slowiak

Task analysis is the first step in the development of a task inventory (job analysis questionnaire) I am not going to have you develop the questionnaire

or get ratings for the task statements or KSAs Project consists of just the first step in the

process - the identification of tasks and KSAs linked to each of the tasks

Three models in the course pack Lindsay Street, Thor Flosason, & Julie Slowiak

8

Task Analysis ProjectTask Analysis Project Task analysis is the most versatile job

analysis procedure because It can be used for both content and criterion-

validity studies and Is legally required for content validity (if tests

are challenged in court)

Task analysis is the most versatile job analysis procedure because It can be used for both content and criterion-

validity studies and Is legally required for content validity (if tests

are challenged in court)

9

Task Analysis ProjectTask Analysis Project Resource on the web that will make this project a

lot easier and a lot less time consuming for you O*NET

Occupational Information Network – Dept. of Labor Web site address is in the instructions

To start with, go to O*NET and use “Bartender” as the example position

Also use this for your own job searchRelated titles and ability “buzz words”for your resumes

Resource on the web that will make this project a lot easier and a lot less time consuming for you

O*NET Occupational Information Network – Dept. of Labor Web site address is in the instructions

To start with, go to O*NET and use “Bartender” as the example position

Also use this for your own job searchRelated titles and ability “buzz words”for your resumes

(more detailed instructions in the description of project)

10


You may conduct the analysis for one job incumbent

You cannot do the task analysis for a job you currently have or have had in the past I want you to have the professional experience of

collecting information about a job that you are not familiar with - this is much harder than you think

The job may be part-time or full-time

You may conduct the analysis for one job incumbent

You cannot do the task analysis for a job you currently have or have had in the past I want you to have the professional experience of

collecting information about a job that you are not familiar with - this is much harder than you think

The job may be part-time or full-time

11

Task Analysis ProjectTask Analysis Project The analysis will consist of

Task statements KSAs linked to each task statement Work activities: Physical activities and requirements

(required for ADA) Work Context: Environmental conditions (also good for

ADA) Work Styles: Typical working incidents (again good for

ADA) This is where many job analysts put things like “persuading

individuals, working under stress, getting along with others” that are critical for ADA

Any licensing or certification requirements

The analysis will consist of Task statements KSAs linked to each task statement Work activities: Physical activities and requirements

(required for ADA) Work Context: Environmental conditions (also good for

ADA) Work Styles: Typical working incidents (again good for

ADA) This is where many job analysts put things like “persuading

individuals, working under stress, getting along with others” that are critical for ADA

Any licensing or certification requirements

(don’t forget job attendance -getting to work on time, and regular attendance, if that is required - ADA; Tasks MUST BE EXACT FORMAT)

12

Task Analysis ProjectTask Analysis Project Task statement format

Gatewood, Feild, & Barrick, Figure 7.1, page 256 My lecture tonight Worksheets provided in course pack

Task statement format Gatewood, Feild, & Barrick, Figure 7.1, page 256 My lecture tonight Worksheets provided in course pack

(Tasks MUST BE EXACT FORMAT)

13

Task Analysis ProjectTask Analysis Project Caution! The task statement format differs in

O*NET O*NET is the end result of a task inventory, not the

starting point O*NET has 8 sections after the tasks and KSAs

I am requiring only the following three work activities work context work styles

But adding professional licensing and certification requirements

Caution! The task statement format differs in O*NET O*NET is the end result of a task inventory, not the

starting point O*NET has 8 sections after the tasks and KSAs

I am requiring only the following three work activities work context work styles

But adding professional licensing and certification requirements

14


You should have between 15-30 task statements Part of the problem with writing task statements is

determining how specific or how broad they should be

You will understand this difficulty when you begin to write the task statements

You should have between 15-30 task statements Part of the problem with writing task statements is

determining how specific or how broad they should be

You will understand this difficulty when you begin to write the task statements

15

Task Analysis Project: StepsTask Analysis Project: Steps

See the project description which indicates where you find instructions for

Writing task statements Writing KSAs Documenting work activities, work context, work

styles and licensure/certification requirements

See the project description which indicates where you find instructions for

Writing task statements Writing KSAs Documenting work activities, work context, work

styles and licensure/certification requirements

16

Task Analysis Project: StepsTask Analysis Project: Steps Preparing for the meetings

Before the meeting, obtain any written documentation that exists about the job

Job descriptions, organizational chart, etc. Locate job title/position using O*Net (if you can find a similar

position) Meet with the job incumbent three times

First meeting: Task statements only (1.5-2 hrs) Ask the incumbent just to identify/state the major aspects or

areas of his/her job After you have that list, go back to the first major area and

ask for the tasks in that area Move onto the next major area, etc.

Preparing for the meetings Before the meeting, obtain any written

documentation that exists about the job Job descriptions, organizational chart, etc. Locate job title/position using O*Net (if you can find a similar

position) Meet with the job incumbent three times

First meeting: Task statements only (1.5-2 hrs) Ask the incumbent just to identify/state the major aspects or

areas of his/her job After you have that list, go back to the first major area and

ask for the tasks in that area Move onto the next major area, etc.

(different interview procedure from GFB; they provide interview guides on pages 254&259 are very good, but 3 meetings rather than 1;Also strongly recommend that you record the interview – never going to remember everything, miss things )

17


Second meeting (1.5-2 hours) Confirm accuracy of task statements, obtain KSAs,

and perhaps work activities, work context, work styles and licensure/certification (time permitting)

Before meeting with the incumbent, type up a list of the task statements Identify any missing components to the tasks that you need to

ask about Have the incumbent review the task statements for accuracy

and completeness Take each task statement one at a time and ask the

incumbent what KSAs are required to perform that particular task GFB interview guide, Figure 7.3 on page 254 may be helpful

here

Second meeting (1.5-2 hours) Confirm accuracy of task statements, obtain KSAs,

and perhaps work activities, work context, work styles and licensure/certification (time permitting)

Before meeting with the incumbent, type up a list of the task statements Identify any missing components to the tasks that you need to

ask about Have the incumbent review the task statements for accuracy

and completeness Take each task statement one at a time and ask the

incumbent what KSAs are required to perform that particular task GFB interview guide, Figure 7.3 on page 254 may be helpful

here

18


Third meeting: Review of entire document Type up the entire document and have the job

incumbent review it. If revision is required, then you should ask the

incumbent to review it one more time Some students have done this last step virtually which

is fine – but it is not good to do the first two steps virtually

Third meeting: Review of entire document Type up the entire document and have the job

incumbent review it. If revision is required, then you should ask the

incumbent to review it one more time Some students have done this last step virtually which

is fine – but it is not good to do the first two steps virtually

19

Task Analysis Project: FormatTask Analysis Project: Format

See models in the course pack Description of the company and position

Include the name of the position in the company and the corresponding O*Net position name, if there is one

Description of the methods/steps you used to develop the task analysis.

How many times did you meet with the incumbent? What did you do during each meeting? How long was each meeting?

See models in the course pack Description of the company and position

Include the name of the position in the company and the corresponding O*Net position name, if there is one

Description of the methods/steps you used to develop the task analysis.

How many times did you meet with the incumbent? What did you do during each meeting? How long was each meeting?

20

Task Analysis Project: FormatTask Analysis Project: Format

Task Statements with KSAs linked to each task statement

Work Activities* Work Context*

Environmental conditions and physical requirements Work Styles*

Key ADA requirements - getting along with others, dependability, coming to work regularly, etc.

Licensure/Certification Requirements

Task Statements with KSAs linked to each task statement

Work Activities* Work Context*

Environmental conditions and physical requirements Work Styles*

Key ADA requirements - getting along with others, dependability, coming to work regularly, etc.

Licensure/Certification Requirements

*See bartender example on O*Net for examples

21

Questionsabout the project?

Questionsabout the project?

22

SO2: Job Analysis (review)SO2: Job Analysis (review)

Discrepancy between the professional and the legal requirements regarding job analysis Job analysis is not required professionally for predictive

and concurrent validity (it is required for content and validity generalization) The proof of job-relatedness rests with a statistically significant

correlation between test scores and job performance measures

It is, however, required legally for all types of validity procedures

Discrepancy between the professional and the legal requirements regarding job analysis Job analysis is not required professionally for predictive

and concurrent validity (it is required for content and validity generalization) The proof of job-relatedness rests with a statistically significant

correlation between test scores and job performance measures

It is, however, required legally for all types of validity procedures

23

SOs 7&8: Task Statement FormatSOs 7&8: Task Statement Format

What activity is performed? Use a single present tense action verb - verb ends in

an “s” To whom or what?

The object of the verb Why?

What is produced - what is the expected outcome of the action

How? Using what materials, tools, procedures, guidelines,

or equipment

What activity is performed? Use a single present tense action verb - verb ends in

an “s” To whom or what?

The object of the verb Why?

What is produced - what is the expected outcome of the action

How? Using what materials, tools, procedures, guidelines,

or equipment

(moving to Sos 7&8; four components of a task statement; two tasks for a job you have held)

24

Task Statement ExamplesWelfare Eligibility ExaminerTask Statement ExamplesWelfare Eligibility Examiner

What activity is performed? Asks questions, listens, and records answers

To whom or what? Of client on eligibility form

Why? In order to determine eligibility for food stamps

How? Using eligibility form, eligibility criteria in manual,

interviewing techniques

What activity is performed? Asks questions, listens, and records answers

To whom or what? Of client on eligibility form

Why? In order to determine eligibility for food stamps

How? Using eligibility form, eligibility criteria in manual,

interviewing techniques

(from GFB; note more than one action verb that are all related to the same WHY; Why is the most frequently omitted component when students write task statements; clean up the task statement – next slide)

25


Clean it up

Asks client questions, listens, and records answers on standard eligibility form to gather information from which the client’s eligibility for food stamps can be determined using knowledge of interview techniques and eligibility criteria.

Clean it up

Asks client questions, listens, and records answers on standard eligibility form to gather information from which the client’s eligibility for food stamps can be determined using knowledge of interview techniques and eligibility criteria.

(notice, several different verbs, actions that go together for the same “in order to”; often a question students have, nextSlide another example)

26


What activity is performed? Determines eligibility

To whom or what? Of applicant

Why? In order to complete client’s application for food

stamps How?

Using regulatory policies as a guide

What activity is performed? Determines eligibility

To whom or what? Of applicant

Why? In order to complete client’s application for food

stamps How?

Using regulatory policies as a guide

(another example - same job; once they obtain information from the client, then they must decide whether the person is eligible using policies; next slide different topic)

27

SO9: Problems with Job Interview as only source (this should not surprise you)

SO9: Problems with Job Interview as only source (this should not surprise you)

Lack of standardization across job analysts Likely to get different information

Not practical for large numbers of incumbents because it’s one-on-one (2-3 hrs at least) Once you have more than 5-10 job incumbents, it

gets very time consuming and hence expensive Legally, you must have a representative sample

stratified by geographical region (if relevant), and protected classes, thus you may end up interviewing more incumbents than would otherwise be necessary

Lack of standardization across job analysts Likely to get different information

Not practical for large numbers of incumbents because it’s one-on-one (2-3 hrs at least) Once you have more than 5-10 job incumbents, it

gets very time consuming and hence expensive Legally, you must have a representative sample

stratified by geographical region (if relevant), and protected classes, thus you may end up interviewing more incumbents than would otherwise be necessary

(again, rarely, if ever used alone; I have just asked you to list any three of the ones given; more on next page)

28

SO9: Problems with job interview as only source, cont.

SO9: Problems with job interview as only source, cont.

Easy to miss things when you are interviewing someone due to the quick verbal exchanges (as you will find out when you do your project) Thus, unless you thoroughly document each interview, the legal

requirements for the job analysis may not be met Results are highly dependent upon the skills of both the

interviewer and interviewee Text lists some others - but clearly it is not good to use

interviews alone in most cases

Easy to miss things when you are interviewing someone due to the quick verbal exchanges (as you will find out when you do your project) Thus, unless you thoroughly document each interview, the legal

requirements for the job analysis may not be met Results are highly dependent upon the skills of both the

interviewer and interviewee Text lists some others - but clearly it is not good to use

interviews alone in most cases

29

SOs 10 & 11: Task Statement and KSA Rating Scales


The Uniform Guidelines discusses the technical standards for a job analysis if it is used for content validation Google or see www.ipmaac.org/files/ug.pdf

All of the following four rating scales must be included for task statements Frequency of task performance Task importance or criticality (add essential function?) Task difficulty Whether the task can be learned on the job within a

relatively short period of time (within 6 months)

The Uniform Guidelines discusses the technical standards for a job analysis if it is used for content validation Google or see www.ipmaac.org/files/ug.pdf

All of the following four rating scales must be included for task statements Frequency of task performance Task importance or criticality (add essential function?) Task difficulty Whether the task can be learned on the job within a

relatively short period of time (within 6 months)

(KSA rating scales next)

30



All of the following rating scales must be included for KSAs (if, again, used for content validation) Is the KSA necessary for successful job performance? Is the KSA required upon entry to the job or can it be

learned within a relatively short period of time (within 6 months)

How difficult is it to obtain the KSA? What type of education/experience/licensure/certification is

necessary?

All of the following rating scales must be included for KSAs (if, again, used for content validation) Is the KSA necessary for successful job performance? Is the KSA required upon entry to the job or can it be

learned within a relatively short period of time (within 6 months)

How difficult is it to obtain the KSA? What type of education/experience/licensure/certification is

necessary?

(as you can see, the task inventory/questionnaire becomes quite extensive; 4 scales for tasks at least 3 for KSAs)

31

SO12: Obtaining names of the job incumbentsSO12: Obtaining names of the job incumbents

I strongly disagree with the authors about obtaining names on questionnaires Employees tend to be very nervous and concerned

about how the information will be used When I have interviewed individuals, even one-on-

one, I have almost always had to assure them that they would not be identified - rather the information would be combined with the information of others, with no identifiers.

Most employees just don’t trust the “corporate office,” HR or their bosses not to use the information against them (evaluation, downsizing, elimination of job)

You will probably be interviewing people you don’t know

I strongly disagree with the authors about obtaining names on questionnaires Employees tend to be very nervous and concerned

about how the information will be used When I have interviewed individuals, even one-on-

one, I have almost always had to assure them that they would not be identified - rather the information would be combined with the information of others, with no identifiers.

Most employees just don’t trust the “corporate office,” HR or their bosses not to use the information against them (evaluation, downsizing, elimination of job)

You will probably be interviewing people you don’t know

(you have no established relationship with these individuals and if you are from “corporate” you are immediately distrusted)

32

SO12: Obtaining names, cont. (this slide NFE)SO12: Obtaining names, cont. (this slide NFE)

You should handle this just like you handle a participant list for research You should create a master list with names and code

numbers that correspond to your stratified sample Keep it under lock and key - you are the only one who

should have access to it (or if you are not in charge of the analysis, the person that is)

Put the code numbers on the questionnaire, so you know whether you have a representative sample when the questionnaires are returned

You should handle this just like you handle a participant list for research You should create a master list with names and code

numbers that correspond to your stratified sample Keep it under lock and key - you are the only one who

should have access to it (or if you are not in charge of the analysis, the person that is)

Put the code numbers on the questionnaire, so you know whether you have a representative sample when the questionnaires are returned

(Why codes, next slide)

33

SO12: Obtaining names, cont. (this slide NFE)SO12: Obtaining names, cont. (this slide NFE)

Why codes on questionnaires? There is a legal requirement to have a representative

sample for your job analysis The legal requirement pertains to who participated in the

job analysis, not simply who you sent questionnaires to If you don’t get the sample you need, contact everyone

in the underrepresented categories, with the standard, “We are contacting you because we recently sent you a questionnaire to complete. If you have not completed the questionnaire, we would greatly appreciate it if you would do so. If you have, please accept our thanks, and ignore this message.”

Why codes on questionnaires? There is a legal requirement to have a representative

sample for your job analysis The legal requirement pertains to who participated in the

job analysis, not simply who you sent questionnaires to If you don’t get the sample you need, contact everyone

in the underrepresented categories, with the standard, “We are contacting you because we recently sent you a questionnaire to complete. If you have not completed the questionnaire, we would greatly appreciate it if you would do so. If you have, please accept our thanks, and ignore this message.”

34

SO13: Advantages and Disadvantages of the Task Inventory


Advantages It is an efficient way to collect data from a large

number of individuals, particularly if they are geographically dispersed

Data can be easily quantified due to the rating scales

Advantages It is an efficient way to collect data from a large

number of individuals, particularly if they are geographically dispersed

Data can be easily quantified due to the rating scales

35



Disadvantages Development is time consuming and expensive Motivation to complete the questionnaire can be a

problem because it is often very long due to the rating scales that must be included

Ambiguities and questions may arise when individuals complete the form that can’t be dealt with, which may result in inaccurate and less reliable information

Disadvantages Development is time consuming and expensive Motivation to complete the questionnaire can be a

problem because it is often very long due to the rating scales that must be included

Ambiguities and questions may arise when individuals complete the form that can’t be dealt with, which may result in inaccurate and less reliable information

36

SO14: Court case related to job analysis, the two implications: intro

SO14: Court case related to job analysis, the two implications: intro

Kirkland v NY State Department of Correctional Services, 1974 The litigated test was a promotional exam for correctional officers The passing rates for whites was 31% while the passing rate for

blacks was 8% Adverse impact had been demonstrated thus the burden of proof

shifted to the organization Organizational defended the exam on the basis of content validity The court ruled in favor of the plaintiffs ruling that

The job analysis was not adequate, but more importantly because job analysis had been stressed in other cases

The process used to construct the test and the content of the test was not adequate

Kirkland v NY State Department of Correctional Services, 1974 The litigated test was a promotional exam for correctional officers The passing rates for whites was 31% while the passing rate for

blacks was 8% Adverse impact had been demonstrated thus the burden of proof

shifted to the organization Organizational defended the exam on the basis of content validity The court ruled in favor of the plaintiffs ruling that

The job analysis was not adequate, but more importantly because job analysis had been stressed in other cases

The process used to construct the test and the content of the test was not adequate

(very important case related to job analysis when used for content validity; not in text)

37

SO14: Court case, the two implications*SO14: Court case, the two implications* Not only do you have to document the job analysis

process, but you also must document the process used to construct the test Did you develop a matrix that linked the KSAs to specific items on

the test? Who participated in that process and when did those meetings

occur? Remember, court cases often occur years after the development

of the test and you are simply not going to remember the dates of the meetings, who attended, or the outcome of those meetings

*For the exam

Not only do you have to document the job analysis process, but you also must document the process used to construct the test Did you develop a matrix that linked the KSAs to specific items on

the test? Who participated in that process and when did those meetings

occur? Remember, court cases often occur years after the development

of the test and you are simply not going to remember the dates of the meetings, who attended, or the outcome of those meetings

*For the exam

(first one this slide, second next slide)

38

SO14: Court case, the two implicationsSO14: Court case, the two implications The court may look at the test itself, not just the job

analysis procedure and ask why specific items were included and why they were weighted the way they were given the job analysis This is why it is important/essential to construct a test if you use

content validity rather than using an off-the-shelf test, and why I have been stressing that point

The court may look at the test itself, not just the job analysis procedure and ask why specific items were included and why they were weighted the way they were given the job analysis This is why it is important/essential to construct a test if you use

content validity rather than using an off-the-shelf test, and why I have been stressing that point

39

Introduction: Position Analysis Questionnaire (PAQ) (NFE)


The PAQ is a structured, standardized job analysis questionnaire that can be used to analyze virtually any job except managerial and professional positions McCormick, Jeanneret, & Mecham Professional and Managerial Position Questionnaire (PMPQ)

There is no other job analysis procedure that has been researched more extensively Research started in 1974, over 30 years ago

Over 330,000 different jobs have been analyzed in the past 30 years

Can also be used for job evaluation (salary determination) and development of competencies

The PAQ is a structured, standardized job analysis questionnaire that can be used to analyze virtually any job except managerial and professional positions McCormick, Jeanneret, & Mecham Professional and Managerial Position Questionnaire (PMPQ)

There is no other job analysis procedure that has been researched more extensively Research started in 1974, over 30 years ago

Over 330,000 different jobs have been analyzed in the past 30 years

Can also be used for job evaluation (salary determination) and development of competencies

(onto the PAQ)

40



The authors moved a description of the PAQ to the validity chapter – job component validity*

Job components being identified by the PAQ analysis and then using validity data/information from the PAQ database, selecting test instruments

Using them without doing a local validity study I don’t recommend that for legal reasons although at

least one court has upheld its use this way I recommend a more conservative approach, using it to

identify selection tests and then conducting a local empirical validity study Cannot be used for content validity – job/worker attributes, not

tasks

The authors moved a description of the PAQ to the validity chapter – job component validity*

Job components being identified by the PAQ analysis and then using validity data/information from the PAQ database, selecting test instruments

Using them without doing a local validity study I don’t recommend that for legal reasons although at

least one court has upheld its use this way I recommend a more conservative approach, using it to

identify selection tests and then conducting a local empirical validity study Cannot be used for content validity – job/worker attributes, not

tasks(relates job components to skills measured by selection tests; transportability for jobs for which job tasks are not similar – requirement for validity generalization; building block approach )

41



The PAQ analyzes jobs in terms of 187 job elements or attributes that are broken down into six categories Information input

When and how a worker gets information needed to perform the job

Mental processes The reasoning, decision making, planning, and information

processing activities that are involved in performing the job Work output

The physical activities, tools, and devices used by the worker to perform the job

The PAQ analyzes jobs in terms of 187 job elements or attributes that are broken down into six categories Information input

When and how a worker gets information needed to perform the job

Mental processes The reasoning, decision making, planning, and information

processing activities that are involved in performing the job Work output

The physical activities, tools, and devices used by the worker to perform the job

42



Relationships with other persons What relationships with other persons are required to

perform the job? Job context

The physical and social context in which the work is performed

Other job characteristics The activities, conditions, and characteristics other than

those already covered

Relationships with other persons What relationships with other persons are required to

perform the job? Job context

The physical and social context in which the work is performed

Other job characteristics The activities, conditions, and characteristics other than

those already covered

43



Information input (broken into visual, nonvisual, sensory & perceptual processes, and estimation activities) Visual sources of job information - examples

Written material (books, reports, office notes, signs) Quantitative materials (graphs, accounts, tables of numbers) Pictorial materials (drawings, blueprints, diagrams, maps)

Nonvisual sources of job information - examples Verbal sources (instructions, requests, interviews, orders) Nonverbal sources (noises, engine sounds, sonar, music) Touch (pressure, pain, feeling the texture of a surface) Taste (food tasting, wine tasting)

Information input (broken into visual, nonvisual, sensory & perceptual processes, and estimation activities) Visual sources of job information - examples

Written material (books, reports, office notes, signs) Quantitative materials (graphs, accounts, tables of numbers) Pictorial materials (drawings, blueprints, diagrams, maps)

Nonvisual sources of job information - examples Verbal sources (instructions, requests, interviews, orders) Nonverbal sources (noises, engine sounds, sonar, music) Touch (pressure, pain, feeling the texture of a surface) Taste (food tasting, wine tasting)

(break down Information input: each element/attribute is rated on a 6 point scale from 0, does not apply to 5, very substantial)

44

Introduction: Position Analysis Questionnaire (PAQ)

Introduction: Position Analysis Questionnaire (PAQ)

The PAQ focuses on worker attributes rather than job tasks

For example, Arvey & Begalla conducted PAQ analyses of a home maker’s position and a police officer’s position, and found the profiles to be very, very similar

Because of that, it cannot be used for content validity*, (which is its main disadvantage)

If you use the PAQ you must conduct a criterion-related validity study or use validity generalization* (or job component validity)

The PAQ focuses on worker attributes rather than job tasks

For example, Arvey & Begalla conducted PAQ analyses of a home maker’s position and a police officer’s position, and found the profiles to be very, very similar

Because of that, it cannot be used for content validity*, (which is its main disadvantage)

If you use the PAQ you must conduct a criterion-related validity study or use validity generalization* (or job component validity)

*SO23 asks you to learn these two points

45

SO20A: Underlying Rationale of the PAQ (2 assumptions*)

SO20A: Underlying Rationale of the PAQ (2 assumptions*)

1. There are a limited number of generalized human behaviors**/attributes/job elements and all jobs can be characterized by the extent to which any particular job requires each of those job attributes/elements That is, there are a certain number of job attributes/elements that

are the “building blocks” for all jobs that exist (the PAQ authors identified 187), and each job is constructed from a set of these building blocks

Jobs can be analyzed by identifying which of the “building blocks” are required and the extent to which each is required in comparison to all other jobs that exist

1. There are a limited number of generalized human behaviors**/attributes/job elements and all jobs can be characterized by the extent to which any particular job requires each of those job attributes/elements That is, there are a certain number of job attributes/elements that

are the “building blocks” for all jobs that exist (the PAQ authors identified 187), and each job is constructed from a set of these building blocks

Jobs can be analyzed by identifying which of the “building blocks” are required and the extent to which each is required in comparison to all other jobs that exist

(**the authors refer to the “job elements” as generalized human behaviors, but they are not using the term the way we as Behavior analysts use it, so I prefer to talk about the job attributes or elements)

*The study objectives ask you to learn the two assumptions

46

SO20A: Underlying Rationale of the PAQ (2 assumptions)

SO20A: Underlying Rationale of the PAQ (2 assumptions)

2. Each job attribute/element is normally distributed across all possible jobs that exist Each of the 187 job attributes/elements has its own normal

distribution across jobs (I’ll show a slide of this in a moment) e.g., (a) the use of manually powered tools, (b) planning and

scheduling, (c) the amount of oral communication, etc.

2. Each job attribute/element is normally distributed across all possible jobs that exist Each of the 187 job attributes/elements has its own normal

distribution across jobs (I’ll show a slide of this in a moment) e.g., (a) the use of manually powered tools, (b) planning and

scheduling, (c) the amount of oral communication, etc.

47

SO20B: How are the important job attributes for each job determined by the PAQ?

SO20B: How are the important job attributes for each job determined by the PAQ?

Each job analyzed is compared to the normal distribution for each of the 187 job attributes/elements to determine how much of this job attribute/element is required in comparison to all of the other jobs that exist

Data are reported in terms of percentiles If a particular job attribute/element falls at the 90th percentile, it

means that 90% of all other jobs require less of this attribute than the current job and thus it is a very important job attribute/element for this job

What would it mean if planning and scheduling fell at the 10th percentile?

Each job analyzed is compared to the normal distribution for each of the 187 job attributes/elements to determine how much of this job attribute/element is required in comparison to all of the other jobs that exist

Data are reported in terms of percentiles If a particular job attribute/element falls at the 90th percentile, it

means that 90% of all other jobs require less of this attribute than the current job and thus it is a very important job attribute/element for this job

What would it mean if planning and scheduling fell at the 10th percentile?

48

Percentiles

The percentage of jobsthat require this attribute less than the target job position

Each job attribute isnormally distributed across all jobs

There is a normal distribution curve for each of the 187 attributesin the PAQ

49

Percentiles

Use of manually powered tools: 80th percentileThis job requires more useof manually powered toolsthan 80% of all jobs

Percentiles

Planning and scheduling: 10th percentileThis job requires moreplanning and scheduling than only 10% of all jobs

50

PAQ (NFE)PAQ (NFE)

Obviously, the PAQ analysis requires a very large data base of other jobs so you can compare your job to those jobs

This means if you use the PAQ, you must use the on-line data analysis of PAQ Services

See www.paq.com

Obviously, the PAQ analysis requires a very large data base of other jobs so you can compare your job to those jobs

This means if you use the PAQ, you must use the on-line data analysis of PAQ Services

See www.paq.com

51

SO23: PAQ and type of validity studySO23: PAQ and type of validity study

Because the PAQ does not provide task statements and KSAs that are operationalized, the PAQ cannot be used as the basis for content validity

If you use the PAQ, you must conduct a criterion-related validity study (job component notwithstanding) Concurrent Predictive

Because the PAQ does not provide task statements and KSAs that are operationalized, the PAQ cannot be used as the basis for content validity

If you use the PAQ, you must conduct a criterion-related validity study (job component notwithstanding) Concurrent Predictive

(redundant, but students have had trouble with this in the past; or job component - )

52

SO24: PAQ DisadvantagesSO24: PAQ Disadvantages First, the PAQ requires the reading level of a

college graduate It is a very complicated questionnaire, with different

rating scales used for different sections In most settings, you can’t just hand over the PAQ to

someone and ask him/her to complete it The best way to administer it is individually, with a

trained job analyst reading the questions and recording the answers

Also may be more reliable because incumbents may rate certain elements as being more significant than they are, just because they haven’t been exposed to a lot of jobs (and remember the PAQ analysis relies on a comparison to other jobs)

First, the PAQ requires the reading level of a college graduate It is a very complicated questionnaire, with different

rating scales used for different sections In most settings, you can’t just hand over the PAQ to

someone and ask him/her to complete it The best way to administer it is individually, with a

trained job analyst reading the questions and recording the answers

Also may be more reliable because incumbents may rate certain elements as being more significant than they are, just because they haven’t been exposed to a lot of jobs (and remember the PAQ analysis relies on a comparison to other jobs)

53

SO24: PAQ DisadvantagesSO24: PAQ Disadvantages Second, because the PAQ analyzes jobs in

terms of job elements and does not actually identify task statements, it cannot be used for some purposes that job analyses are typically used for Content validity Job descriptions Performance appraisals

Second, because the PAQ analyzes jobs in terms of job elements and does not actually identify task statements, it cannot be used for some purposes that job analyses are typically used for Content validity Job descriptions Performance appraisals

54

SO25: PAQ Major ADA DisadvantageSO25: PAQ Major ADA Disadvantage Since the passage of ADA, the fact that the PAQ cannot

be used to develop job descriptions has become a major disadvantage because ADA requires a job description for each and every job prior to selection (PAQ was developed in 1974, ADA was passed in 1990) How else is the organization and applicant going to know what

the essential job functions are and whether or not a disabled individual can perform them with or without reasonable accommodation?

Since the passage of ADA, the fact that the PAQ cannot be used to develop job descriptions has become a major disadvantage because ADA requires a job description for each and every job prior to selection (PAQ was developed in 1974, ADA was passed in 1990) How else is the organization and applicant going to know what

the essential job functions are and whether or not a disabled individual can perform them with or without reasonable accommodation?

55

SO25: Major ADA disadvantage, cont. but this slide NFE

SO25: Major ADA disadvantage, cont. but this slide NFE

Selection specialists usually do not develop job descriptions, rather that responsibility usually falls to the compensation specialists who have to justify salaries assigned to positions in terms of their tasks and responsibilities

Common organizational “silo” in organizations But, a job analysis/study can be used for both purposes Some organizations use the same procedure for selection

and compensation purposes, but very, very few

Selection specialists usually do not develop job descriptions, rather that responsibility usually falls to the compensation specialists who have to justify salaries assigned to positions in terms of their tasks and responsibilities

Common organizational “silo” in organizations But, a job analysis/study can be used for both purposes Some organizations use the same procedure for selection

and compensation purposes, but very, very few

56

SO26: PAQ AdvantagesSO26: PAQ Advantages 26A: First, it provides a standardized, quantitative

means for collecting job information that permits you to assess all jobs the same way, thus enabling you to compare across positions/jobs

26B: Delete (in the SO, I refer to validity generalization, not job component validity)

25C: Third, from a professional technical perspective it provides very valid and reliable job information There are extensive data attesting to its validity and reliability Makes it easy to defend the job analysis process if selection

tests are questioned in court 25D: Fourth, although not stated in the text, it has a

very important legal advantage, in that it has stood up in court and the EEOC has strongly recommended its use

26A: First, it provides a standardized, quantitative means for collecting job information that permits you to assess all jobs the same way, thus enabling you to compare across positions/jobs

26B: Delete (in the SO, I refer to validity generalization, not job component validity)

25C: Third, from a professional technical perspective it provides very valid and reliable job information There are extensive data attesting to its validity and reliability Makes it easy to defend the job analysis process if selection

tests are questioned in court 25D: Fourth, although not stated in the text, it has a

very important legal advantage, in that it has stood up in court and the EEOC has strongly recommended its use

(now, let’s look at its advantages; delete SO 26B)

57

PAQ Advantage (NFE)PAQ Advantage (NFE) Another advantage is that the PAQ has been shown to

produce very reliable and valid job information with as few as three job incumbents! I used to have students learn this as an advantage but

it tended to confuse them because: Usually you need to have more than three job incumbents

participate in a job analysis in order to have a legally and professionally acceptable representative sample (stratified by gender and ethnic/racial background)

You can’t use it for content validity, so you can’t use it for positions that have a small number of incumbents as a basis for justifying the job-relatedness of a selection procedure

Another advantage is that the PAQ has been shown to produce very reliable and valid job information with as few as three job incumbents! I used to have students learn this as an advantage but

it tended to confuse them because: Usually you need to have more than three job incumbents

participate in a job analysis in order to have a legally and professionally acceptable representative sample (stratified by gender and ethnic/racial background)

You can’t use it for content validity, so you can’t use it for positions that have a small number of incumbents as a basis for justifying the job-relatedness of a selection procedure

58

SO27: Selecting tests after a PAQ AnalysisSO27: Selecting tests after a PAQ Analysis Analyze PAQ data on-line using PAQ Services Based on the PAQ report, select tests

PAQ indicates tests in the GATB that are most relevant You cannot use these, but you can locate similar off-

the-shelf tests or ask PAQ to send you a list of similar tests

Empirically validate the tests Remember you cannot use the PAQ for content validity

Check the correlation for statistical significance

Analyze PAQ data on-line using PAQ Services Based on the PAQ report, select tests

PAQ indicates tests in the GATB that are most relevant You cannot use these, but you can locate similar off-

the-shelf tests or ask PAQ to send you a list of similar tests

Empirically validate the tests Remember you cannot use the PAQ for content validity

Check the correlation for statistical significance

59

Sample PAQ Report:

Senior Shipping and Receiving Clerk Sample PAQ Report:

Senior Shipping and Receiving Clerk

GATB Test

Mean

Score

Predicted

Validity

Coefficient

Probability

Of Use1

Predicted Low Score

General Intelligence 95.1 .19 .33 82.2

Verbal Aptitude 96.0 .10 .21 80.3

Numerical Aptitude 95.6 .18 .24 78.7

Spatial Aptitude 96.5 .19 .47 < 79.3

Clerical Perception 98.3 .12 .34 < 83.0

1 A “<“ indicates the test should be considered as a selection measure

Lists the nine tests in the GATB• These tests are restricted by the government• They are not available to private employers• Thus, PAQ services provides a list of alternative tests

60

Sample PAQ Report:



GATB Test

Mean

Score

Predicted

Validity

Coefficient

Probability

Of Use1

Predicted Low Score







Mean score on the exam that is predicted for your position. The higher the score, the more relevant the test is

61

Sample PAQ Report:



GATB Test

Mean

Score

Predicted

Validity

Coefficient

Probability

Of Use1

Predicted Low Score







This is the predicted validity coefficient for your position if you correlated the test scores with a measure of performance

• But remember, this is only predicted; you still must conduct an empirical validity study

62

Sample PAQ Report:



GATB Test

Mean

Score

Predicted

Validity

Coefficient

Probability

Of Use1

Predicted Low Score







This indicates whether the US Employment Service would be likely touse this test in a selection test battery• The higher the score, the more likely the USES would use it• Notice that the tests that are selected are not always the ones with the highest predicted validity coefficients• The tests are intercorrelated, thus you don’t get unique information from each one

(click)

63

Sample PAQ Report:



GATB Test

Mean

Score

Predicted

Validity

Coefficient

Probability

Of Use1

Predicted Low Score







This shows the potential cutoff or passing score for the test• It is set at 1 SD below the mean score listed in column 2• If used, approximately 1/3 of the current employees would have been eliminated when they applied for the position

• The 1/3 is based on the normal distribution (1/3 of scores fall below 1 SD below the mean)

64

PAQ: Alternatives to GATB TestsPAQ: Alternatives to GATB Tests General Intelligence

Wonderlic Personnel Test Adaptability Test Test of Learning Ability

Verbal Aptitude Personnel Tests for Industry - Verbal Short Employment Tests - Verbal Employee Aptitude Survey - Verbal

Numerical Aptitude Personnel Tests for Industry - Numerical Short Employment Tests - Numerical Employee Aptitude Survey - Numerical Arithmetic Index Flanagan Industrial Tests - Arithmetic Arithmetic Fundamentals Test

General Intelligence Wonderlic Personnel Test Adaptability Test Test of Learning Ability

Verbal Aptitude Personnel Tests for Industry - Verbal Short Employment Tests - Verbal Employee Aptitude Survey - Verbal

Numerical Aptitude Personnel Tests for Industry - Numerical Short Employment Tests - Numerical Employee Aptitude Survey - Numerical Arithmetic Index Flanagan Industrial Tests - Arithmetic Arithmetic Fundamentals Test

(cont.)

65

PAQ: Alternatives to GATB TestsPAQ: Alternatives to GATB Tests Spatial Aptitude

Revised Minnesota Paper Form Board Employee Aptitude Survey - Spatial Flanagan Industrial Tests - Assembly

Clerical Perception Employee Aptitude Survey - Visual Speed and Accuracy Short Employment Tests - Clerical Minnesota Clerical Tests - Names

Spatial Aptitude Revised Minnesota Paper Form Board Employee Aptitude Survey - Spatial Flanagan Industrial Tests - Assembly

Clerical Perception Employee Aptitude Survey - Visual Speed and Accuracy Short Employment Tests - Clerical Minnesota Clerical Tests - Names

Note that the names of the tests vary even though they measureapproximately the same thing.

It is also true that even though tests may be named the same, they may measure different things.

(last slide)

66

That’s All FolksThat’s All Folks

Questions on U6?Questions on U6?

PSY 6430 Unit 6

Documents

Transcript of PSY 6430 Unit 6