1 icfi.com | Social Security Administration Call Order 0001: Job Analysis Methodology Briefing of...

1icfi.com |

Social Security Administration

Call Order 0001: Job Analysis Methodology

Briefing of Final Project Results

OIDAP Quarterly Meeting

September 21, 2011

Lance Anderson, Ph.D.

Brian Cronin, Ph.D.

2icfi.com |

• Introductions/Overview of Team– SSA

– ICF

– Subcontractors

• Introduction to Project and Purpose• Project Methodology• Recommendations

– Job Analysis Procedures

– Job Analysis Models

• Summary– Overarching Recommendations

– Potential Next Steps

• Questions?

Agenda

3icfi.com |

Introductions

4icfi.com |

SSA • Debra Tidwell-Peters• David Blitz• Michael Dunn• Elizabeth Kennedy• Mark Trapani

Overview of Team

ICF International • Brian Cronin, Ph.D. • Lance Anderson, Ph.D. • Beth Heinen, Ph.D. • Jessica Jenkins, MPhil• Allison Cook, M.S.• Daniel Fien-Helfman

Subcontractors• Paul Davis, Ph.D.• Kelly Day, OTD• Len Matheson, Ph.D.

5icfi.com |

Introduction to Project and Purpose

6icfi.com |

• SSA is developing new occupational information system (OIS) tailored specifically to SSA’s disability programs and adjudication process.

– OIS will replace the Dictionary of Occupational Titles (DOT) and companion volumes, including the SCO and RHAJ.

– To develop OIS, SSA needs detailed methodology and strategy

for analysts to perform job analysis throughout U.S. labor market.

– Purpose of Call 0001 was to perform the research needed to identify useful features of existing practices to support development of SSA’s job analysis methodology.

Introduction to Call Order 0001

7icfi.com |

Project Methodology

8icfi.com |

Project Methodology

9icfi.com |

• Overarching Goal: Summarize wealth of information on job analytic procedures across disciplines to allow SSA decision makers to quickly understand and evaluate various job analysis practices.

• To ensure consistency, we use the following nomenclature:

– Project Method(ology) – Refers to steps taken to conduct research for this call order

– Job Analysis Method(ology) – Refers to ultimate SSA job data collection process developed to address OIS needs

– Practice – Refers to all job analysis approaches, models and procedures, identified through this call order

– Model – Refers to an established, ‘off-the-shelf’ job analysis approach identified through this call order

– Procedure – Refers to a data collection technique identified through this call order

Call Order 0001 Terminology

10icfi.com |

Recommendations on Job Analysis Procedures

11icfi.com |

• Recommendation: Effective and cost-efficient starting point for almost all job analysis. Should be included in SSA’s ultimate data collection methodology.

• Potential Usage for SSA: Analysts are able to become more familiar with the target job.

• Potential Challenges: Dependent upon the usefulness and availability of source documents. Should only be used in conjunction with other data collection procedures.

• Outcome: Knowledge gained can be used to inform development of subsequent data collection techniques (e.g., interviewing).

Procedure: Review of Written Materials

12icfi.com |

• Recommendation: Use for jobs that include more manual, less cognitive tasks. Should be included in SSA’s final data collection methodology.

• Potential Usage for SSA: Useful for collecting detailed information about job tasks, equipment/materials used, and work environment; does not rely solely on testimony of incumbents.

• Potential Challenges: Can be costly and time-consuming; may not be appropriate for all jobs (e.g., highly cognitive jobs).

• Outcome: Should be used in combination with other data collection procedures to determine prevalence of work activities observed.

Procedure: Job Observation

13icfi.com |

• Recommendation: Use when gathering data from geographically dispersed incumbents. Should be further considered for inclusion in SSA’s ultimate methodology.

• Potential Usage for SSA: Can be effective and efficient means of collecting data from numerous incumbents; useful for assessing prevalence of work activities/; can make comparisons across jobs.

• Potential Challenges: May be costly to develop and administer; Some threats to validity, such as inflated ratings, lack of understanding rating elements, lack of effort in completing survey.

• Outcome: Other data collection procedures should be used to supplement the data collected especially when more detail is needed.

Procedure: Survey

14icfi.com |

• Recommendation: Use when job is more complex and needs additional clarification or when comparison is needed (as long as highly structured). Should be incorporated into SSA’s ultimate data collection methodology.

• Potential Usage for SSA: Allow analysts to collect detailed job information through the direct questioning of incumbents.

• Potential Challenges: Can be time-consuming and costly to prepare protocol, coordinate participants’ schedules, and arrange travel (for face-to-face interviews).

• Outcome: Analyst should undergo training on interviewing skills; analysts should conduct multiple interviews to gain full value of this procedure.

Procedure: Structured Interviews

15icfi.com |

• Recommendation: May be more time-efficient and cost-efficient than interviews, but we recommend that interviews be used instead of focus groups whenever possible to allow for clarification questioning.

• Potential Usage for SSA: Allows for discussion with multiple incumbents at one time; useful for idea generation or other data collection that depends on interaction to elicit information.

• Potential Challenges: Group dynamics can influence participant responses; schedule coordination can be difficult; specialized training needed for job analysts to be skilled facilitators.

• Outcome: Participants should be provided with an agenda, ground rules, and background information prior to the session.

Procedure: Focus Groups

16icfi.com |

• Recommendation: May be valuable for jobs requiring manual labor given the disability determinations SSA must make. Depending on SSA’s ultimate construct model and data collection instrument, should include instrument measurement of physical demands (i.e., tools that measure PDs) but with limited use.

• Potential Usage for SSA: Advantages include precise nature of the collected data, high reliability, high validity, and data that are typically easy to aggregate.

• Potential Challenges: Can be resource intensive to administer and require specialized training; can be intrusive to incumbents.

• Outcome: Must have technical expertise in the use of specific measurement tools selected.

Procedure: Physical Demands Measures

17icfi.com |

Recommendations on Job Analysis Models

18icfi.com |

Model: AET

Examples of Effective Features

• Use of descriptors that isolate specific types of physical effort

• Use of scales that focus on frequency, duration, and significance

• Use of examples to assist in coding level of demand

Example Limitations for SSA Context

• Significant time and labor commitment to develop career ladders

• Long term commitment to upkeep of career ladders

The AET involves conducting an observation and interview to complete an ergonomic questionnaire.

19icfi.com |

Model: Common Metric Questionnaire (CMQ)


• Matrix structure of the questionnaire • Computerized interface to allow for

effective use of the questionnaire

• Use of behavioral and observable descriptors that are easy for incumbents and supervisors to rate


• Does not include a comprehensive set of descriptors at a broad level

• Some respondents might not have the access to a computer to use the computerized interface

• CMQ-like items on the OIS would need to be continually updated

The CMQ collects data via a survey administered directly to incumbents and/or their immediate supervisors.

20icfi.com |

Model: Cognitive Task Analysis (CTA)


• Use of structured observation and interviews together with completion of a structured tool by the analyst

• Identification of the various types of knowledge needed to do the job


• Tends to lack the detailed information needed by SSA on various physical abilities

• Would require extensive training of analysts

The CTA approach involves using a variety of data collection procedures to ultimately identify the cognitive processes underlying a job with a particular focus on the processes that distinguish an expert from a novice.

21icfi.com |

Model: Fleishman Job Ability Requirement Scales (F-JAS)


• Generally well supported by research

• Use of level scales anchored with observable behaviors


• Lacks some generalizable physical abilities constructs that are important to SSA

• Tends to provide details on the variation of jobs at the high end of many abilities, whereas SSA might be more interested in information about the variation in the jobs at the low end of abilities

The primary data collection procedure for the F-JAS model involves the administration of the Ability Requirements Scales to collect data on 52 types of abilities.

22icfi.com |

Model: Functional Job Analysis (FJA)


• Inclusion of work context and worker environment variables

• Use of procedures that can be easily trained

• Structured framework and structured protocols build validity


• Lacks standardization on important issues such as how jobs are sampled, how interviews are conducted, and how many interviews are conducted

• The DOT scales lack detail on cognitive abilities and interpersonal skills

The FJA approach gathers a variety of different types of job analysis data typically collected via interview and observation but may also include other data collection procedures.

23icfi.com |

Model: Job Element Model (JEM)


• Approach to conducting interviews and observations to gather job specific information provides data for understanding the job


• Experts previously rated this model low in terms of reliability and standardization

• Although a low cost approach, it involves a significant amount of time to administer

JEM focuses on the human attributes required for superior performance on the job and collects data via focus groups, interviews, and surveys.

24icfi.com |

Model: O*NET


• The hierarchical arrangement and use of the content domain

• A nationwide database supported and maintained by an external entity


• Some items will likely have low reliability relative to other instruments because constructs are not observable

• Tends to focus on differentiating high-tech jobs as opposed to low skill jobs that are usually the focus of disability claims

O*NET was developed using a job analysis approach that focuses primarily on surveys, with supplementary use of interviews and reviews of written material.

25icfi.com |

Model: Position Analysis Questionnaire (PAQ)


• Use of an instrument that focuses on generalizable work activities provides data for cross-job comparison

• Focus on observable behaviors ensures greater verifiability of the findings


• Constructs at a level of abstraction that may not provide a clear picture of the job.

The PAQ is an existing job analysis model that uses a standardized 195-item instrument to collect data, which is typically completed by a job analyst based upon data collected in job observations and interviews.

26icfi.com |

Model: Task Inventory


• Method of conducting interviews and observations to gather job specific information provides data for understanding the job


• No generalizable constructs or scales to allow for cross-job comparison

The Task Inventory approach involves collecting data through procedures such as review of written materials, job observation, interviews, and surveys to ultimately develop a list of task descriptions.

27icfi.com |

Model: Threshold Traits Analysis (TTA)


• The 33 traits include a parsimonious and simply worded set of constructs that might provide an effective perspective for sorting and locating jobs

• In general, the TTA is well supported by research


• SSA would need more detail than is provided via the 33 trait focused scales

• In our judgment, the TTA rating tool is not appropriate for use as incumbent/supervisor survey. It should only be used by trained analysts

The TTA collects worker trait, job demand, and job function data using data collection procedures such as review of written materials, job observations, interviews, and surveys.

28icfi.com |

Summary: Overarching Recommendations and Potential Next Steps

29icfi.com |

• Identify Work Taxonomy and Constructs to be Measured– Measuring different constructs may necessitate different data collection

procedures

• Data Should be Collected and Stored Using a Computerized System or Online Application/Tool– Provides a centralized location for data collection and minimize potential

security issues/concerns

• Need to Determine Factors that are Most Important and Consider Job Analysis Practices Accordingly

• Fully conceptualize multiple prototypes of integrated systems that SSA might use and compare the systems side-by-side – Integrated system= Occupational analysis system

Recommendations and Potential Next Steps

30icfi.com |

Comparing Prototype Integrated Systems (i.e., Occupational Analysis Systems) Side-by-Side

31icfi.com |

• Full Methodology Must Include a Comprehensive Set of Procedures that Include Guidelines for Maintaining Data Security and Confidentiality

• Features of Existing Job Analysis Models Should be Adapted for SSA’s Specific Purposes and Data Needs– Additional job analysis questions or more precise questions may need to

be added to an existing questionnaire

• Need Pilot Testing to Ensure that the Final Methodology Meets SSA Objectives– Ensures the final set of combined data collection procedures and/or

models appropriately measure the desired characteristics

Recommendations and Potential Next Steps

32icfi.com |

Questions?

• For more information, please contact:

Dr. Brian Cronin

Senior Manager

ICF International

(512) 388-3389

[email protected]

Dr. Lance Anderson

Vice President

ICF International

(703) 934-3000

[email protected]

1 icfi.com | Social Security Administration Call Order 0001: Job Analysis Methodology Briefing of...

Documents

Transcript of 1 icfi.com | Social Security Administration Call Order 0001: Job Analysis Methodology Briefing of...