Physical Ability Testing: How to Measure an Employee’s Ability to Perform the Physical Aspects of a Job

© Copyright 2009, Fire & Police Selection, Inc.

All Rights Reserved

Visit BCGi Online

While you are waiting for the webinar to begin, Don’t forget to check out our other training

opportunities through the BCGi website. Join our online learning community by

signing up (it’s free) and we will notify you of our upcoming free training events as well as other information of value to the HR community.

www.BCGinstitute.org

HRCI Credit BCG is an HRCI Preferred Provider CE Credits are available for attending

this webinar Only those who remain with us for at

least 80% of the webinar will be eligible to receive the HRCI training completion form for CE submission

About Our Sponsor BCGi is sponsored by Biddle Consulting

Group, Inc.

55

About Our Sponsor: BCG• Assisted hundreds of clients with cases involving Equal

Employment Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense)

• EEO Litigation Support / OFCCP (federal contracting) Audit Support

• Compensation Analyses / Test Development and Validation• Published: Adverse Impact and Test Validation, 2nd Ed., as a

practical guide for HR professionals• Editor & Publisher: EEO Insight an industry e-Journal • Creator and publisher of a variety of productivity Software/Web Tools:

– OPAC® (Administrative Skills Testing)– CritiCall® (9-1-1 Dispatcher Testing)– AutoAAP™ (Affirmative Action Software and Services)– C4™ (Contact Center Employee Testing)– Encounter™ (Video Situational Judgment Test)– Adverse Impact Toolkit™ (free online at www.disparateimpact.com)– AutoGOJA® (Automated Guidelines Oriented Job Analysis®)

IndustryLeader

www.Biddle.com

Contact Information

Stacy Bell, M.S.SBell@biddle.com

Biddle Consulting Group, Inc.193 Blue Ravine, Ste. 270Folsom, CA 956301-800-999-04381-800-999-0438www.biddle.com

Questions?Should you have any questions during the webinar you have two options:

Send an email to stacy@fpsi.com and we will try to answer it during the webinar.

Ask a question through the GoToMeeting screen console and we will try to address it at the end of the webinar.

www.BCGInstitute.org

Should you have any questions regarding OFCCP Audits, Testing and Selection, or Statistical Analysis, visit:

Should you have any questions regarding pre-employment testing and selection in the public service industry, visit:www.FPSI.com

Physical Ability Testing: How to Measure An Employee’s Ability to Perform the Physical Aspects of a Job

© Copyright 2009, Biddle Consulting Group

All Rights Reserved

Defensibility, Fairness, Productivity

What is a PAT?

PATs in Court

Important Issues For Using PATs

Developing vs. Transporting a PAT: Issues to Consider

Presentation Overview

Developing and Validating a PAT: Case Study

Simulations

Physical Components

The Entire Job

What is a PAT?

Specificity

An assessment procedure that measures a candidate’s ability to successfully perform the critical “physical” aspects of a job at a level that minimizes risks to safety of self and others.

Examples of successes and failures

PATs in Court

Court Reviewed PATsCharacteristics of Winning Tests

Job Analyses

Used Subject Matter Experts (SMEs), incumbent firefighters, to identify the knowledge, skills, and abilities (KSAs) needed for satisfactory performance of the designated job duties (6, 9, 12).

Included frequency and importance ratings of the KSAs identified by Subject Matter Experts (12).

Obtained information about the types of equipment used in the course of performing the job of firefighter (12).

Validation Procedures Both content and criterion-related validity studies

when ranking (3, 5, 12). Content validity when using only a cut-off score (9).

Ranking Procedures Which were utilized only when a direct relationship

between test scores and job performance could be demonstrated through criterion-related validity. Individuals who performed better on the test performed better on the job (3, 6, 12).

Court Reviewed PATsCharacteristics of Winning Tests

Cut-Off Procedures

Which were documented and justified to the court (3).

Court Reviewed PATsCharacteristics of Winning Tests

Fatal Flaws of Losing Tests

Job Analyses

Did not exist or were not described to the court (4,7). Subject Matter Expert sample too small: 35 of 11,616

incumbents (1). No inter-rater correlations for Subject Matter Experts (1). Failed to include frequency and importance data for the

KSA’s designated by the Subject Matter Experts (11). Did not specify exact KSA’s necessary to perform groups of

job duties (11). Did not define the amount of physical strength needed, or the

extent of physical exertion required to properly perform the designated job duties (10).

No link of test content to job (1).

Fatal Flaws of Losing Tests Validation Procedures

Did not exist (2, 7, 11). Sought to demonstrate a relationship between test scores

and training performance instead of job performance (4).

Ranking Procedures Were not supported by procedures showing a linear

relationship between the job duties and the test events (such as a criterion-related validity study) (1,5,8).

Cut-Off Procedures Used one standard deviation above the mean to set the

passing score, a procedure which will cause the cut-off to fluctuate with each test administration (8).

Cut-Off Procedures Used subjective scoring and cut-offs (8). No consideration given to the results of incumbent testing

(8). No consideration given to the maximum time which could be

allowed (8).

Test Events Did not accurately represent the KSA’s identified in the Job

Analysis (5, 10). That had been used by other cities but had never been

validated (10). Tested techniques that could and should have been taught

during training (8). City rejected test developer’s advice and substituted non-

validated events (8).

Fatal Flaws of Losing Tests

1. Berkman v. City of New York. March 4, 1982 (DC NY) 30 EPD 33,320 – 536 FSupp 177.

2. Berkman v. City of New York. March 29, 1983 (CA-2) 31 EPD 33,511 – 705 F2d 584.

3. Berkman v. City of New York. February 17, 1987 (CA-2) 42 EPD 36,902 – 812 F2d 52.

4. Blake v. City of Los Angeles. May 3, 1979 (CA-9) 19 EPD 9,251 – 595 F2d 1367.

5. Brunet v. City of Columbus. May 13, 1986 (DC OH) 41 EPD 36,498 – 642 FSupp 1214.

6. Brunet v. City of Columbus. July 28, 1993 (CA-6) 62 EPD 42,479 – 1 F3d 390.

Case List:Physical Ability Testing

Case List:Physical Ability Testing7. Cohen v. West Haven Board of Police Commissioners.

December 23, 1980. (CA-2) 24 EPD 31,440 – 485 FSupp 958.

8. Evans v. City of Evanston. July 27, 1989 (CA-7) 51 EPD 39,261 – 881 F2d 382.

9. Hardy v. Stumpf. April 13, 1978. (CA SCt) 16 EPD 8,249 – 576 F2d 1342.

10. Harless v. Duck. April 23, 1980 (CA-6) 22 EPD 30,871 – 619 F2d 611.

11. Legault v. A Russo. February 10, 1994 (DC NH) 65 EPD 43,302 – 842 FSupp 1479.

12. Zamlen v. City of Cleveland. June 11, 1990 (CA-6) 53 EPD 40,004 – 906 F2d 209.

13. Cleghorn v. Herrington. March 27, 1987 (CA-9) 42 EPD 36,957 – 813 F2d 992.

14. (Not cited as the case was tried under age discrimination and involved a non-fire/ police occupation.)

15. Hail v. White. August 31, 1973 (DC CA) 8 EPD 9,637.

16. (Not cited as the case was brought under the 14th Amendment.)

Case List:Physical Ability Testing

Example using the Fire & Police Selection, Inc. content valid

work sample PAT

PATs: A Case Study

FPSI’s Work Sample PAT Developed in 1996 by 41 departments in Southern

California including the City of Los Angeles, the County of Los Angeles, and Orange County

Over 230 firefighters completed validation surveys and normed the test for identification of an appropriate cutoff time

The test has been successfully transported into over 70 fire departments across the country

Never been challenged

Custom Development

Transport Existing Test

Two Ways to Validate a PAT

Transporting a PAT Randomly sample incumbents to complete surveys and

participate in the norming session.

Incumbents complete Job Analysis and Critical Events Surveys.

One (1) Fire Captain or Training Officer completes the Tools & Equipment Survey.

Department builds props based on Test Administration specifications.

Department has two proctors independently complete the “Site Certification Form.”

Department runs selected incumbents through the PAT and records times.

Department and FPSI determine an appropriate cut-off time.

Department runs candidates through the course.

Transporting a PAT

FPSI determines #’s needed

1-2 hours to complete

30 minutes to complete

Run all selected FFs through course and record their time

Validated test ready for use!

4-6 weeks from beginning to end!

Random Sample

Department provides FPSI with a list of all incumbents who perform fire suppression duties by department and shift.

FPSI randomly selects those incumbents who will complete Job Analysis Surveys, Critical Events Surveys, and participate in the norming of the PAT.

Listed below are the number of incumbents required to complete surveys or participate in the norming session. The following sample sizes are based on an 90% confidence interval of +/- 10%.

Department Size# of Incumbents

Required

10 915 1250 2990 39200 51500 60

Surveys Job Analysis Survey

Completed by selected fire suppression personnel Used to gather frequency, importance, and performance differentiating

data

Critical Events Survey

Completed by selected fire suppression personnel Used to gather specific information pertaining to how 70 physical duties

are performed on the job

Tools & Equipment Survey

Completed by one Fire Captain or Training Officer Used to gather specific information pertaining tools and equipment used

on the job

Job Analysis Sample

Job Duties

Incumbents provide frequency, importance, and performance differentiating ratings.

The duties have been provided in 5 categories: Rescue and First Aid, Deploying Hose and Pumping, Laddering, Ventilation and Forcible Entry, and Extinguishing Fire.

Rescue and First Aid

Uses extrication equipment such as shovels, picks, and other equipment to free victims trapped, injured, or unconscious in areas such as tunnels, pipes, sewers, excavations, and/or structures.

Cuts or pries open vehicles and/or machinery to free persons trapped inside, using tools such as rotary and chain power saws, prying tools, come-alongs, pneumatic cutting tools, port-o-powers, and/or hydraulic rescue tools.

Critical Events Survey SampleRemoves pre-connected dry hose line from transverse hose bed, extends to designated area, waits for hose line to become charged, and advances charged hose line through windows, doors, and/or around bends to attack fire.

1. What type of hose line do you typically use in this duty?

Diameter (Circle One):

1.5 inch 1.75 2.02.5 3.0 4.0

5.0

Comments: ____________________________________

2. I typically advance the DRY hose line _____ feet when performing this duty, with a range of _____feet to ______ feet.

IMPORTANT! If there is more than one type (or size) of any item listed below, please describe all types or sizes used by your department.

When a weight is requested, please weigh the equipment. Do not estimate or guess about the weight of the equipment.

BREATHING APPARATUS

Weight(s): _________________________________________

HOSES

Internal/Supply Lines

Dry weight(s) per 50-foot section: _________________________

Tools & Equipment Survey Sample

Test Administration Handbook Sample

PAT TEST SITE CERTIFICATION FORM

DIRECTIONS: Listed below are descriptions of necessary equipment and event requirements essential for maintaining the integrity of the Physical Ability Test (Version 1.4). Please have two (2) proctors independently review each statement and initial the box if all of the requirements are met at the site. Once you have initialed all of the boxes, please answer the questions at the bottom of this form, and fax (916.294.4240) or mail this form to: 193 Blue Ravine Rd., Ste. 270 Folsom, CA 95630.

Event 1: Dry Hose Deployment

The dry hose line is 150 feet. The hose line is 1.75-inch and made from synthetic (not

cotton jacketed) material. The hose is laid out in a "Lazy W" shape.

Site Certification Form Sample

The Physical Ability Test Consists of Eleven (11) Continuously-Timed Events

FPSI PAT: A Detailed Look Under The Hood

Event 1:Dry Hose Deployment

Candidate deploys 1.75-inch supply line for 150 feet.

Hose is deployed around two obstacles.

Each 50-foot section weighs 20 pounds.

Event 2: Charged Hose Deployment

Candidate deploys pre-connected 1.75-inchcharged hose line 70 feet.

Hose is deployed through a simulated doorway.

Hose is deployed progressively lower for a distance of 38 feet.

Candidate raises the fly of a 35-foot aluminum extension ladder.

Event 3:Halyard Raise

Event 4:Roof Walk

Candidate ascends and descends an 11-foot distance on a roof ladder.

Candidate must walk on the rungs of the ladder.

Candidate carries a 20-pound chain saw.

Event 5:Attic Crawl

Candidate crawls a distance of 21 feet across a simulated attic area.

Candidate carries a simulated flashlight in his or her hands.

Event 6:Roof Ventilation

Candidate stands on a pitched roof.

Candidate strikes roof 30 times with sledge hammer.

Candidate carries or drags a dummy for a total of 26 feet.

Dummy weighs 154 pounds.

Event 7:Victim Removal

Candidate removes a 24-foot aluminum extension ladder from mounted hooks.

Candidate carries ladder 54 feet.

Candidate replaces ladder on mounted hooks.

Event 8:Ladder Removal/Carry

Event 9:Stair Climb with Hose

Candidate climbs four flights of stairs (a total of 63 steps) with a49-pound hose bundle on his or her shoulder.

The hose bundle is composed of 100 feet of 1.75-inch hose line.

Candidate crawls on the platform of the fourth floor for a distance of 60 feet.

Candidate returns to ground floor with hose bundle.

Event 10:Crawling Search

Candidate hoists 100 feet of extended 1.75-inch hose line up to a third floor balcony.

Event 11:Hose Hoist

End of Test

Candidate returns to the ground floor.

Test time stops.

Maximizing Legal Defensibility While Selecting the Best Candidates for the Job

Important Issues For Using PATs

How Should A Physical Ability Test Be “Used”?

Pass/Fail Banding Ranking

Psychometric Issues to Consider

Reliability (e.g., Stability) Variance Mean Distribution Characteristics Testing Effect Applicant Data

Setting the Standard

Standard is not relative to applicant pool, but must be anchored by incumbent performance. -- Evans v. City of Evanston [51 EPD 39, 261]

“Normal expectations of acceptable proficiency in the workforce”[GUIDELINES, SEC. 5(H)]

SME Opinions

The Pass/Fail Drawback

The “minimum” score is considered equal with the “superior” score.

Need a .30 OR BETTER validity coefficient (to ensure practical significance): Boston Chapter N.A.A.C.P, Inc.

v. Beecher [8 EPD, 9, 678] Brunet v. City of Columbus

[62 EPD 42, 479] Clady v. County of Los Angeles

[38 EPD 35, 516] Zamlen v. City of Cleveland

[49 EPD 38, 829]

Support for Ranking/Banding

Strict, Top-Down Ranking Ordering

SUPPORTABLE? Yes

A GOOD IDEA? Probably not

Tests cannot discern between small score differences Adverse impact

Applicant Score

Tom 100Stacy 100Bob 100

Frank 100Julie 99

Rozanne 99Mark 98Luke 98Henry 97Paul 97Peter 96

Rebecca 96Alyssa 95

Matthew 94John 93

Annette 93Ray 92

Thomas 91Julissa 90

Cutoff Options Ranking assumes one applicant is

reliably more qualified than the other

Banding considers the unreliability of the test battery and “ties” applicants

Pass/fail cutoffs treat all applicants as either “qualified” or “not qualified”

Requirements for Ranking/Banding

Demonstrate That: Better Scores = Better Job Performance

Criterion-Related Validity .30 or better correlation

Content Validity “If a user can show, by a job analysis or otherwise, that a higher

score on a content valid selection procedure is likely to result in better job performance, the results may be used to rank persons who score above minimum levels…the selection procedure should measure those aspects of performance which differentiate among levels of job performance.”[Guidelines, Sec. 14(C)9]

Comparison of Score Uses

Factor Ranking Banding Pass/Fail Cutoffs

Validation Requirements

High Moderate Low

Adverse Impact High Moderate Low

Defensibility Low High High

Litigation "Red Flag"

High Moderate Low

Cost to do so Defensibly

High Moderate Low

Applicant Flow Restrictive/Controllable

Moderate/Controllable

High

Development Time to do so Defensibly

High Moderate Low

Aiming for the Best Candidates

When criterion-related validity is found, the “usefulness” of a test is determined by at least three factors:

1. “Base Rate”: What proportion of test takers are “already” successful in the test-taker pool?

2. “Selection Ratio”: What proportion of test takers will pass the cutoff?

3. “Correlation Value”: What job variance can be accounted for within the test variance?

Developing vs. Transporting a PAT

Considering The Costs

Developing vs. Transporting Costs

Developing a Physical Ability Test from beginning to ends typically costs between $40,000 to $60,000.

Transporting or “adopting” FPSI’s Physical Ability Test to own cost between $16,000 and $22,000 (depending on department size).

In 2008, FPSI reduced this cost by 50%.

Consortium, or group testing, can also significantly reduce the department’s costs and benefit the applicants by reducing the likelihood that they will have to take the same test for several different agencies.