Bottom-line Evaluation:Measuring Results from

Training and Performance Improvement

Tuesday May 19, 2003

8:00-9:30 a.m.

ASTD International Conference

Donald J. Ford, Ph.D., CPT

Training Education Management

Redondo Beach, CA 90277

BiographyDonald J. Ford, Ph.D., C.P.T., is president and owner of Training Education Management, a training and performance improvement consulting firm in Redondo Beach, California. Dr. Ford also teaches graduate courses in Human Resource Development for Cal State Northridge and Antioch University, L.A. His consulting clients include: ASTD, Toyota, Nissan Motors U.S.A., Rockwell International, U.S. Marines Corps, Orange County Transportation Authority, Glendale Memorial Hospital, and National Education Corp., among others. He has worked in the field of human resource development for eighteen years, including training management positions at Southern California Gas Company, Magnavox, Allied-Signal and Texas Instruments. He possesses another eight years of experience in public education at the secondary, adult and college levels.

He holds a B.A. and M.A. in history and a Ph.D. in education, all from UCLA. He has published 35 articles and three books on topics in training, education and business. He edited a book entitled, In Action: Designing Training Programs published by ASTD in 1996. His third book entitled, Bottom-Line Training: How to Design and Implement Programs That Boost Profits, was released by Gulf Publishing in July, 1999 and forms the basis for this presentation. Dr. Ford has presented at the ASTD International Conference four times, and at ASTD L.A. conferences and workshops.

He may be reached at:

1874 Pacific Coast Highway, Suite 205

Redondo Beach, CA 90277

Telephone/Fax: 310-316-2240

E-mail: donaldford@earthlink.net

Web: www.trainingeducationmanagement.com

Bottom-Line Training Design Model

• Needs Analysis• Performance Analysis• Needs assessment

• Job/Task Analysis• Learner Analysis• Context Analysis• Skill Gap Analysis

AnalyzeAnalyze

•Objective• Deliverables/Strategies• Budgets/Schedules• Project management• Blueprints/Prototypes

DesignDesign

• Materials• Tests/Assessments• Quality Control• Production

DevelopDevelop• Train the Trainer• Classroom Delivery• Non-Classroom Delivery

ImplementImplement

• Formative Evaluation• Reactions/Learning• Transfer of Training• Business Results

EvaluateEvaluate• Learning• Performance• Financial• Strategic

ResultsResults

Work Environment

Level 4:Results

• Performance• Financial

Level 4:Results

• Performance• Financial

Level 3:Job Behavior

• Learner• Organization

Level 3:Job Behavior

• Learner• Organization

Training Environment

Level 1: Reactions

• Learner• Client

Level 1: Reactions

• Learner• Client

Level 2:Learning

• Learner• Organization

Level 2:Learning

• Learner• Organization

Kirkpatrick Evaluation Model

Learning Event

Typical Evaluation Points

AnalyzeDesign/Develop

Implement

Evaluate

Evaluation Design• Tests • Surveys• Formative Evaluation

•Reactions• Pre-Test•Post-Test

•Skills Transfer •Performance•Business Results •Data Analysis •Evaluation Reports

Baseline Data:• Learners Performance

• Skill Gaps• Business Measures

Two Types of Evaluation

Formative Improve the process

& the program Focus on process

Measures: Acceptance Satisfaction Quality

Summative Sum up the impact,

Go-No Go Focus on results

Measures: Performance change Business results ROI

HPI Evaluation Model

BusinessAnalysis

PerformanceAnalysis

CauseAnalysis

InterventionSelection

InterventionImplementation Evaluation

Evaluation Baseline

Formative Evaluation

Summative Evaluation

BusinessGoals

PerformanceGaps

RootCauses

Solutions

ProjectManagement

ChangeManagement

BusinessMeasures

PerformanceMeasures

AnalysisEvaluation

EvaluationDesign

CustomerSatisfaction

Performanceand ROI

HPIOutputs

Evaluation Process

HPI Process

EvaluationOutputs

Figure 21-1: One -Shot Case Study Design

T M

T = training M = measurement

Figure 21-2: One-Group Pre-Post Evaluation Design

M = measurement (1 = Pre, 2 = Post) T = training

M1 M2T

Figure 21-5: Pre-Post Control Group (Classic Experimental Design)

M1 M2

Group 1: R M1 M2T

Group 2: R

R = randomize T = trainingM = measurement (1 = Pre, 2 = Post)

Figure 21-7:Interrupted Time Series Design

Examples: Accident rates pre and post safety training Sales revenues pre and post sales training Error rates pre and post quality training Turnover rate pre and post management training

M2 M5TM1 M6M3 M4

M1- 6 = measures at regular intervals T = training

A Training Parable

One day in Training Land, the Senior Vice-President of the company’s largest division came to request training for all of his 2,000 employees. His reason is that the company is about to launch a new line of products and services and he fears employees will not be able to deliver these successfully without being retrained. Training needs to start in six months.

The content must include: product features and benefits, sales techniques, customer service and a customer information system.

Your assignment is to design an evaluation plan for this request.

Figure 24-1:Bottom-Line Impact of Training

Strategic GrowthStrategic Growth

RevenuesRevenues

SavingsSavings

TrainingTraining

Cost AvoidanceCost Avoidance

Monthly Sales

Sales Training Impact

Product 1/02 2/02 3/02 4/02 5/02 6/02 7/02 8/02 9/02 10/02 11/02 12/02

Us 4% 3% 5% 4% 3% 5% 7% 6% 7% 8% 6% 6%

#1 Cp 46 47 43 44 46 44 43 42 40 39 41 42

#2 Cp 20 21 19 22 20 21 19 20 19 21 20 21

#3 Cp 15 14 13 14 15 13 12 12 14 14 15 14

#4 Cp 8 10 11 10 8 7 9 10 11 10 9 10

#5 Cp 7 7 9 6 7 10 10 10 9 7 8 7

New Sales Training

What was the impact of the sales training on market share? If the total annual market for these drugs is $10 million, what was

the financial benefit of the sales training?

Cancer Drug Sales as % of Market Share

Figure 23-4: Productivity Gains Caused by Workplace Literacy Training Program

Productivity Level (% of Ideal)

Training Group Monthly Productivity Average

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Training Ends= Control Group Productivity Level

What conclusions can you draw from from this chart about

the impact of the training?

If each percentage increase in productivity was worth an average of $5,000 what would be the total value (benefit) of this training?

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Training Begins

Training Return on Investment (ROI)

Total benefits ÷ total costsExample:

$ 750,000 total benefits

$ 187,500 in total costs

750,000/187,500 = 4:1

Company receives $4 in return for each $1 spent or a return of 400%

ROI Example $$

Category Costs

Development $250,000

Direct (delivery) $250,000

Indirect (participant salaries) $500,000

Benefits

Direct (sales increase) $750,000

Indirect (lower turnover, absenteeism, higher productivity) $750,000

What is the ROI of this training?

Evaluating Behavior, Performance & Skill Transfer

Success factors: Learned new skills Desire to change Supportive work climate Supportive supervision Lack of barriers Rewards and incentives

Performance Measurement Methods

Action planning Pre-post surveys Pre-post interviews Job observation Performance documentation

Output Quality Cost Time Behavior

Figure 23-1: Sample Action Plan for Customer Service Training

Summary of Skills LearnedIn this course, I learned the following new customer service skills:

- How to greet customers. - How to identify customer needs.- How to locate customer account information in the customer database.- How to make simple changes to an account, like updating the customer’s name,

address, applying payments, waiving late charges and fees.- How to handle customer complaints and document problems.

Ways the New Skills Can be AppliedOnce I return to work, I can apply these new skills in the following ways:

- Use the standard customer greeting I learned in class with every customer.- Ask questions of the customer, following the Inquiry Matrix I learned about in class.- Listen actively to identify the customer’s real need.- Use the Customer Database (CD) to access account number in the system.- Use the system features to update the customer’s account.- Handle any complaints by using the Customer Complaint Resolution Job Aid.

Obstacles or Barriers to Skills ApplicationBarriers include:

- Lack of time to practice on the job.- Waiting time for system to access account information.- Some customers do not speak English well and are hard to understand.- Some customers are already so upset by the time they call that it is difficult

to solve their problem.

Enablers for Skill ApplicationEnablers include:

- My supervisor wants met to succeed and is willing to give me time to learn the new skills.- My co-workers are willing to help me and answer questions when they have time.- I like helping people and will try to satisfy upset customers. - The computer system is being improved to speed up access time.

Sample Action Plan for Customer Service Training, continued

Specific Steps To Be Taken and Deadlines Step: Deadline:1. Use standard greeting on all calls. Tomorrow2. Begin using the Inquiry Matrix on all calls. Tomorrow3. Practice looking up customer information. Tomorrow4. Become proficient on the Customer Database System. One month from now5. Use the Customer Complaint Resolution Job Aid to One Month from now

resolve all customer complaints.

Follow up Actions to be Taken to Review Progress- Meet with my boss to discuss this action plan within one week.- Begin using the job aids immediately to help me.- Ask co-workers for help when I need it.- Register to attend another customer service class next month.

Sample Action Plan for Customer Service Training, continued

District Manager Training Pre-Post Survey

This manager Pre Post Pre Post Pre Post Pre Post Pre Post

Communicates effectively with employees 3.5 4.5 2.4 3.0 2.1 2.9 3.0 3.3 2.8 3.4 21%

Establishes clear performance expectations and goals 2.7 4.1 1.9 2.7 3.1 3.9 3.5 3.6 2.8 3.6 29%

Provides coaching and on the job training 3.2 3.8 3.0 3.3 2.5 2.9 3.7 3.7 3.1 3.4 10%

Question Self Manager Employee Colleague Average%

Gain

1. What conclusions would you draw about the training’s impact on supervisors’ job behavior?

2. What recommendations would you make about changes to the program based on this data?

3. How could you translate this data into financial results?

Balanced Scorecard

Training Activity % Payroll

US Avg.=1.9% ‡

$/Employee/yearUS Avg.=$569 ‡

Hrs./Empl./yearUS Avg.= 36 ‡

Training Efficiency Cost/Hour

US Avg.=$28 ‡

% Billable Hrs.US Avg.=70% †

Training Results % Positive Course Ratings

Avg.=91%†

% Learning GainAvg.=45%†

% Behavior ChangeAvg.=25%†

Revenues/EmployeeAvg.=$100K†

Market/Book† Avg.=10:1

† Source=Author’s Study of High Tech firms

‡ Source: ASTD Training Data Book, 1996.

Your Scorecard

Training Activity % Payroll:_____

Training budget/Payroll

Hrs./Empl.____ Training hours/Employees

$/Empl. _____ Training budget/Employees

Training Efficiency Cost/Hr. ______

Training budget/ training hours

% Billable ____ Billable hours/staff hours

Training Results % Positive Ratings

Positive ratings/total ratings _________________ % Learning Gain

Post-pre / Pre-test scores

_________________ % Performance Gain

Post-pre / Pre-performance

_________________ Revenues/Employee

Total revenues/employees

_________________ Market/Book Ratio

Total market worth/ total physical assets

Training Design Job Aid

Analysis Phase

Needs Analysis Performance Analysis Needs Assessment Job Analysis

Task Analysis Learner Analysis Context Analysis Skill Gap Analysis

Design Phase

Training Objectives Deliverables/Strategies Budgets/Schedules

Project management Blueprints/Prototypes

Development Phase

Draft Materials Tests/Assessment

Quality Control Full-scale Production

Training Design Job Aid, continued

Implementation Phase

Train the Trainer Classroom Delivery

Non-Classroom Delivery

Evaluation Phase

Formative Reactions Learning

Transfer to Job Results/ROI

Closing commentsand Q&A

Donald J. Ford, Ph.D., CPT

Training Education Management

Redondo Beach, CA 90277

Telephone/Fax: 310-316-2240

E-mail: donaldford@earthlink.net

Web: www.trainingeducationmanagement.com