ROI Evaluation Approach

32
Session 3 – Evaluation Approach

description

ROI Evaluation Approach

Transcript of ROI Evaluation Approach

Page 1: ROI Evaluation Approach

Session 3 – Evaluation Approach

Page 2: ROI Evaluation Approach
Page 3: ROI Evaluation Approach
Page 4: ROI Evaluation Approach

1. Reaction, Satisfaction

2. Learning3.

Transfer/application

4. Business Impact

5. Return on Investment

6. Intangibles

Page 5: ROI Evaluation Approach

Element Description

1. Reaction Happiness with learning interventions

2. Learning Changes in knowledge, skills and attitudes

3. Application and implementation

Changes in on the job behaviors over time (competence)

4. Business Impact Changes in impact variables

5. ROI Intervention benefits in relation to cost

Intangibles Intervention benefits in relation to policy and/or strategic direction and related factors

Page 6: ROI Evaluation Approach

• Skills to master:– Ability to define a planning

framework for each level:• Reaction• Learning• Transfer/application• Business

– Ability to work from the business level to the reaction level

Page 7: ROI Evaluation Approach

Level Need Program Objective

Evaluation Strategy

Metric

1

2

3

4

5

Page 8: ROI Evaluation Approach

Training Need Program Objectives

Evaluation Approach

Level

Business need Impact objectives

Business impact Level 4

Job performance needs

Application objectives

Transfer and application

Level 3

Competence needs

(Skills; knowledge; attitudes)

Learning objectives

Learning and competence

testing

Level 2

Preferences Satisfaction objectives

Reaction Level 1

Page 9: ROI Evaluation Approach
Page 10: ROI Evaluation Approach

Training Need Program Objectives

Evaluation Approach

Level

Business need Impact objectives

Business impact Level 4

Job performance needs

Application objectives

Transfer and application

Level 3

Competence needs

(Skills; knowledge; attitudes)

Learning objectives

Learning and competence

testing

Level 2

Preferences Satisfaction objectives

Reaction Level 1

Page 11: ROI Evaluation Approach
Page 12: ROI Evaluation Approach

Training Need Program Objectives

Evaluation Approach

Level

Business need Impact objectives

Business impact Level 4

Job performance needs

Application objectives

Transfer and application

Level 3

Competence needs

(Skills; knowledge; attitudes)

Learning objectives

Learning and competence

testing

Level 2

Preferences Satisfaction objectives

Reaction Level 1

Page 13: ROI Evaluation Approach

Cognitive Affective Psychomotor

Recall Awareness Imitation - copy

Understanding Respond Manipulation – follow instructions

Application Value (understand and act)

Develop precision

Analysis Organize personal value system - talk

Articulation – combine and relate related skills

Synthesize Internalize personal value system - walk

Naturalization - automate

Evaluate

Page 14: ROI Evaluation Approach

Training Need Program Objectives

Evaluation Approach

Level

Business need Impact objectives

Business impact Level 4

Job performance needs

Application objectives

Transfer and application

Level 3

Competence needs

(Skills; knowledge; attitudes)

Learning objectives

Learning and competence

testing

Level 2

Preferences Satisfaction objectives

Reaction Level 1

Page 15: ROI Evaluation Approach

Disagree Agree

1 2 3 4 5 6

The content was relevant for what I required

The manual followed the lectures logically

The presentation material was of a high quality

Presentations were interesting and engaging

The lecturer know the material

The lecturer knows the subject well

The lecturer was well prepared

The venue was conducive to learning

The food was of an acceptable standard

Page 16: ROI Evaluation Approach
Page 17: ROI Evaluation Approach
Page 18: ROI Evaluation Approach

• Wissan Motor Manufacturing Company has been rated as a very top quality, highly productive automobile manufacturer. Wissan relates much of its success on four major areas: reliable systems, problem resolution, follow-up, and continuous improvement through learning.

• One of the successful learning programs is on shop-floor supervision where teams and team leaders are taught how to:– Plan as a team– Conduct team review meetings– Have performance discussions– Implement shop floor disciplinary code

Page 19: ROI Evaluation Approach

• The program is designed to improve quality, reduce costs, improve delivery, and improve safety. Delivery is defined as the percent of daily production schedule met.

 • Questions for discussion• At what level could this program be evaluated? How?• What are the highest level measures?

Page 20: ROI Evaluation Approach
Page 21: ROI Evaluation Approach

• In response to increased claims of sexual harassment, the Department of Indiscreet Affairs, a large federal agency, developed a one-day sexual harassment workshop for supervisors and managers. After the workshop, supervisors were expected to meet with their employees and discuss the issues with them.

Page 22: ROI Evaluation Approach

After attending this workshop, participants should be able to:

• Understand, interpret, and enforce the agency’s sexual harassment policy.

•  Identify comments, actions, and behavior that might constitute sexual harassment.

•  Reduce the number of complaints regarding sexual harassment.

•  Communicate to employees the policy and expected behavior.

   Questions for discussion:•  At what levels could you evaluate this program?

Please explain.• How would you evaluate at Levels 2, 3, and 4?

Page 23: ROI Evaluation Approach
Page 24: ROI Evaluation Approach

• Accra Wholesale Brickworks, find that there production targets are not being met. The Company is midsize but established in the City. It employs a production team consisting of three foreman, that supervise approximately 8 workers each and these teams take responsibility for procurement of the raw materials, mixing, filling the moulds, drying and baking. The production function is managed by a section manager.

Page 25: ROI Evaluation Approach
Page 26: ROI Evaluation Approach
Page 27: ROI Evaluation Approach

Decision Tree 1:

Their systems are manual and

procurement/production planning is not

possible

Can this be solved

through training?

Decide type of training and proceed

to Decision Tree 2

Page 28: ROI Evaluation Approach

Decision Tree 1:

Foreman don’t know how to conduct

proper production planning and

monitoring meetings

Can this be solved

through training?

Decide type of training and proceed

to Decision Tree 2

Page 29: ROI Evaluation Approach

Decision Tree 2:

The basic business need is to raise improve profits

False/True

Decide what metric will you use to show

the value of an increase in production

Page 30: ROI Evaluation Approach

Decision Tree 2:

The basic business need is to raise

production

False/True

Decide what metric will you use to show

the value of an increase in production

Page 31: ROI Evaluation Approach

Level Need Program Objective

Evaluation Strategy

Metric

1

2

3

4

5

EXERCISE

Page 32: ROI Evaluation Approach