The Bottom Line on ROI: Measuring the Impact and ROI of ... · The Bottom Line on ROI: Measuring...

The Bottom Line on ROI:Measuring the Impact and ROI of

Organization Development Programs, Projects, and Initiatives

Timothy R. Brock, PhD, CPT, CRP

Director of Consulting Services

ROI Institute

BOTTOMLINE on ROIThe

2

Reaction Objectives

The intent is for you to perceive this 90-minute session content as:

Relevant to their work

Important to their success

Immediately applicable


3

After completing this session, you should be able to:

Identify the drivers for ROI

Make the business case for ROI

Explain how business alignment is achieved

Identify 7 of the 12 guiding principles

Describe the 10 steps in the ROI Methodology

Learning Objectives


4

Within two weeks of this session, you should:

Conduct a briefing on ROI.

Develop Level 3 and Level 4 objectives for a specific program.

Application Objectives


5

There is greater demand to show the value of investments in OD programs, projects, & initiatives at my organization

than in the past.

A. Yes

B. No

C. I don’t know


6

What is Value?

Exercise:

Think about a project or program that you recently implemented.

How would you define the value delivered by the program?


7

The New Definition of ValueValue Must:

Be balanced, with qualitative and quantitative data

Contain financial and nonfinancial perspectives

Reflect strategic and tactical issues

Satisfy all key stakeholders

Be consistent in collection and analysis

Be grounded in conservative standards

Come from credible sources


8

The “Show Me” Evolution


9

Shifting ParadigmsActivity Based Results Based

No business need for the program Program linked to specific business needs

No assessment of performance issues Assessment of performance effectiveness

No specific measurable objectivesSpecific objectives for behavior and business impact

No effort to prepare program participants to achieve results

Results expectations communicated to participants

No effort to prepare the work environment to support transfer

Environment prepared to support transfer

No efforts to build partnerships with key managers

Partnerships established with key managers & clients

No measurement of results or benefit-cost analysis

Measurement of results and benefit-cost analysis

Planning and reporting is input focused Planning and reporting is outcome focused

Source: Measuring ROI in Healthcare: Tools and Techniques to Measure the Impact and ROI in Healthcare Improvement Projects and Programs, 2013, V. V. Buzachero, J. Phillips, P. P. Phillips, and Z. L. Phillips.


10

The Executive View *Measure

CurrentMeasure

ShouldMeasure

Measure Importance

1. InputsLast year, 78,000 employees received formal learning.

94%

2. EfficiencyFormal learning costs $2.15 per hour of learning consumed.

78%

3. ReactionEmployees rated our training very high, averaging 4.2 out of 5.

53%

4. Learning92% of participants increased knowledge and skills

32%

5. ApplicationAt least 78% of employees are using the skills on the job.

11%

6. ImpactOur programs are driving our top 5 business measures in the organization.

8%

7. ROIFive ROI studies were conducted on major programs yielding an average of 68% ROI.

4%

8. AwardsOur learning and development program won an award from the Association for Talent Development.

40%

* CEO Survey—Fortune 500 and Large Private Company, ROI Institute

85%

82%

22%

28%

61%

96%

74%

44%

6

5

8

4

1

2

3

7


MeasurementLevels

Measurement Category

2006Status

2011Status

0 Inputs/IndicatorsNumber of programs, participants, costs, efficiencies

100% 100%

1 ReactionReaction and satisfaction with experience,content, value of program

92% 89%

2 LearningAcquisition of information, knowledge, skills, contacts

48% 59%

3 ApplicationUse of information, knowledge, skill, insights

11% 34%

4 ImpactChanges in business impact measures such as output, quality, time, costs

8% 21%

5 ROICompares monetary benefits to cost of program, project, or initiative

2% 11%

2011 Global Leadership Development Practices Survey Results

N=232; large organizations with leadership development budgets averaging $3 million annually.


12

The ROI Methodology

Generates Six Types of Measures:

• Reaction and Planned Action

• Learning

• Application

• Business Impact

• Return on Investment

• Intangible Measures

The ROI Methodology also includes a technique to isolate the effects of the program or solution.


13

An Evaluation System Must Have Five Elements


14

Level Measurement Focus

1. Reaction and PlannedAction

Measures participant satisfaction with the program/project and captures planned action

2. LearningMeasures changes in knowledge andskills

3. ApplicationMeasures changes in on-the-job behavior or actions

4. Business ImpactCaptures changes in business impact measures

5. Return on InvestmentCompares monetary benefits to the costs

ROI Framework


15



16

Develop/ Review

Objective of Program

Develop EvaluationPlans and

Baseline Data

Collect Data During

Program

Implementation

Collect DataAfter

Program

Implementation

Evaluation Planning Data Collection

LEVEL 1: REACTION AND PLANNED ACTION

LEVEL 2:LEARNING

LEVEL 3: APPLICATION

LEVEL 4: BUSINESS IMPACT

The ROI Methodology, Part 1


17

Isolate the Effects of

Program

Convert Data To

MonetaryValue

Calculatethe Return

OnInvestment

GenerateImpact

Study

Data Analysis Reporting

LEVEL 5: ROI

Intangible Benefits

Capture Costs of Solution

Identify Intangible Measures



18



19

Guiding Principles, Part 1

1.When conducting a higher-level evaluation, collect data at lower levels.

2.When planning a higher level evaluation, the previous level of evaluation is not required to be comprehensive.

3.When collecting and analyzing data, use only the most credible sources.

4.When analyzing data, select the most conservative alternatives for calculations.

5.Use at least one method to isolate the effects of the program or project.

6.If no improvement data are available for a population or from a specific source, assume that no improvement has occurred.


20

Guiding Principles, Part 2

7.Adjust estimates of improvements for the potential error of the estimates.

8.Avoid use of extreme data items and unsupported claims when calculating ROI calculations.

9.Use only the first year of annual benefits in the ROI analysis of short-term solutions.

10.Fully load all costs of the solution, project, or program when analyzing ROI.

11.Intangible measures are defined as measures that are purposely not converted to monetary values.

12.Communicate the results of the ROI Methodology to all key stakeholders.


21



22

Example Application Case Studies


23



ROI Myths• ROI is too complex for most users.

• ROI is too expensive, consuming too many critical resources.

• If senior management does not require ROI, there is no need to pursue it.

• ROI is a passing fad.

• ROI is too subjective

• ROI is for post analysis only

24


25


26

The ROI Calculation

Benefit Cost Ratio (BCR) =

ROI = X 100

$300,000Program Costs

$300,000 - $200,000(Program Benefits – Program Costs)

$200,000

$200,000Net Program Benefits

Program Costs$100,000

= 1.50:1

= 0.50 X 100% = 50%

Program Benefits


27

It’s That Easy!


28

Develop/ Review

Objective of Program

Develop EvaluationPlans and

Baseline Data

Collect Data During

Program

Implementation

Collect DataAfter

Program

Implementation

Evaluation Planning Data Collection

LEVEL 1: REACTION AND PLANNED ACTION

LEVEL 2:LEARNING

LEVEL 3: APPLICATION

LEVEL 4: BUSINESS IMPACT



29

Reaction & Planned Action

Learning

Application & Implementation

Isolate the Effects of the Program

Intangible Benefits

Impact

Ch

ain

of

Imp

act


30

Level 0 Input and Indicators

• Number of programs

• Participants

• Hours

• Requests

• Efficiencies

• Costs

• Time to Deliver

Level 1 Reaction and Planned Action

• Relevance*

• Importance*

• Usefulness

• Recommend to Others

• Necessary

• Practical

• Uniqueness

• Concreteness

• New Information*

• Motivation

• Appropriateness

• Intent to Use**Correlates with Application

Five Levels of Measurement: Examples


31

Level 2 Learning

• Information

• Knowledge

• Understanding

• Capability

• Contacts

• Confidence

• Perceptions

• Skills

Level 3 Application

• Use of Information

• Use of Knowledge

• Use of Skill*

• Frequency of Use*

• Completion of Actions

• Completion of Tasks

• Implementation of Ideas

• Following the Policy

• Use of Procedure

• Use of Regulation

• Success with Application

• Barriers*

• Enablers* * Essential Measures at this Level



32

Level 4 Business Impact Level 5 Return on Investment• Productivity

• Quality

• Errors

• Incidents

• Re-Work

• Efficiency

• Compliance Discrepancies

• Citizen Complaints

• Costs

• Employee Engagement

• Employee Retention

• Service Delivery

• Cycle Time

• Customer Satisfaction

• ROI (%)

• Benefit/Cost Ratio

• Payback Period

Intangible Measures

…. includes a technique to isolate the effects of the communication project.

• Image

• Reputation

• Teamwork

• Stress

• Brand



33

Considerations when Evaluating Programs


34

Criteria for Selecting Programs for Level 4 and Level 5 Evaluation

Benchmarking*

• Cost of the program 52%

• Importance of program to strategic objectives 50%

• Executive interest in the evaluation 48%

• Visibility of the program 45%

• Linkage of program to operational goals and issues 29%

• Life cycle of the program 14%

• Investment of time required 7%

• Size of target audience 6% *Survey of Users, N = 235


35

The Levels Serve Three Purposes!

Needs

Objectives

Evaluation


36

Program Alignment: The V Model


37

Program

Payoff Need

Business Need

Performance Need

Learning Need

Preference NeedInitial

Analysis

Start Here End Here

5 ROI5

4 Impact4

3 Application3

2 Learning2

1 Reaction1

Absenteeism is costing $10K/month

• Unexpected absenteeism is 9% & growing

• Benchmark data is 5%

Discussion between team member & supervisor is not occurring when there is an unplanned absence

Deficiency is counseling/ discussion skills

• One day counseling skills workshop must provide relevant, practical skills

• Facilitator-led• Participants are supervisors

• Reaction questionnaire at end of workshop• Planned Actions

• Skills practice during workshop• Participant assessments• Facilitator assessments

• Questionnaire – at 3 months• Action Plan – 6 months

• Monitor absenteeism records for 6 months

• Action Plan – 6 months

Calculate ROI/BCR

Program rated 4 out of 5 on: Need, Relevance, Practicality

+ Commitment

Applies counseling skills to standard in work-like setting

Reduce absenteeism to 5% six months after course

ROI of 25%

• Apply Action Plans• Conduct counseling sessions in 95% of

situations when an unexpected absence occurs

Business Alignment & Forecasting

Measurement & Evaluation

ROI Process Model

Needs Objectives Evaluation

ROI Objectives

Impact Objectives

Application Objectives

Learning Objectives

Reaction Objectives


38


39


40


41


42


43


44

Levels of Objectives

Measurement Focus

0. Inputs and IndicatorsThe input into the project in terms of scope, volume, efficiencies, costs

Participants, hours, costs, timing

1. Reaction & PlannedAction

Reaction to the project or program, including the perceived value

Relevance, importance, usefulness, appropriateness, intent to use, motivation to take action

2. LearningLearning to use the content and materials, including the confidence to use what was learned

Skills, knowledge, capacity, competencies, confidence, contacts

3. Application &Implementation

Use of content and materials in the work environment, including progress with actual items and implementation

Extent of use, task completion, frequency of use, actions completed, success with use, barriers to use, enablers to use

4. Business ImpactThe consequences of the use of the content and materials expressed as business impact measures

Productivity, revenue, quality, time, efficiency, customer satisfaction, employee engagement

Typical Measures

5. ROIComparison of monetary benefits from program to program costs

Benefit-cost ratio (BCR), ROI%, payback period


46

Matching Evaluation Levels with Objectives

1. Reaction and Planned Action

2. Learning

3. Application

4. Business Impact

5. Return on Investment


47

1. Error rates on reports will decrease by 20 percent.

1. Error rates on reports will decrease by 20 percent.

44

2. Participants will use disciplinary discussion skills in 90 percent of discussions.

2. Participants will use disciplinary discussion skills in 90 percent of discussions.

3. Customer complaints will decrease.3. Customer complaints will decrease.

4. Participants will indicate an intention to implement the new procedure within two

months.

4. Participants will indicate an intention to implement the new procedure within two

months.

5. Participants will score 75 or higher on the new product quiz.

5. Participants will score 75 or higher on the new product quiz.

33

44

11

22

1 Reaction 2 Learning 3 Application 4 Impact 5 ROI


48

6. Participants will achieve a post-test score increase of 30 percent over pretest.

6. Participants will achieve a post-test score increase of 30 percent over pretest.

7. Participants will initiate at least three cost reduction projects within 15 days.

7. Participants will initiate at least three cost reduction projects within 15 days.

8. Per project costs will be reduced by $2,000. 8. Per project costs will be reduced by $2,000.

22

33

44

9. Time to complete a project will decrease.9. Time to complete a project will decrease. 44

10. Program implementation will realize a 2:1 benefit-cost ratio within one year.

10. Program implementation will realize a 2:1 benefit-cost ratio within one year.

55



49

11. Program relevance to the job will receive a score of at least 4.5 out of 5.0.

11. Program relevance to the job will receive a score of at least 4.5 out of 5.0.

12. External customer satisfaction index increase by 25 percent in three months.12. External customer satisfaction index increase by 25 percent in three months.

13. CSRs representatives will handle customer complaints with the 5-step process in 95% of

complaint situations

13. CSRs representatives will handle customer complaints with the 5-step process in 95% of

complaint situations

14. Canal operators will achieve a simulation test score average of 75.

14. Canal operators will achieve a simulation test score average of 75.

15. Participants will use new customer services interaction skills with every customer at least 50%

of the time.

15. Participants will use new customer services interaction skills with every customer at least 50%

of the time.

11

44

33

22

33



50

17. Participants will use new software daily as reflected by an 90% score on an unscheduled

audit of use.

17. Participants will use new software daily as reflected by an 90% score on an unscheduled

audit of use. 33

18. Customer response time will be reduced. 18. Customer response time will be reduced.

19. Supervisors will conduct a meeting with direct reports to implement new policy.

19. Supervisors will conduct a meeting with direct reports to implement new policy.

20. Employees will submit ideas or suggestions for improvement in the first year.

20. Employees will submit ideas or suggestions for improvement in the first year.

44

33

33

16. Participants will rate the training as a good investment for the company.

16. Participants will rate the training as a good investment for the company.

11



51

This ROI Methodology is a Process Improvement toolFund, Fix, or Fold

• Just because a program has a negative ROI does not mean the program is discontinued

• Must understand what happened so we can make changes to make it positive in the future to drive business value

• Our analysis statistics indicate that 20%-30% of programs have a negative ROI with 10% of those being discontinued.

• It is rare when ROI is the only factor• Intangibles can offset a negative ROI

• Ownership, engagement, commitment, communication, teamwork, cooperation, etc.

• If it can’t produce any business value, then we have a data-driven case to let it die in peace


52

The Power of Objectives

Level 3 and Level 4 Objectives provide:

• Direction to designers and developers

• Guidance to facilitators and coordinators

• Goals for participants

• Satisfaction for program sponsors

• A framework for evaluators


53

Data Collection During and After ProgramMethod Level 1 Level 2 Level 3 Level 4

Surveys

Questionnaires

Observation

Interviews

Focus Groups

Tests/Quizzes

Demonstrations

Simulations

Action planning/improvement plans

Performance contraction

Performance monitoring


54


Program

Convert Data To

MonetaryValue

Calculatethe Return

OnInvestment

GenerateImpact

Study


LEVEL 5: ROI

Intangible Benefits





55

6.2


56

6.3


57

Systems/ProceduresChanges

Several Factors Contribute to an Improvement After a Program in Conducted

External Factors

ManagementAttention

Incentives

ODPrograms

TOTALIMPROVEMENT

AFTERPROGRAM

EFFECT OF HR ON IMPROVEMENT


58

Isolating the Effects of the ProgramBenchmarking Data*

* Survey of Users, N = 235


59

M1 M2

M1 Program M2Experimental

Control

M2

Program M2Experimental

Control

M2

Program M2Experimental

Control

M2Control

M1

M1

Control Groups


60

2%

1%

J F M A M J J A S O N D J

1.85% Pre Program Six-Month Average

Projected Average —Using Pre Data as a

Base 1.45%

0.7% Post Program Six-MonthAverage

CPI Program Conducted

MONTHS

Reject Rate

Trend Line Analysis

5.8


61

Monthly increase in credit card accounts: 175 (fact)

Contributing Factors Consensus

Impact (%)

Average Confidence (%)

Sales Training 32% 83%

Incentives 41% 87%

Management Reinforcement

14% 62%

Market Fluctuations 11% 75%

Other _________ 2% 91%

100%

5.12

175 x = 56 x = 46.48(at least)

Participant Estimates


Influence Fact % Contrib. Est. Impact Confidence Adjusted Impact

Sales Training 175 32% 56 83% 46.48

Fact: 175 New Credit Card Accounts

% Contribution: 32%

Est. Impact: 56

Uncertainty 17%

Margin of Error: +/- 9.52

65.52

56

46.48

62

Participant Estimates


63


Program

Convert Data To

MonetaryValue

Calculatethe Return

OnInvestment

GenerateImpact

Study


LEVEL 5: ROI

Intangible Benefits




64

Data Conversion

• Converting output to contribution – standard value

• Converting the cost of quality – standard value

• Converting employee’s time – standard value

• Using historical costs

• Using internal and external experts

• Using data from external databases

• Linking with other measures

• Using participants’ estimates

• Using supervisors’ and managers’ estimates

• Using staff estimates

Slide 64

The BOTTOMLINE on ROI


Five Steps to Data Conversion

1. Focus on the unit of measure.

2. Determine the value (V) of the unit of measure.

3. Calculate the change in performance(∆P).

4. Determine the annual change in performance (A∆P).

5. Calculate the annual monetary benefits (A∆P x V).


1. One grievance

2. V = $6,500

3. ∆P = Prevented 7 out of 10 grievances/month

4. A∆P =

5. A∆P x V =

7 x 12 = 84

84 x $6,500 = $546,000

Five Steps to Data Conversion


To Convert or Not to Convert

67

No

Is there a standard value?

Yes

Is there a method to get there?

Can we get there with minimum resources?

No

Yes

No No

Yes

Add to numerator

Leave as Intangible

Can we convince our executives of credibility in 2 minutes?

Convert data

Yes

68

Typical Intangible Benefits

Adaptability

Awards

Brand awareness

Career minded

Caring

Collaboration

Communication

Conflicts

Cooperation

Corporate social responsibility

Creativity

Culture

Customer complaints

Customer response time

Customer satisfaction

Decisiveness

Slide 68


69

Typical Intangible Benefits

Employee complaints

Engagement

Execution

Image

Innovation

Job satisfaction

Leadership

Networking

Organizational climate

Organizational commitment

Partnering

Reputation

Resilience

Stress

Talent

Teamwork

Slide 69



70


Program

Convert Data To

MonetaryValue

Calculatethe Return

OnInvestment

GenerateImpact

Study


LEVEL 5: ROI

Intangible Benefits





71


72

Micro Level Scorecard Macro Level Scorecard

0 Indicators

1 Reaction

2 Learning

3 Application

4 Impact

5 ROI

Intangibles

0

1

2

3

4

0

1

2

3


73



74

The ROI Methodology

The Bottom Line on ROI:Measuring the Impact and ROI of

Organization Development Programs, Projects, and Initiatives

Timothy R. Brock, PhD, CPT, CRP

Director of Consulting Services

ROI Institute

[email protected]

Thank you for attending.


Please Complete the Survey Before You Leave

Your feedback is greatly appreciated

Thank you!


77

ROI Calculation

Net Project BenefitsProject Costs

ROI =

Cost of project $230,000Benefits of project (1st year) $430,000

ROI =$430,000 - $230,000

$230,000= 0.87 x 100 = 87%


Impact Measures

78Slide 78

Output

• Units Produced

• Items Assembled

• Revenue

• Items Sold

• Forms Processed

• Inventory Turnover

• Output Per Hour

• Productivity

• Work Backlog

• Incentive Bonus

• Shipments

• New Accounts Generated


79Slide 79

Cost

• Budget Variances

• Unit Costs

• Cost by Account

• Variable Costs

• Fixed Costs

• Overhead Cost

• Operating Costs

• Number of Cost Reductions

• Project Cost Savings

• Accident Costs

• Sales Expense

Impact Measures


80Slide 80

Time

• Equipment Downtime

• Overtime

• On Time Shipments

• Time to Project Completion

• Processing Time

• Cycle Time

• Supervisory Time

• Training Time

• Repair Time

• Efficiency

• Work Stoppages

• Order Response

• Late Reporting

• Lost Time Days

Impact Measures


81Slide 81

Quality

• Scrap

• Waste

• Rejects

• Error Rates

• Rework

• Shortages

• Product Defects

• Deviation From Standard

• Product Failures

• Inventory Adjustments

• Number of Accidents

Impact Measures


82Slide 82

Customer Service


Survey


Index

• Customer Complaints

• Customer Comments

• Customer Defection

• Customer Retention

Work Habits

• Absenteeism

• Tardiness

• First Aid Treatments

• Violations of Safety Rules

• Excessive Breaks

Impact Measures


83Slide 83

Work Climate

• Number of Grievances

• Number of Discrimination

Charges

• Employee Complaints

• Job Satisfaction

• Organizational

Commitment

• Employee Turnover

• Reduced Litigation

Impact Measures


Case Study: The Result

84

Four months after implementing an expensive, strategic program, you conduct an impact study and discover from participants involved:

• Reacted very positively to the program and found it to be very relevant to their work;

• Learned new skills and gained new insights about themselves;

• Applied the skills and insights routinely with their teams, although they had some difficulty in a few areas;

• Improved several important work unit measures, with some measures improving as much as 28%;

• Achieved an impressive 105% return on investment; and

• Reported an increase in job satisfaction in the work unit.

Slide 84


85Slide 85

What were the objectives?

How many people were involved?

What was the scope of analysis?

What assumptions were made in the analysis?

What specific measures were improved?

When were the data collected?

What was the data collection plan

beyond reaction?

How were the needs and measures determined?

What assumptions were made in the ROI

calculations?

How were results isolated from other

factors?

What is the upfront business connection?

How are you tracking costs?

Case Study: The Result

The Bottom Line on ROI: Measuring the Impact and ROI of ... · The Bottom Line on ROI: Measuring...

Documents

Transcript of The Bottom Line on ROI: Measuring the Impact and ROI of ... · The Bottom Line on ROI: Measuring...