Using an ROI Approach to Better Manage Learning ... Conf... · Better Manage Learning Investments...
Transcript of Using an ROI Approach to Better Manage Learning ... Conf... · Better Manage Learning Investments...
©2007 KnowledgeAdvisors. All rights reserved.
Using an ROI Approach to Better Manage Learning Investments and Reduce Scrap Learning
Page 211-Nov-07
Overview of KnowledgeAdvisors
Visit our website at www.knowledgeadvisors.com
Benchmarking
We help organizations measure and improve the results of their people programs.
Professional Services
Benchmarking
Technology
Human Capital Analytics
Page 311-Nov-07
Should you calculate ROI?
We make choices based on:
•Reaction
•Knowledge/Attitude
•Utility
•Economics
A
B
Page 411-Nov-07
Redirect poor-performing investments into high-impact programs
Drive Business Results
Time
Low ImpactProgramsHigh ImpactPrograms
Earnings
Page 511-Nov-07
Shift Paradigm
Page 611-Nov-07
What CEO’s Want
revenue
employees
• Revenue grows
• Human capital investment is flat
revenueemployees
• Revenue is flat
• Human capital investment declines
OR
Page 711-Nov-07
Data reporting20%
Data analysis19%
Data filtering9%
Data collection33%
Data aggregation19%
Learning Measurement Resource Allocation Today
Source: KnowledgeAdvisors Learning Analytics Best Practices Research Study, 2004.
• 81% of measurement resources focused on administration (collection, aggregation, filtering, reporting).
• < 20% is available for analysis and decision-making. • Technology reverses the ratio: 80% on analysis, 20% on
administration.
Page 811-Nov-07
Reasonable Indicators
Source: KnowledgeAdvisors Learning Analytics Best Practices Research Study, 2004.
In your organization, what best describes the level of accuracy that analytics must have in order to be
perceived as useful for information decision-making purposes of senior management?
There is an enormously high cost
to data accuracy versus obtaining a
reasonable indicator.
Organizations should balance when to spend extra time and money
to yield highly statistical results.
Given how executives make decisions, reasonable data
provided in a timely manner outweighs
data with more precision delivered in a
less timely manner.
Periodic "word of mouth"
statements11%
Highly precise, statistically valid
measures11%
Reasonable, quantitative / qualitative indicators
78%
Page 911-Nov-07
Human Capital Impact
Smart investments in learning drive future business results, which ultimately increase stakeholder value.
Page 1011-Nov-07
Why Invest in Learning?
Time
$$
Earnings without a strong corporate
education program
Earnings with a strong corporate
education program
Page 1111-Nov-07
How do Shareholders Benefit?
Time
$$
Book Value
Market Value
Human Capital Value
Page 1211-Nov-07
Optimizing Training Investments
Nobel Prize Winner, Gary BeckerHuman Capital Formula
MP = Marginal product of traineesG = Present value of returnsW = Wages paidi = Market discount rateC = Cost of training
Page 1311-Nov-07
Collect Data
Isolate the effects of training
Convert data to monetary
value
Calculate the return on
investment
Tabulate program
costs
Identify tangible benefits
Evaluation Instruments
Evaluation Purposes
Evaluation Timing
Evaluation Levels
The ROI Process
Source: Return on Investment in Training and Performance
Improvement Programs, Jack J. Phillips, Gulf Publishing Company,
Houston Texas, 1997, p. 25
Page 1411-Nov-07
Estimate the change in performance(ex. How much did sales increase since the sales improvement initiative?) 20%
Isolate the effect that training had on the change(ex. What % of sales increase was driven by the training vs. other factors?)=10%
Adjust for confidence (or lack thereof) in estimates (ex. How confident are you in your estimates?)=50%
Monetize the performance change related to training(ex. $100,000 is a 20% increase in sales) = $100,000 x.10 x .50 = $5,000 benefit from initiative
Fully loaded costs to compare against the benefit(ex. Cost to rollout initiative is $1,000, travel is $500, lost work time is $800) =$2,300
Estimate, Isolate, Adjust
Page 1511-Nov-07
The Measurement Process
Data Collection
Define metrics (KPI’s)Design instrumentsDetermine populationDisseminate instrumentsCollect data
Because data collection, processing and reporting is automated, you have more time to analyze the data and turn it into actionable business intelligence to:
• Improve your learning programs• Demonstrate value to your stakeholders
Data Storage
Data Processing
Data Reporting
Database creationCentralize data Data entryData securityRaw data access
Aggregation of dataFiltering of dataConvert into metrics
Standard reports (canned)Ad-hoc queryingTrend-line analysisStatistical analysisBenchmark comparisonsTransactional detailsManager reportsExecutive summaries
Page 1611-Nov-07
After Training: Practical Indicators
ReactionLevel 1
LearningLevel 2
Job ImpactLevel 3
ResultsLevel 4 & EstimationIsolation
Adjustment
ROI/ROELevel 5
Page 1711-Nov-07
ROI Models to Manage Investments
Cost and Complexity
Page 1811-Nov-07
ROI – The Cost Side
Direct and measurable costs
Analysis costsDesign, development: over the life of the programMaterials: workbooks, participant handoutsInstructor prep and deliveryFacilitiesContent acquisitionAdministrative and allocated overhead Evaluation costs
Indirect or imputable costs
Participant tuitionParticipant travel expensesParticipant opportunity costs
Conservatism is a best practice!
Page 1911-Nov-07
Human Capital ROI
A Job Impact ROIEasy to use, practical, scaleable, replicableProvides a financial ROI relative to the improvement in human capital performanceUses principles of estimation, isolation, and adjustmentPart of a regular ‘balanced scorecard’ for learning measurementAssumption is that human capital’s value (i.e. salary) is like any other asset that has expected value that can appreciate or depreciate over time
Page 2011-Nov-07
Human Capital ROI: Example
Factor Item Amount
Actual Fully loaded salary of participants (average) $50,000
Actual Training cost per participant $1,000
Estimate Performance improvement, including training 30%
Isolate Performance improvement due to training 60%Isolate Performance relative to time spent performing
those skills on the job20%
Adjust Bias, confidence, conservatism 65%
Training impact on performance improvement, adjusted for bias: 30% x 60% x 20% x 65% = 2.34%Calculate monetized benefit = $50,000 x 2.34% = $1,170Calculate ROI = $1,170 - $1,000 = $170Calculate Benefit to Cost Ratio: $1,170 / $1,000 = 1.17
Page 2111-Nov-07
Case Study: New Horizons
Largest IT Training Company in the worldTrain in multiple certified IT channelsUsed by NH centers around the worldOver 1.5 million surveys collectedCollect data using multiple IT channel forms (Microsoft, Citrix, Cisco)Use the data to manage their operations quality and customer serviceLeverage the data for strategic marketing purposes (see PR)
Page 2211-Nov-07
New Horizons Highlights
38% of performance improvements were directly attributable and isolated to training 70% of participants apply training to job within 8 weeks97% utilize/plan to utilize the training on the job60% increase in skill/knowledge from trainingA 4:1 predicted benefit to cost ratio when completing the training
“Prior to our relationship with New Horizons we were not effectively providing a productivity baseline for our knowledge workers.”-- Anthony Garreffi, Manager E-Learning / Technical Training, Iron Mountain
Page 2311-Nov-07
New Horizons Highlights
Source: KnowledgeAdvisors Metrics that Matter Database
This shows the predicted ROI results for Iron Mountain. It is calculated based on the improvement in human
capital isolated to the training, adjusted for bias.
4/1 to 4/30Iron Mountain ROI Data
Iron Mountain Human Capital ROI Score Card
3.43Payback Period (months)250%ROI Percentage
3.5Benefit to Cost Ratio (per person)3,500Monetary Benefits1,000Costs (per person)
3.43Payback Period (months)250%ROI Percentage
3.5Benefit to Cost Ratio (per person)3,500Monetary Benefits1,000Costs (per person)
Level 5 Return on Investment Post Event
Page 2411-Nov-07
Business Results ROI
A deeper Level 4 ROILeverage technology to make practical and scaleableProvides a financial ROI relative to the change in the business result that prompted the trainingUses principles of estimation, isolation, and adjustment specific to the business resultPart of a regular ‘balanced scorecard’ for learning measurement
Page 2511-Nov-07
Business Result ROI: Example
Training cost per participant: $5,000Business Result Significantly Impacted: SalesEstimate change in sales (pre vs. post training) = +20% or $20,000 (sales were $100,000 before training, now they are at $120,000)(annualize this)Isolate performance improvement due to training: 60% (of the 20% above)Adjust for bias, confidence, conservatism: 65%Training impact on sales adjusted for bias: Calculate monetized benefit = 20% x 60% x 65% = +7.8%Calculate monetized benefit = $100,000 x 7.8% = $7,800 Calculate ROI = $7,800 - $5,000 = $2,800Calculate Benefit to Cost Ratio: $7,800 / $5,000 = $1.56
Page 2611-Nov-07
Case Study: PeopleSoft Education
Collected over 20,000 evaluations since September 2003Require participants to predict and estimate training linkage to the following business results:
Increased SalesDecreased CostsIncreased ProductivityIncreased QualityDecreased Cycle TimeIncreased Customer SatisfactionIncreased Employee Satisfaction
Can run a Business Result Score Card by Client, course, program etc. to look at linkage to business results
Page 2711-Nov-07
PeopleSoft Education Highlights
PR Issued March 22, 200420% improvement in user productivity after completing PeopleSoft training class24% improvement in cycle time 22% improvement in quality>90% feel the training was a worthwhile investment for their employers
“We see great value in being able to link PeopleSoft’ education courses to an increase in productivity.”- -Joseph Cummings, IS Director, DePaul (client of PeopleSoft)
Page 2811-Nov-07
Impact Study
A process whereby you can be certified to do itA formal step by step process for comprehensive impact study analysisA set of defined cost components for each impact studyMore comprehensive data collection instrumentsBusiness result analysis for each participant’s business resultsA specific Phillips ROI Scorecard with defined metrics at each level of learning measurementUsed for specific projects or programs where deep analysis is needed (about 5 to 10% of the time)
Page 2911-Nov-07
Impact Studies:Go in Depth to Prove Results
Sales Training Program: Impact Study revealed that trained employees average retail weekly sales were $12,075 vs. $10,449 for similar employees who were not trained.
Performance Management Program: A performance management program at a restaurant generated over $170,000 in benefits directly tied to the training yet the training only cost less than $50,000.
Masters Degree Program for Federal Agency: Goal was to reduce high turnover. Turnover 1 year after the program was 34% for employees NOT in the program and 3% for employees IN the program. It yielded a benefit to cost ratio of over 2 to 1.Above data from the ROI Institute, Drs. Jack and Patti Phillips
Page 3011-Nov-07
Identify Program Benefits
30
Hard Data
Output Cost
Quality Time
Soft Data
Customer Service Work Climate
Work Habits Initiative
Scrap Learning
Page 3111-Nov-07
55%55% of learning is not applied to the of learning is not applied to the
jobjob…….that.that’’s Scraps Scrap
How much learning is applied to the job?
When is learning applied to the job?
What are the barriers to impact?
What can be done to reduce scrap?
©2007KnowledgeAdvisors. All rights reserved.
Page 3211-Nov-07
Top 10 Ways to Waste Learning
1. Investments not aligned with business objectives2. Learning not applied to job3. Poorly designed programs4. Ineffective instructors5. Poor learning environment6. Insufficient support7. Inappropriate course selection8. Under-utilized resources9. Costs outweigh benefits10. Lower cost alternatives
Page 3311-Nov-07
Dashboards
Definition: A convenient place for vital information organized into a graphical representation that is both easy to use and easy to understand.
Page 3411-Nov-07
Examples of Macro Learning Constructs
OperationalActivity constructs (how much we train)
PerformanceOptimization constructs (how well we train)
FinancialBudget/Fiscal constructs (how much it costs)
Employee / CulturalSupportive Environment (how conducive is it)
Note: these constructs are part of KnowledgeAdvisors Metrics that Matter® standards
Page 3511-Nov-07
Sample Dashboards to Analyze Metrics
A report showing actual results over time with a color coded analysis to compare variances against prior periods and goals/benchmarks
Source: Metrics that Matter®
Operational Performance
Page 3611-Nov-07
Case Study Example: Sprint
SituationSix Learning InstitutesHundreds of programsUses the Phillips / Kirkpatrick Model
RequirementsTool that can forecast ROI and key metrics Populate the HR Scorecard for top management
Analytics Solution:Each institute has own metrics account to manage their dataRollup account created for the CLO to view metricsMTM tactical tools for day to day improvementsBenchmark against the database for external comparisonData from MTM goes directly to Nextel Scorecard
Page 3711-Nov-07
Metrics that Matter Report Card for Sprint
Page 3811-Nov-07
Sprint HRD Scorecard Example
Human Resources Development Scorecard
2004Metric Goal Jan Feb Mar Apr May Jun Mo. Avg
Metrics That Matter Captured DataTraining Participant Satisfaction w ith Instructors 6.00 6.36 6.55 6.50 6.48 6.32 6.47 6.45
Goal Achievement 106% 109% 108% 108% 105% 108% 107%Learning Effectiveness from Training 5.60 5.60 5.88 5.97 5.92 5.98 5.90 5.88
Goal Achievement 100% 105% 107% 106% 107% 105% 105%Job Impact f rom Training 5.60 5.58 5.66 5.98 5.86 5.94 5.88 5.84
Goal Achievement 100% 101% 107% 105% 106% 105% 104%Business Impact f rom Training 5.00 4.81 4.94 5.51 5.32 5.39 5.38 5.26
Goal Achievement 96% 99% 110% 106% 108% 108% 105%HRD System Captured Data
Total Participants for Instructor Led Training 1,607 1,636 2,103 2,909 3,831 2,615 2,450Total Participants for Online Facilitated Training 145 37 270 86 68 102 118Total Participants for WBT/Self-Study 16,461 10,118 9,219 9,348 18,438 10,394 12,330
Total Participants Trained (All) 18,213 11,791 11,592 12,343 22,337 13,111 14,898Participant Hours for Instructor Led Training 14,754 12,053 18,847 22,613 28,076 25,679 20,337Participant Hours for Online Facilitated Training 390 112 799 282 130 243 326Participant Hours for WBT/Self-Study Training 23,951 10,442 10,215 10,587 22,477 12,000 14,945
Total Participant Hours of Training 39,095 22,606 29,861 33,482 50,682 37,922 35,608% of Nextel Learning via ILT 38% 53% 63% 68% 55% 68% 57%% of Nextel Learning via Online Facilitation 1% 0% 3% 1% 0% 1% 1%% of Nextel Learning via WBT/Self Study 61% 46% 34% 32% 44% 32% 42%Number of ILT/Interw ise Evaluations Received 474 434 1,155 1,322 1,280 813 913Number of WBT/Self -Study Evaluations Received 874 889 535 827 2,583 1,332 1,173
Total Post-Event Evaluations Received 1,348 1,323 1,690 2,149 3,863 2,145 2,086Evaluation Response Rate - ILT/Interw ise 80% 75% 68% 71% 77% 75% 48% 69%Evaluation Response Rate - WBT 20% 5% 9% 6% 9% 14% 13% 9%
> 100% Goal Achievement= 85 - 99% < 85%
HRD 2004 Goals Values pulled f rom Metrics That Matter
Page 3911-Nov-0739
Page 4011-Nov-07
Contact Information
www.knowledgeadvisors.com
For further information on the information contact:Jeffrey BerkChief Operating Officer+1 312 676 4411 (phone)+1 312 676 4401 (fax)[email protected]