© 2013 The Forum Corporation.
Using Measurement to
Increase the Impact &
Effectiveness of Your Leadership
Development Programs
© 2012 The Forum Corporation.
2 © 2013 Forum Corporation
Your Presenters
Tom Atkinson, Ph.D.
Learning & Development Consultant
The Forum Corporation
Jeffrey Berk
Chief Operating Officer
KnowledgeAdvisors
3 © 2013 Forum Corporation
Why Measure Leadership Initiatives?
Improve
Prove
Reinforce
4 © 2013 Forum Corporation
You’ll walk away with:
1. Insights on the challenges and benefits of measuring
leadership development
2. The four big differences between the best
and the rest when it comes to measuring leadership development
3. Frameworks for developing a measurement plan
4. Case studies of what high-performing organizations are
doing to measure their leadership initiatives
Four Takeaways for Today
5 © 2013 Forum Corporation
Quiz
How effective is your organization in using measurement to increase
the impact of your leadership development programs?
– We collect and use data to engage all stakeholders
– We collect and use data within the L&D function
– We collect data efficiently but don’t do much with it
– We collect data, but it’s a cumbersome process
– We have no formal measurement process
6 © 2013 Forum Corporation
Measuring and Demonstrating Impact:
What Top Performing Organizations Do
They treat measurement as part of the solution —
just as a dashboard is part of a car and a lab test is part of a
medical exam.
They involve all key stakeholders — especially business
sponsors — in devising the measurement plan.
They scale the measurement process to the size of the
initiative and the need for precision (and err on the side of simpler
and less costly).
They follow a disciplined process for collecting and
reviewing data to drive action — not to “see if anything
happened” but to “make it happen.”
7 © 2013 Forum Corporation
Gaining buy-in to
the initiative’s
goals, metrics, and
implementation
plan
Laying the
foundation for
behavior change
by teaching new
concepts and
skills
Supporting
leaders in
applying and
embedding new
attitudes and
behaviors on the
job
ALIGN EQUIP SUSTAIN
Measurement Helps Accelerate
Leadership Development
Executing, measuring and evaluating with discipline
8 © 2013 Forum Corporation
Actions:
Conduct a planning
meeting with
stakeholders
ALIGN EQUIP SUSTAIN
Elements of a Measurement Plan
Questions:
What are expected
outcomes?
Who is accountable for
what?
Actions:
Survey participants
and managers during
training
Questions:
Is the learning process
effective & efficient?
Are leaders prepared
to apply learning?
Actions:
Survey participants
and managers after
training; organizational
follow-up
Questions:
What is the evidence of
effectiveness?
What tools, training or
support are needed?
9 © 2013 Forum Corporation
Creating Alignment
What You
Measure
Examples of Specific
Metrics
L&D Accountability Line Accountability
1. Reaction
Increased awareness,
positive attitude toward
topics, motivation to
change
■ Course evaluation ratings
of learning satisfaction
■ Design quality
■ Develop materials
■ Select/ train instructor
■ Project management
■ Clarity of objectives
■ Subject matter experts and data
access
■ Visible support for training
2. Learning
Acquisition of concepts
and skills
■ Pre- and post-course
understanding of material
■ Measurement tools
■ Debrief learning with managers
■ Learning plans and meetings
3. Job Impact
Application of concepts
and skills on the job—
Behaviour change
■ 360-feedback scores
■ Employee satisfaction with
leadership behaviour
■ Observed employee
competencies
■ On-the-job application
activities
■ Application activity
debriefs
■ Model desired behavior
■ Post-training reinforcement
meetings
■ Observe/coach on specific
behaviours
■ Align learning priorities for
leaders with performance
management goals
4. Business Results
Achievement of
positive outcomes for
the company and/or
individual
■ Profitability and growth
■ Employee satisfaction
■ Turnover
■ Costs of recruiting
■ Design process to create
a management-led
measurement plan
■ Design & implement a regular
measurement plan
■ Manage processes, provide
technology, recognise & reward
behaviour change
10 © 2013 Forum Corporation
Creating Alignment
What You
Measure
Examples of Specific
Metrics
L&D Accountability Line Accountability
1. Reaction
Increased awareness,
positive attitude toward
topics, motivation to
change
■ Course evaluation ratings
of learning satisfaction
■ Design quality
■ Develop materials
■ Select/ train instructor
■ Project management
■ Clarity of objectives
■ Subject matter experts and data
access
■ Visible support for training
2. Learning
Acquisition of concepts
and skills
■ Pre- and post-course
understanding of material
■ Measurement tools
■ Debrief learning with managers
■ Learning plans and meetings
3. Job Impact
Application of concepts
and skills on the job—
Behaviour change
■ 360-feedback scores
■ Employee satisfaction with
leadership behaviour
■ Observed employee
competencies
■ On-the-job application
activities
■ Application activity
debriefs
■ Model desired behaviour
■ Post-training reinforcement
meetings
■ Observe/coach on specific
behaviours
■ Align learning priorities for
leaders with performance
management goals
4. Business Results
Achievement of
positive outcomes for
the company and/or
individual
■ Profitability and growth
■ Employee satisfaction
■ Turnover
■ Costs of recruiting
■ Design process to create
a management-led
measurement plan
■ Design & implement a regular
measurement plan
■ Manage processes, provide
technology, recognise & reward
behaviour change
11 © 2013 Forum Corporation
Creating Alignment
What You
Measure
Examples of Specific
Metrics
L&D Accountability Line Accountability
1. Reaction
Increased awareness,
positive attitude toward
topics, motivation to
change
■ Course evaluation ratings
of learning satisfaction
■ Design quality
■ Develop materials
■ Select/ train instructor
■ Project management
■ Clarity of objectives
■ Subject matter experts and data
access
■ Visible support for training
2. Learning
Acquisition of concepts
and skills
■ Pre- and post-course
understanding of material
■ Measurement tools
■ Debrief learning with managers
■ Learning plans and meetings
3. Job Impact
Application of concepts
and skills on the job—
Behavior change
■ 360-feedback scores
■ Employee satisfaction with
leadership behavior
■ Observed employee
competencies
■ On-the-job application
activities
■ Application activity
debriefs
■ Model desired behavior
■ Post-training reinforcement
meetings
■ Observe/coach on specific
behaviors
■ Align learning priorities for
leaders with performance
management goals
4. Business Results
Achievement of
positive outcomes for
the company and/or
individual
■ Profitability and growth
■ Employee satisfaction
■ Turnover
■ Costs of recruiting
■ Design process to create
a management-led
measurement plan
■ Design & implement a regular
measurement plan
■ Manage processes, provide
technology, recognise & reward
behaviour change
12 © 2013 Forum Corporation
Creating Alignment
What You
Measure
Examples of Specific
Metrics
L&D Accountability Line Accountability
1. Reaction
Increased awareness,
positive attitude toward
topics, motivation to
change
■ Course evaluation ratings
of learning satisfaction
■ Design quality
■ Develop materials
■ Select/ train instructor
■ Project management
■ Clarity of objectives
■ Subject matter experts and data
access
■ Visible support for training
2. Learning
Acquisition of concepts
and skills
■ Pre- and post-course
understanding of material
■ Measurement tools
■ Debrief learning with managers
■ Learning plans and meetings
3. Job Impact
Application of concepts
and skills on the job—
Behavior change
■ 360-feedback scores
■ Employee satisfaction with
leadership behavior
■ Observed employee
competencies
■ On-the-job application
activities
■ Application activity
debriefs
■ Model desired behavior
■ Post-training reinforcement
meetings
■ Observe/coach on specific
behaviors
■ Align learning priorities for
leaders with performance
management goals
4. Business Results
Achievement of
positive outcomes for
the company and/or
individual
■ Profitability and growth
■ Employee satisfaction
■ Turnover
■ Costs of recruiting
■ Design process to create
a management-led
measurement plan
■ Design & implement a regular
measurement plan
■ Manage processes, provide
technology, recognize & reward
behavior change
13 © 2013 Forum Corporation
KnowledgeAdvisors
Jeffrey Berk
Chief Operating Officer
KnowledgeAdvisors
14 © 2013 Forum Corporation
Leadership program Key Tools
June 26, 2013 14
1. Identify Leadership KPIs
2. Create Leadership ‘Smart Sheet’
3. Design the Leadership Dashboard
4. Provide a Leadership Program Executive Report
5. Deploy On-the-Job Reinforcement
6. Conduct Business Impact Exercise
15 © 2013 Forum Corporation
Identify Leadership KPIs
15
Outcome Metrics - Business Outcomes
Leader Retention
Employee Engagement
Employee Retention
Leader Performance Ratings
Leader Financial Management
Effectiveness Metrics – Program
Performance
Delivery Quality
Knowledge Gain
Application to Job
Business Alignment
Value
Efficiency Metrics – Cost and Activity
# of Participants
% of Employees Trained
Total L&D Investment
L&D Investment per Participant
Courses Utilized
Classes Conducted
Instructors Utilized
Locations Used
16 © 2013 Forum Corporation
Create Leadership ‘Smart Sheet’
16
Not only
evaluate
The training
but also the
process
such as on-
boarding or
leadership
development
or sales
effectiveness
.
17 © 2013 Forum Corporation
Smart Sheet Forecasting
Connecting training to specific outcomes
18 © 2013 Forum Corporation
Design Leadership Dashboard
18
A visual, concise data
representation
Shows the efficiency,
effectiveness and outcome
metrics
Data from evaluations, tests,
assessments imports, LMS /
HRIS feeds, uploads, etc.
19 © 2013 Forum Corporation
Leadership Executive Reporting
19
Reporting for a non L&D
audience that emphasize
efficiency, effectiveness
and outcomes.
Reporting for L&D
executives that focus on
satisfaction to impact and
value
Hold quarterly reviews
with L&D execs and
sponsors
20 © 2013 Forum Corporation
On-the-Job Reinforcement
20
Use goal setting tools to establish and track changes in business results or
behaviors when participants are on-the-job. It also can identify scrap learning.
21 © 2013 Forum Corporation
Measuring Leadership
Development Programs:
Case Examples
22 © 2013 Forum Corporation
Manager Engagement Case Example
Consumer goods strategic program
Two teams with 2 different managers trained
One manager used reinforcement post training
One manager didn’t use reinforcement
Manager w/ reinforcement had 67% change
Manager w/out reinforcement had 33% change
22 June 26, 2013 © 2012,
KnowledgeAdvisors
23 © 2013 Forum Corporation
Leadership Program Impact Example
23
Leadership program with optional ‘project’
Determine if optional project influenced employee retention in
leadership program
Employees with no learning projects averaged a turnover rate of
21%
Employees with 2 or more learning projects averaged a turnover
rate of 2%
Clearly the control group vs. learning group shows a difference
Source: Dr. John Sullivan, Presentation at Analytics Symposium
24 © 2013 Forum Corporation
Business context
– Global energy company
– Large-scale merger
– Need to drive operational efficiencies and results
– Shift to leaner, team-oriented environment
Implications for front-line supervisors
– Driving functional results and meeting organizational objectives
– Developing employees to be productive and effective
– Achieving results through others
– Developing future leaders
Front-line Leadership Example
25 © 2013 Forum Corporation
Front-line Leadership
Core Program Elements
Measurement and Documentation:
MTM “Smart Sheet” Surveys for Level 1 (Reaction), Level 2 (Learning), and
Level 3 and 4 (Subjective Evaluation of Impact and Results)
MTM Follow Up Surveys with Participants and Managers for Level 3 and 4
(Impact and Results)
Intermodule
Launch
Webinar:
Setting You Up
for Success as
a Supervisor
4-day Classroom Program:
Specific to Front-line manager
competencies such as coaching,
getting results through others, etc.
Reconnect
Webinar:
Share and
Evaluate
Results,
Recommit
Intermodule
26 © 2013 Forum Corporation
Front-line Leadership Program
Measurement Reporting
• Utilization Reporting
• Cost Reporting
Efficiency Measures: Measures the extent to which
time or effort is well used for the intended task or purpose
• Metrics the Matter Program Reporting
• Post Report
• Follow-up Report
Effectiveness Measures: Measure the quality of the
program
• Selected Business Metrics Outcome Measures:
Measures the relationship and/or value of the program to
broader business goals
27 © 2013 Forum Corporation
Leadership Development
Measurement Diagnostic Think of an upcoming Leadership Development Initiative (LDI) you are leading (e.g. high potential leadership
development system, onboarding for supervisors, management essentials, leading change program). Answer
the following questions with respect to that LDI, indicating low, medium or high.
To What Extent… Low Med. High
1. Are executives involved in shaping/defining this specific LDI?
2. Does this LDI link to key business metrics?
3. Will leadership skills/tools introduced in this LDI be immediately applicable on the job for
participants?
4. Will executives of the company be directly involved in implementing this LDI?
5. Is there a measurement discipline and cadence for reporting on talent management?
6. Do managers of participants in this LDI hold their direct reports accountable for critical
leadership behaviors?
7. Is there a measurement discipline and cadence for reporting on leadership development?
8. Is there a documented and communicated leadership development strategy aligned to the
broader business strategy?
9. Is real work application (e.g. individual or group action learning) a part of this LDI?
10.Do managers of participants in this LDI provide ongoing development opportunities, support and
coaching to their direct reports?
TOTAL
(Add up your checks in each column for use on the next page)
28 © 2013 Forum Corporation
Measurement Diagnostic Results
Leadership Development Strategy
Total the number of your “low,”
“medium” and “high” ticks from the
Leadership Development
Measurement Diagnostic.
Mostly “
Low
”
Mix
of
“Mediu
m”
and
“Low
”
Mix
of”
Mediu
m”
and
“Hig
h”
Mostly “
Hig
h”
Efficiency measures: Measures the extent to which time or effort is
well used for the intended task or purpose
Effectiveness measures: Measure the quality of the program from
the participant perspective immediately following the program
Effectiveness measures: Measure the quality of the program from
the participant/participant manager perspective with follow up
reports (60 days after the program)
Outcome measures: Measures the relationship and/or value of the
program to broader business goals using existing business KPIs
Define what you will measure (below) based on
your responses (at right)
29 © 2013 Forum Corporation
Four Takeaways for Today
We have shared:
1. Insights on the challenges and benefits of measuring
leadership development
2. The four big differences between the best
and the rest when it comes to measuring leadership development
3. Frameworks for developing a measurement plan
4. Case studies of what high performing organizations
are doing to measure their leadership initiatives
30 © 2013 Forum Corporation
At Forum, we help our clients develop their leaders, managers and sales
teams to realize specific business objectives faster and more effectively. We
are proud to provide award-winning, comprehensive training solutions backed
by in-depth research to clients throughout North America and around the
world.
forum.com
KnowledgeAdvisors is a talent and learning analytics solutions
and technology firm that helps organizations measure,
communicate and improve the impact of their people by better
managing processes through reliable metrics. Its proprietary
measurement systems and benchmarking expertise help
companies more successfully manage their learning and talent
management investments.
[email protected] knowledgeadvisors.com
Questions?
Top Related